One of my customers is using 2-node vSANs on multiple branch offices. One of many reasons of using 2-node vSAN is the possibility to leverage existing 1 Gb network and use 25 Gb Direct Connect between ESXi hosts (vSAN nodes) without the need of 25 Gb Ethernet switches. Generally they have very good experience with vSAN, but recently they have experienced vSAN Direct Connect outages when testing the network resiliency. The resiliency test was done by administrative shutdown of one vmnic (physical NIC port) on one vSAN node. After further troubleshooting, they realized their particular NICs (Network Adapters) do not propagate link down state to the physical link, when vmnic is administratively disabled by command "esxcli network nic down -n vmnic2".
It is worth to mention, that such network outage does not mean 2-node vSAN outage because that's the reason why we have vSAN witness, however, vSAN is in degraded state and cannot provide mirror (RAID1) protection of vSAN objects.
Such network behavior is definitely strange and we have opened discussion and root cause analysis with hardware vendor, however, we have also started the internal discussion about design alternatives we have to mitigate such weird situations and increase resiliency and the overall availability of vSAN system.
Here are three design options how to implement direct connect networking between two ESXi hosts.
Design Option 1 - Switch independent teaming with explicit fail-over
This design option is generally recommended by VMware.
Benefits: simple configuration, highly available solution
Drawbacks: in case of link state hardware problem, you can be in situation when one vSAN node is using VMkernel interface via vmnic2 uplink and
Design Option 2 - Link Aggregation (LACP)
Benefits: LAG virtual interface with LACPDU heart beating can mitigate the risk of black hole scenario in case of problems with link state.
- LACP configuration is more complicated than switch independent teaming, therefore it has a negative impact on manageability.
- Network availability is not guaranteed with multiple vmknics in some asymmetric failures, such as one NIC failure on one host and another NIC failure on another host. However, more bundled links can increase vSAN traffic availability, because vSAN L3 connectivity would stay up and running until single L1 link is up.
Useful LACP commands
- esxcli network vswitch dvs vmware lacp status get
- esxcli network vswitch dvs vmware lacp stats get
- esxcli network nic down -n vmnic2
- esxcli network nic up -n vmnic2
Design Option 3 - Two vSAN Air Gap Network
Two vSAN Air Gap Networks actually means two vSAN vmkernel interfaces connected to two totally independent (air gap) networks.
Benefits: Little bit easier configuration than LACP.
- Setup is complex and error prone, so troubleshooting is more complex.
- Requires multiple L3 VMkernel interfaces for vSAN traffic.
- Network availability is not guaranteed with multiple vmknics in some asymmetric failures, such as one NIC failure on one host and another NIC failure on another host.
- Source: Pros and Cons of Air Gap Network Configurations with vSAN
Conclusion and design decision
In this blog post, I have described three different option of network configuration for vSAN direct connect. I personally believe, the design option 2 (LACP for vSAN Direct Connect) is the optimal design decision, especially if NIC link state propagation is not reliable as is the case for my customer. However, the design option 2 is solving the issue as well. The final design decision is on the customer.