Tuesday, September 22, 2020

vSAN - vLCM Capable ReadyNode

VMware vSphere Lifecycle Manager (aka vLCM) is one of the very interesting features in vSphere 7.  vLCM is a powerful new approach to simplified consistent lifecycle management for the hypervisor and the full stack of drivers and firmware for the servers powering your data center.

There are only a few server vendors who have implemented firmware management with vLCM.

At the moment of writing this article, these vendors are:

  • Dell and HPE for vSphere 7.0
  • Dell, HPE, Lenovo for vSphere 7.0 Update 1

Recently I have got the following question from one of my customers.

"Where I can find official information about certified vLCM server vendors?"

It is a very good question. I would expect such information in VMware Compatibility Guides (VCG), however, there is no such information on "Systems / Servers" VCG but you can find it in "vSAN" VCG.

vSAN VCG contains "vSAN ReadyNodes Additional Features" where one feature is "vLCM Capable ReadyNode". So, there you can find Server Vendors successfully implemented firmware management integration with vLCM, but it is available only for vSAN Ready Nodes. I can imagine, that in the future, vLCM capability may or may not be available even for standard servers and not only for vSAN Ready Nodes.

Tuesday, September 15, 2020

vSAN 7 Update 1 - What's new

vSAN 7 Update 1 has been announced, so let's look at what it brings into the table. 


In the figure below, you will see what new features are available in this release.

New features in vSAN 7 Update 1

VMware HCI Mesh

It is a possibility to mount remote vSAN datastore from external (aka Server) vSAN Cluster to multiple Client vSAN Clusters. An example topology is depicted in the figure below.


HCI Mesh allows multiple client vSAN clusters can mount and share a remote datastore from vSAN Server Cluster. A single datastore can be mounted up to a maximum of 64 hosts, including the server cluster's hosts. With such topology, you can do vMotion (Compute only) across multiple vSphere/vSAN Clusters.

With HCI Mesh you can also do a Full Mesh where vSAN Cluster acts as Client and Server of HCI Mesh. Such topology is depicted below, where all three clusters are both clients and servers.

Full Mesh

Such Full Mesh topology is ideal for homogeneous clusters and equalizes storage consumption across clusters. Scalability of such topology is limited to 5 remote datastores and 5 client datastores. In other words, Client clusters can mount a maximum of 5 remote datastores, and Server clusters can export up to a maximum of 5 client clusters.

Few more notes about HCI Mesh 

  • HCI Mesh Client server is vSAN Cluster. So, the minimum node count is 2-node vSAN Cluster
  • Compute-only vSAN cluster technically works but not recommended and supported at the moment.
  • Meshing Hybrid and All-Flash datastores is supported

vSAN Native File Services

VMware is extending vSAN File Services to SMB protocol. SMB is integrated with Microsoft Active Directory and supports Kerberos authentication. This means that vSAN now supports NFS (version 3 and 4.1) and SMB.


vSAN Data-in-Transit Encryption

vSAN 7 Update 1 increases overall security with native Inter-node encryption of vSAN data traffic over TCP, which ensures data privacy, authentication, and integrity, leveraging existing FIPS-2 validated crypto module. The interesting fact is, that external Key Management Server (KMS) is not required for this feature. However, please be aware that vSAN Mesh and Data-in-Transit Encryption together are not supported in this release.

vSAN Data-in-Transit Encryption

SSD Secure Erase (Secure wipe method)

vSAN 7 Update 1 has the option to securely erase SSDs for Dell and HPE supported devices at this release, so HPE & Dell vSAN Ready Nodes and DellEMC VxRail should be able to use this feature. Other hardware vendors will obviously come in the future.


Overall performance optimization

Based on VMware internal performance tests, vSAN 7 Update 1 should be approximately 30% faster in comparison to vSAN 6.7 U3, which was the fastest vSAN release so far. I know this is a kind of vague statement without further details but I personally believe, the vSAN performance, especially in the All-Flash model, was already good enough for the majority of traditional workloads. Of course, additional performance improvements are always nice to have but I think there are other factors which are more important at least for customers I work with. 


vSAN prior 7 Update 1 supported compression together with deduplication. vSAN 7 Update 1 decouples the compression feature from the deduplication feature to allow space efficiency with a lower performance overhead caused by the deduplication algorithm. 

When both features (Dedup & Compress) are turned on, it works in the following way ... 


  • Per disk group
  • Occurs when destaging to the capacity tier
  • 4KB fixed blocks


  • Occurs after dedup, prior to data being destaged
  • If block is compressed <= 2KB
  • Otherwise full 4KB block is stored

When Deduplication is Turned On, the failure domain is the whole disk group.
In Compression-only mode, the failure domain is reduced to a disk, which is another benefit improving availability, reducing data to resync on failures, therefore improves availability and recoverability SLA.

Compression-only has a significantly higher performance than dedup, that's why OLTP workloads primarily benefit from this feature.


Enhanced Durability During Maintenance Operations

First of all, it is important to understand the difference between Durability and Availability. We are talking here about Durability during vSphere/vSAN Cluster node maintenance mode. The feature is nicely depicted in the figure below.

Enhanced Durability During Maintenance Mode

When ESXi host enters Maintenance Mode you can do 
  • Full data Evacuation <-- time-consuming operation but vSAN objects stay protected per Storage Policy intent
  • Ensure Availability <-- it checks that none vSAN object becomes unavailable, but can become unprotected
  • Nothing <-- this is very dangerous and can cause a data loss
Now with vSAN 7 Update 1, you can Ensure Availability, acknowledge that some vSAN objects can become unavailable in case of some other failure, but doing delta writes to other Host or Failure Domain for the vSAN object placed on a host with the only active replica. This results in additional durability protection (prevents loss of data) because, in the case of Replica 1 failure, the vSAN object becomes unavailable, but you can recover the data very quickly from Replica 2 and Delta Writes after the ESXi host returns back from the planned maintenance mode.

Implementation details:
  • Only available when another Host or Failure Domain is available
  • Can be the same host as the witness component
  • Applies to both RAID Mirroring and Erasure Coding
Faster Host Reboots and Cluster Upgrades
  • Significant improvement in cluster upgrades due to faster host reboots
  • Host metadata is written to disk before a reboot and read back to memory after reboot. This is faster than rebuilding metadata.
  • Average 5x improvement in host reboot times

Shared Witness for 2 Node vSAN Deployments

This feature enables 2-Node vSAN deployments to share a common witness instance. vSAN 7 Update 1 supports up to 64 ROBO 2-Node clusters. With vSAN witness consolidation, customers can reduce deployment cost and operational complexity.

Slack Space optimized, operationalized, and renamed into Reserved Capacity

In the past, VMware recommended keeping 25% - 30% capacity as slack space. Now, the new Reserved Capacity is optimized to requires less space and is dependent on deployment variables and decreases with the number of hosts in vSAN Cluster. Example deployment:
  • 12 node cluster = ~16%
  • 24 node cluster = ~12%
  • 48 node cluster =~ 10%
Reserved capacity is required for:
  • Resync operations such as policy changes, rebalancing, and data movement
  • Rebuild activities due to failures
Slack space is Reserved Capacity

On top of disk space optimization for reserve capacity, vSAN now optionally prevents the consumption of vSAN reserved capacity with optional capacity reserves including
  • Operations reserve
  • Host rebuild reserve
Capacity reserves are soft-thresholds that prevent provisioning activities, thus existing VM I/O is not prevented. And once again, this prevention is completely optional, so it is OPT-IN setting and it is not enabled by default. You can see UI in the figure below.

Please note that the vSAN Reserved Capacity feature is not supported on stretched clusters and 2-node vSAN.

vLCM Enhancements

Dell and HPE are supporting vLCM since vSAN 7 release. Now, in vSAN 7 Update 1, Lenovo ReadyNode models are supported as well.

From the technical features point of view, vLCM has been enhanced in the following areas 
  • vSAN Fault Domains, 2-node, and Stretched Clusters awareness
  • hardware compatibility pre-checks
  • parallel cluster remediation of up to 64 clusters
  • support for environments running NSX-T 3.1
The technical features above are depicted in the following figure.


Simplified Routing for vSAN Network Topologies

Prior to this release, vSAN topologies with external witness had to use static routing in each ESXi host which was a significant management pain. In vSAN 7 Update 1, alternate default gateway can be specified and static routes do not need to be used. This is a very useful feature from an operational point of view if you ask me.

vSAN alternate default gateway

vSAN I/O Insight

This is another very useful feature, to analyze storage workload I/O patterns. It is
  • Quick and easy tool in vSphere Client to capture workload IO characteristics on VSAN
  • Rich IO Pattern metrics and histograms to analyze R/W ratio, Seq/Random ratio, 4K aligned / unaligned ratio, IO size distribution
  • Finer granular IO performance metrics
The tool provides solid data points for the infrastructure team to triage issues with app users without the need for complex external tools. See. the screenshot below.

vSAN IO Insight


vSAN 7.0 U1 release is definitely a significant step forward for VMware software-defined storage, which is an important component of full VMware SDDC stack. And this is a very nice proof of a key SDDC benefit ... Quick response to customers and industry requirements ... This is the reason, why I really like software-defined and hyper-converged infrastructure, leveraging commoditization, integration, and continuous product improvement. vSphere administrator is the king of the house ... at least from the infrastructure point of view :-)

Friday, September 11, 2020

Datacenter Network Topology - Dell OS10 MultiDomain VLT

Yesterday, I have got the following e-mail from one of my blog readers ...

Hello David,

Let me introduce myself, I work in medium size company and we began to sell Dell Networking stuff to go along with VxRail. We do small deployments, not the big stuff with spine/leaf L3 BGP, you name it. For a Customer, I had to implement this solution. Sadly, we are having a bad time with STP as you can see on the design.


Customer design with STP challenge

Is there a way to be loop-free ? I thought about Multi Domain VLT LAG but it looks like it is not supported in OS10. 

I wonder how you would do this. Is SmartFabric the answer ?
Thank you

Well, first of all, thanks for the question. If you ask me, it all boils down to specific design factors - use cases, requirements, constraints, assumptions.

So let's write down design factors


  • Multi-site deployment
  • A small deployment with a single VLT domain per site.
  • Robust L2 networking for VxRail clusters


  • Dell Networking hardware with OS10
  • Networking for VMware vSphere/vSAN (VxRail)

  • No more than a single VLT domain per site is required
  • No vSphere/vSAN (VxRail) Clusters are Stretched across sites
Any unfulfilled assumption is a potential risk. In the case of unfulfilled assumption, the design should be reviewed and potentially redesigned to fulfill the design factors. 

Now, let's think about network topology options we have. 

The reader has asked if DellEMC SmartFabric can help him. Well, SmartFabric can be the option as it is Leaf-Spine Fabric fully managed by External SmartFabric Orchestrator. Something like Cisco ACI / APIC. SmartFabric uses EVPN, BGP, VXLAN, etc. for multi-rack deployment. I do not know the latest details, but AFAIK, it was not multi-site ready a few months ago. The latest SmartFabric features should be validated with DellEMC. Anyway, SmartFabric can do L2 over L3 if you need stretching L2 over L3 across racks. Eventually, it should be possible to stretch L2 even across sites.

However, because our design is targeted to a small deployment, I think the Leaf-Spine is the overkill for small deployment and I always prefer the KISS (Keep It Simple, Stupid) approach. 

So, here are two final options of network topology I would consider and compare.

OPTION 1: Stretched L2 Loop-Free across sites 
OPTION 2: L3 across sites with L2/L3 boundary in TOR access switches 

 Option 1 Stretched L2 Loop-Free across sites

Option 2 - L3 across sites with L2/L3 boundary in TOR access switches 

So let's compare these two options. 

Option 1 - Stretched L2 Loop-Free across sites 


  • Simplicity
  • Stretched L2 across sites allows workload (device, VM, container, etc.) migrations across sites without L2 over L3 network overlay (NSX, SmartFabric, etc.) and re-IP.


  • Topology is not scalable for more TOR access switches (VLT domains), but this ok with the design factors
  • Topology optimally requires 8 links across sites. Optionally, can be reduced to 4 links.
  • Only two routers. One per site.
  • Stretched L2 topology across sites also extends L2 network fault-domain across sites, therefore broadcast storms, unknown unicast flooding, and potential STP challenges are the potential risks.
  • This topology has L3 trombone by design - https://blog.ipspace.net/2011/02/traffic-trombone-what-it-is-and-how-you.html. This drawback can be accepted or mitigated by NSX distributed routing.

OPTION 2 - L3 across sites with L2/L3 boundary in TOR access switches 


  • Better scalability, because other VLT domains (TOR access switches) can be connected to core routers. However, this benefit is not required by the design factors above. 
  • Topology optimally requires 4 links across sites. Optionally, can be reduced to 2 links. This is less than Option 1 requires.
  • Each site is local fault-domain from L2 networking point of view, as L2 fault-domain is not stretched across sites. L2 faults (STP, broadcast storms, unknown unicast flooding, etc.) are isolated within the site. 


  • More complex routing configuration with ECMP and dynamic routing protocol like iBGP or OSPF
  • Four routers. Two per site.
  • L3 topology across sites restricts workload (device, VM, container, etc.) migrations across sites without L2 over L3 network overlay (NSX, SmartFabric, etc.) or changing the IP address of migrated workload.

Conclusion and Design Decision

Both considered design options are L2 loop-free topologies and I hope it should fit all design factors defined above. If you do not agree, please write a comment because anybody can make an error in any design or not foresee all situations, until the architecture design is implemented and validated. 

If I should make a final design decision, it would depend on two other factors
  • Do I have VMware NSX in my toolbox or not?
  • What is the skillset level of network operators (Dynamic Routing, ECMP, VRRP) responsible for the operation?
If I would not have NSX and network operators would prefer Routing High Availability (VRRP) over Dynamic Routing with ECMP (high availability + scalability + performance), I would decide to implement Option 1.

In the case of NSX and willingness to use dynamic routing with ECMP, I would decide to implement Option 2.

The reader in his question mentioned, that his company do not use spine/leaf L3 BGP, therefore Option 1 is probably a better fit for him. 

Disclaimer: I had no chance to test and validate any of the design option considered above, therefore, if you have any real experience, please speak out loudly in the comments.

Tuesday, September 01, 2020

Why NUMA matters?

This is a very short blog post because more and more VMware customers and partners are asking me the same question ... 

"Why NUMA matters?"

If you want to know more I would highly recommend reading Frank Denneman's detailed blog posts or books about NUMA, however, the table below is worth 1000 words.

Local memory access latency is ~ 75 ns.

Remote memory access latency is ~ 132 ns.

A ~ 40% positive impact on performance is worth to incorporate NUMA considerations in your data center infrastructure design.

If you prefer a comprehensive presentation, Frank Denneman should speak on VMworld 2020 about NUMA in session "60 Minutes of NUMA [HCP2453]".

Wednesday, August 26, 2020

iSCSI Best Practices - 2020 review

I have just listened to the Virtually Speaking podcast episode Back to Basics: iSCSI Back in 2014, I wrote a blog post about iSCSI Best Practices, but it was about general iSCSI best practices for any operating system or hypervisor. All these old best practices should be still considered in full-stack design but four design considerations have been highlighted in the above podcast. These four are

  1. Jumbo Frames - more details in my blog post about iSCSI Best Practices
  2. iSCSI Port Binding - more details here at VMware KB https://kb.vmware.com/s/article/2038869
  3. Delay ACK - more details in my blog post and at VMware KB https://kb.vmware.com/s/article/1002598
  4. NoOp Time out - in the article at https://www.jacobhopkinson.com/2019/05/10/iscsi-a-25-second-pause-in-i-o-during-a-single-link-loss-what-gives/
Hope this info helps other folks in the VMware community.

Wednesday, August 12, 2020

Could not connect to one or more vCenter Server Systems: https://vCenterFQDN: 443/sdk"

When I have logged in vCenter 7 vSphere Client in my home lab, I have experienced the message
"Could not connect to one or more vCenter Server Systems: https://vCenterFQDN: 443/sdk"
Below is the screenshot from vSphere Client ...

The message is very clear but such an issue can be caused by various reasons, therefore vpxd.log in vCenter Server appliance should be checked to identify the specific reason causing the unavailability of vCenter Server service providing API endpoint for other services.

In my particular case, I have seen the following log messages ...

2020-08-05T00:07:38.663Z error vpxd[18559] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
2020-08-05T00:07:38.746Z error vpxd[16483] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
2020-08-05T00:07:38.821Z error vpxd[16578] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
2020-08-05T00:07:38.999Z error vpxd[16549] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
2020-08-05T00:07:39.045Z error vpxd[16506] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
2020-08-05T00:07:39.122Z error vpxd[16503] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
2020-08-05T00:07:39.311Z error vpxd[16553] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000

This means that something exhausted the maximum number (2000) of HTTP sessions vCenter daemon VPXD is accepting. Now the question is who is the troublemaker. The other error messages appearing in vpxd.log were about invalid login of hmsuser

2020-08-05T00:00:39.651Z info vpxd[16492] [Originator@6876 sub=Default opID=3ff7839f] [VpxLRO] -- ERROR lro-16 -- SessionManager -- vim.SessionManager.impersonateUser: vim.fault.InvalidLogin:
--> Result:
--> (vim.fault.InvalidLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = ""
--> }
--> Args:
--> Arg userName:
--> "hmsuser"
--> Arg locale:
--> "en"

Who is this hmsuser? Well, HMS stands for HMS service used by vSphere Replication and it was the culprit. After Power Off vSphere Replication, I was able to login to vCenter again. I had no time for further problem management, but because of the lab environment, I will most probably install brand new vSphere Replication on the latest version.

Hope this can help other folks in VMware community.

Monday, July 06, 2020

vCenter Server Appliance - Update installation is in progress

A few days ago, I have updated my home-lab VCSA vCenter Server 7.0 GA (15952498) to vCenter Server 7.0.0a (16189094). Everything seemed ok from the vCenter (vSphere Client) perspective. I was seeing there vCenter build 16189207, which is obviously VCSA 16189094.

The only problem I had was the fact, that I was not able to log in to VCSA VAMI.

After user authentication into VAMI, I was getting a message ... "Update installation is in progress". During the problem isolation, I tested the REST API and was getting the same message through Appliance REST API.

To be honest, I had no idea, how to check VCSA update status and move it forward, therefore I did the log files troubleshooting.

In /var/log/vmware/applmgmt/vami.log I had regularly repeating messages... "Executing VMware vCenter Server SNMP post upgrade actions..."

The next step was parsing PatchRunner.log. There was nothing remarkable except info about the patch stageDir located at "/storage/core/software-update/updates/" where i had two empty directories and file "patch_phase_context.json"

Normally, the directory /storage/core/software-update/updates/ is empty, however I have there still subdirectory with patch … /storage/core/software-update/updates/

I tried to delete the whole directory /storage/core/software-update/updates/ but it did not help.

It seemed, that post-upgrade actions (specifically VMware vCenter Server SNMP) cannot finish.

I HAD NO IDEA ... I WAS STUCK ... the next step was the research on the internal VMware slack channel. I have not found anything, therefore asked for help. In a few days, somebody else had the same issue in his home lab and he got the tip from VMware Engineering guys.

The trick was in file /etc/applmgmt/appliance/software_update_state.conf

I had there the following content

    "state": "INSTALL_IN_PROGRESS",
    "version": "",
    "latest_query_time": "2020-06-12T17:30:46Z",
    "operation_id": "/storage/core/software-update/install_operation"

so the trick was to change the content into

    "state": "UP_TO_DATE"

After the change, I was able to log in to VAMI and use it as usual. Well, almost as usual.
I realized, that I was not able to start another update as buttons [STAGE ONLY] and [STAGE AND INSTALL] are greyed out.

Nevertheless, this is only GUI problem and VCSA CLI update procedure works like a charm, so the update can be done through SSH session and command software-packages install --url or software-packages install --iso

Another positive thing is, that I was able to perform vCenter native backup, so if needed, I have the option to do a VCSA backup and restore, which would probably solve this cosmetic GUI issue.

Disclaimer: This exercise was done in my home lab. If you will experience a similar issue in a production system, please, contact VMware support before any changes in VCSA.

Monday, June 22, 2020

What's new in VxRail 7.0

This is a very short blog post about VxRail 7.0 which has been launched today. First of all, VxRail naming has been aligned with vSphere versioning, hence VxRail 7.0. Here is the summary of the announcement:

  • VxRail 7.0 includes the vSphere 7.0 and vSAN 7.0
  • Customers can now run vSphere Kubernetes on the Dell Tech Cloud Platform, VMware Cloud Foundation 4.0 on VxRail 7.0.
  • With a more accessible Consolidated Architecture, Dell Technologies Cloud Platform can now be deployed starting with a 4-node configuration.
  • Brand-new Dell EMC VxRail D Series – the most extreme yet. The D560/D560F is a ruggedized, durable platform that delivers the full power of VxRail for workloads at the edge, in challenging environments, or for space-constrained areas.
  • Even more platform flexibility with the new VxRail E Series model based on, for the first time, AMD EPYC processors.
  • The single socket, 1U nodes offer dual-socket performance making them ideal platforms for desktop VDI, analytics, and computer-aided design.
  • Enhanced operational benefits with new automation and self-service features enabling customers to schedule and run upgrade health checks in advance of upgrades with VxRail HCI System Software.
  • The addition of Intel® Optane™ DC Persistent Memory to the E560 and P570 platforms offers high performance and significantly increased memory capacity with data persistence.
  • The latest NVIDIA® Quadro RTX™ 6000 and 8000 GPUs to the V570F bringing the most significant advancement in computer graphics in over a decade to professional workflows.
If you were unable to attend the event, you can always visit the VxRail event page, where you can watch it OnDemand!

Monday, May 11, 2020

Undocumented HA Advanced Option - das.restartVmsWithoutResourceChecks

Some time ago, a colleague of mine (@stan_jurena) was challenged by one VMware customer who experienced APD (All Path Down) storage situation in the whole HA Cluster and he expected that VMs will be killed by VMware Hypervisor (ESXi) because of HA Cluster APD response setting "Power off and restart VMs - Aggressive restart policy". To be honest, I had the same expectation. However, after the discussion with VMware engineering, we have been told, that the primary role of HA Cluster is to keep VMs up and running, so "Aggressive restart policy" will restart VM only in certain conditions which are much better described in vSphere Client 7 UI. See the screenshot below.

APD Aggressive restart policy
A VM will be powered off, If HA determines the VM can be restarted on a different host, or if HA cannot detect the resources on other hosts because of network connectivity loss (network partition).

So, what it means? Aggressive restart policy is the same as Conservative but extended for the situation when there is network partitioning. This can be helpful in situations when you have IP storage and experience IP network issues but it does not help in a situation when you have dedicated Fibre Channel SAN and the storage is not available for the whole vSphere Cluster.

We explained to VMware engineering, that there are situations when it is much better to kill all VMs than keep compute (VMs) running without available storage. Based on these discussions, there was created a Feature Request, which was internally named as "super aggressive option" APD. I'm happy to see, that it was implemented and released in vSphere 7 as vSphere advanced option
das.restartVmsWithoutResourceChecks = false (default) / true (super aggressive)
I think this advanced option will be very useful for infrastructure architects / technical designers who will have a good justification to use this advanced option. Here are my typical justifications
  • When the storage subsystem is unavailable for some time, Linux operating system switch file system to Read-Only mode which has a negative impact on running applications. Such a situation typically leads to server restart anyway.
  • When you have an OS/Application clustering solution (for example MSCS) on top of vSphere clustering, having one Application node on one vSphere cluster and another Application node on different vSphere cluster, you prefer to kill VM (App Node) on the problematic cluster (without available storage) to fail-over to App Node (VM) running on the healthy cluster.
Hope this makes sense.

Please leave the comment if you will find this advanced option useful. VMware Engineering might consider adding this option into GUI, based vSphere architects / technical designers' feedback.

  • Duncan Epping wrote the blog post about it here.
  • For other "Advanced configuration options for VMware High Availability in vSphere 5.x and 6.x" check VMware KB 2033250.

Friday, May 08, 2020

CPU capacity planning and sizing

During infrastructure capacity planning and sizing, the technical designer has to calculate CPU, RAM, Storage, and Network resource requirements. Recently, I had an interesting discussion with my colleagues on how to estimate CPU requirements for application workload.

Each computer application requires some CPU resources for computational tasks and additional resources for I/O tasks. It is obvious that the computational tasks require CPU cycles, however, it is not so obvious that there are CPU cycles associated also with I/O. In other words, each I/O requires some CPU resources. It does not matter if it is memory, storage, or network I/O.

For example, a generally accepted rule of thumb in the networking is that
1 Hertz of CPU processing is required to send or receive 1 bit/s of TCP/IP.
[Source: VMware vSphere 6.5 Host Resources Deep Dive]

This would mean 2.5Gb/s would require ~ 2.5Ghz CPU, thus ~ 100% of one CPU Core @ 2.5 GHz.

It would be nice to have a similar rule of thumb for storage I/O. I did quick research (googling) but was not able to find any information about CPU requirements for storage I/O.  I did a quick test in my home lab and start the synthetic random workload (4KB I/O) on 4vCPU VM on ESXi host having CPU at 2 GHz, where I was able to see 5,000 IOPS with CPU utilization 8.5%. This would mean one 4KB I/O requires 136 Hz.

4KB I/O on 4x vCPU VM with pCPU @ 2 GHz 
I did another test with 512 B I/O where 1 IOPS requires 114 Hz.
And for 64KB I/O size, 1 IOPS requires 161 Hz.

0.5 KB I/O => 512 Bytes I/O (4,096 bits) = 114 Hz
4 KB I/O => 4,096 Bytes I/O (32,768 bits) = 136 Hz
64 KB I/O => 65,536 Bytes I/O (52,4288 bits) = 161 Hz

Based on my observations, it is difficult to define the rule of thumb for 1 Bit/s or Byte/s but rather I would define CPU (Hz) requirements for 1 storage I/O.

Based on multiple assessments of real datacenter environments, I would say that typical average storage I/O size is around 40-50 KB, therefore here is my rule of thumb
1 Storage I/O requires ~ 150 Hz of CPU processing
This would mean 10,000 IOPS would require ~ 1.5 GHz, thus 60% of one CPU Core @ 2.5 GHz.

Please, be aware that this is a very simplified calculation but clearly shows that storage workload is always associated with CPU requirements and it can help with capacity planning and infrastructure sizing.

What do you think about this calculation?
Do you observe different numbers?
Would you calculate it differently?
You can leave the comment below the article.

Sunday, May 03, 2020

vSphere 7 - Storage Requirements for the vCenter Server Appliance

I have upgraded vSphere in my home lab and realized that VCSA 7.0 storage requirements increased significantly.

Here are the requirements of vCenter Server Appliance 6.7

Here are the requirements of vCenter Server Appliance 7.0

You can see the difference by yourself. VCSA 7.0 requires roughly 30%-60% more storage than VCSA 6.7. It is good to know it especially for home labs where hardware resources are limited or during logical designs of new environments where you do some math calculation to plan hardware requirements.

Monday, April 20, 2020

What's New in vSAN 7

vSAN 7.0 introduces the following new features and enhancements.

vSphere Lifecycle Manager (vLCM).

vLCM enables simplified, consistent lifecycle management for your ESXi hosts. It uses a desired-state model that provides lifecycle management for the hypervisor and the full stack of drivers and firmware. vLCM reduces the effort to monitor compliance for individual components and helps maintain a consistent state for the entire cluster. In vSAN 7.0, this solution supports Dell and HPE ReadyNodes.

Integrated File Services. 

vSAN native File Service delivers the ability to leverage vSAN clusters to create and present NFS (v4.1 and v3) file shares. vSAN File Service extends vSAN capabilities to files, including availability, security, storage efficiency, and operations management.

Native support for NVMe hotplug.

This feature delivers a consistent way of servicing NVMe devices and provides operational efficiency for select OEM drives.

I/O redirect based on capacity imbalance with stretched clusters.

vSAN redirects all VM I/O from a capacity-strained site to the other site, until the capacity is freed up. This feature improves the uptime of your VMs.

Skyline integration with vSphere health and vSAN health. 

Joining forces under the Skyline brand, Skyline Health for vSphere and vSAN are available in the vSphere Client, enabling a native, in-product experience with consistent proactive analytics.

Remove EZT for a shared disk. 

vSAN 7.0 eliminates the prerequisite that shared virtual disks using the multi-writer flag must also use the eager zero thick format.

Support vSAN memory as a metric in performance service. 

vSAN memory usage is now available within the vSphere Client and through the API.

Visibility of vSphere Replication objects in vSAN capacity view. 

vSphere replication objects are visible in vSAN capacity view. Objects are recognized as vSphere replica type, and space usage is accounted for under the Replication category.

Support for large capacity drives. 

Enhancements extend support for 32TB physical capacity drives and extend the logical capacity to 1PB when deduplication and compression are enabled.

Immediate repair after a new witness is deployed. 

When vSAN performs a replacement witness operation, it immediately invokes a repair object operation after the witness has been added.

vSphere with Kubernetes integration. 

CNS is the default storage platform for vSphere with Kubernetes. This integration enables various stateful containerized workloads to be deployed on vSphere with Kubernetes Supervisor and Guest clusters on vSAN, VMFS and NFS datastores.

File-based persistent volumes. 

Kubernetes developers can dynamically create shared (Read/Write/Many) persistent volumes for applications. Multiple pods can share data. vSAN native File Services is the foundation that enables this capability.

vVol support for modern applications. 

You can deploy modern Kubernetes applications to external storage arrays on vSphere using the CNS support added for vVols. vSphere now enables unified management for Persistent Volumes across vSAN, NFS, VMFS, and vVols.

vSAN VCG notification service.

You can subscribe to vSAN HCL components such as vSAN ReadyNode, I/O controller, drives (NVMe, SSD, HDD) and get notified through email about any changes. The changes include firmware, driver, driver type (async/inbox), and so on. You can track the changes over time with new vSAN releases.

Thursday, April 16, 2020

Logical design - storage performance sizing

Storage performance is always a kind of magic because multiple factors come in to play and not all disks are equal, however, in logical design, we have to do some math because capacity (and performance) planning is a very important part of logical design.

How I do it? I do math with some performance assumptions.

Here are assumptions about various disk type performance I use for my capacity planning exercises.

The below numbers are estimated for the random I/O of 64KB I/O size.

Mechanical hard drives
SAS 15k - 200 IOPS
SATA 7k - 80 IOPS

Read Intensive Solid-state disks (SSD)
SATA Read Intensive SSD - 5,000 IOPS (read) / 1,500 IOPS (write)
SAS Read Intensive SSD - 10,000 IOPS (read) / 2,000 IOPS (write)
NVMe Read Intensive SSD - 30,000 IOPS (read) / 2,500 IOPS (write)

Mixed Used Solid-state disks (SSD)
SATA Mixed Used SSD - 5,000 IOPS (read) / 1,800 IOPS (write)
SAS Mixed Used SSD - 12,500 IOPS (read) / 5,000 IOPS (write)
NVMe Mixed Used SSD - 45,000 IOPS (read) / 10,000 IOPS (write)

Write Intensive Solid-state disks (SSD)
SAS Write Intensive SSD - 12,500 IOPS / 7,500 IOPS (write)

SSD assumptions are based on hardware vendors' spec sheets. One of these spec sheets is available here https://www.slideshare.net/davidpasek/dell-power-edge-ssd-performance-specifications

So with these assumptions, the performance math is relatively simple.

Let's have for example 4x SAS Read Intensive SSD within a disk group.
Such a disk group should have the aggregated read performance 4 x 10,000 IOPS = 40,000 IOPS

As we see in the performance numbers above, there is a significant performance difference between SSD read and write.

For our SAS Read Intensive SSD disk we have 10,000 IOPS for 100% read but only 2,000 IOPS for 100% write so we have to normalize these numbers based on expected read/write ratio. If the planned storage workload is 70% read and 30% write, we can assume the single SSD disk will give as 7,000 + 600 IOPS, so in total 7,600 IOPS.

Storage is typically protected by some RAID protection, where the write penalty comes into play. Write penalty is the number of I/O operations required on the backend for a single frontend I/O operation.

Here are write penalties for various RAID protections
RAID 0 (no protection) - write penalty 0
RAID 1 (mirror) - write penalty 2
RAID 5 (erasure coding / single parity) - write penalty 4
RAID 6 (erasure coding / souble parity) - write penalty 6

So, let's calculate the write penalty and write overhead.

If the planned storage workload is 70% read and 30% write and we have total aggregated normalized performance 30,400 IOPS (4 x 7,600) and we have to split the available performance into READ bucket and WRITE bucket.

In our example scenario, we have
READ bucket (70%) - 21,280 IOPS
WRITE bucket (30%) - 9,120 IOPS

Now we have to apply write penalty on write bucket. So let's say we would like to have RAID 5 protection, therefore 9,120 IOPS available on the backend can handle only 2,280 IOPS coming from the frontend.

Based on these calculations, the aggregated performance of RAID 5 protected disk group of 4 Read Intensive SSD disks should be able to handle 23,560 IOPS (21,280 + 2,280) of front-end storage workload.  Please note, that the considered workload pattern is random, 64KB I/O size with Read/Write ratio 70/30.

Do not forget, that this is just logical planning and estimation, every physical system can introduce additional overhead. In real systems, you can have bottlenecks not considered in this simplified calculation. Example of such bottleneck can be
  • storage controller, driver, firmware
  • low queue depth int the storage path (controller, switch, expander, disk), not allowing I/O parallelism
  • network or other bus latency
Therefore, any design should be always tested after implementation and performance results validated with expected numbers.

Are you doing similar design exercises? Any comment or suggestion is always welcome and appreciated.

Saturday, March 21, 2020

What's new in VMware vSphere 7

vSphere 7 has been announced and will be GA and available to download into our labs very soon. Let's briefly summarize what's new in vSphere 7 and put some links to other resources.

vSphere with Kubernetes

Project Pacific evolved into Integrated Kubernetes and Tanzu. vSphere has been transformed in order to support both VMs and containers. Tanzu Kubernetes Grid Service is how customers can run fully compliant and conformant Kubernetes with vSphere. However, when complete conformance with the open-source project isn’t required, the vSphere Pod Service can provide optimized performance and improved security through VM-like isolation. Both of these options are available through VMware Cloud Foundation 4.

The important takeaway is that Kubernetes is now built into vSphere which allows developers to continue using the same industry-standard tools and interfaces they’ve been using to create modern applications. vSphere Admins also benefit because they can help manage the Kubernetes infrastructure using the same tools and skills they have developed around vSphere.

Improved DRS

DRS used to focus on the cluster state and the algorithm would recommend a vMotion when it would benefit the balance of the cluster as a whole. This meant that DRS used to achieve cluster balance by using a cluster-wide standard deviation model. The new DRS logic computes a VM DRS score on the hosts and moves the VM to a host that provides the highest VM DRS score. This means DRS cares less about the ESXi host utilization and prioritizes the VM “happiness”. The VM DRS score is also calculated every minute and this results in a much more granular optimization of resources.

Another new feature is "DRS Scalable Shares". Scalable Shares solves a problem many have been facing over the last decade or so, which is that DRS does not take the number of VMs in the pool into account when it comes to allocating resources.

Refactored vMotions

Improvements in live migrations of monster workloads. Monster VMs with a large memory & CPU footprint, like SAP HANA and Oracle database backends, had challenges being live-migrated using vMotion. The performance impact during the vMotion process and the potentially long stun-time during the switchover phase meant that customers were not comfortable using vMotion for these large workloads. With vSphere 7, we are bringing back that capability as we have greatly improved the vMotion logic.

How the improvement was achieved?
  • Multi-threading
  • A dedicated vCPU is used for page tracing which means that the VM and its applications can keep working while the vMotion processes are occurring. Prior to vSphere 7, page tracing occurred on all vCPUs within a VM, which could cause the VM and its workload to be resource-constrained by the migration itself. 

Assignable Hardware

There is a new framework called Assignable Hardware that was developed to extend support for vSphere features when customers utilize hardware accelerators. It introduces vSphere DRS (for initial placement of a VM in a cluster) and vSphere High Availability (HA) support for VM’s equipped with a passthrough PCIe device or a NVIDIA vGPU. Related to Assignable Hardware is the new Dynamic DirectPath I/O which is a new way of configuring passthrough to expose PCIe devices directly to a VM. The hardware address of a PCIe device is no longer directly mapped to the configuration (vmx) file of a virtual machine. Instead, it is now exposed as a PCIe device capability to the VM.

Together, Dynamic DirectPath I/O, NVIDIA vGPU, and Assignable Hardware are a powerful new combination unlocking some great new functionality. For example, let’s look at a VM that requires an NVIDIA V100 GPU. Assignable Hardware will now interact with DRS when that VM is powered on (initial placement) to find an ESXi host that has such a device available, claim that device, and register the VM to that host. If there is a host failure and vSphere HA kicks in, Assignable Hardware also allows for that VM to be restarted on a suitable host with the required hardware available.


Bitfusion stays in vSphere 7 as a Tech Preview feature. It allows us to leverage hardware accelerators (GPUs) across an infrastructure (over network fabric) and integrate it with evolving technologies such as FPGAs and custom ASICs using the same infrastructure. This is actually the first implementation of the software-defined composable infrastructure within VMware SDDC stack, therefore it is a very promising and very needed technology for modern applications such as ML/AI workloads.

Precision Time Protocol (PTP)

Precision Time Protocol is helpful for financial and scientific applications requiring sub-millisecond accuracy. PTP requires VM Hardware 17 and it must be enabled on both an in-quest device and an ESXi service. Thus, you have to choose between NTP or PTP.

VM Template Management (Content Library)

VM template check-in and check-out operations with versioning feature. Content Library should also support of controlled replication into remote locations. With these vSphere 7 Content Library improvements, the Content Library is now a mature and very useful tool for VM template management.

vSphere Lifecycle Manager (vLCM)

Desired state of ESXi hosts image (divers & firmware) and host configuration assigned to vSphere Clusters. It requires integration with hardware vendor system management like Dell OMIVV (OpenManage Integration for VMware vCenter) or HPE OneView for VMware vCenter.

vSphere Update Planner

Update Planner is part of vLCM and it monitors current interoperability based on VMware HCL.

vCenter Server Profiles

Export / Import of VCSA (vCenter) configuration. This is good for effective management of a lot of vCenters but please, do NOT expect export/import of vCenter objects like Clusters, VM Folders, Resource Pools, Virtual Switches, etc... This is export / import of VCSA configurations.

VCSA multihoming

VCSA now supports multiple (up to 4) vNICs. The first vNIC (vNIC0) is for management, the second (vNIC1) is dedicated for vCenter Server HA and other vNICs can be used for other purposes like a backup or so.

vCenter and SSO Architecture

vCenter Server Appliance (VCSA) with embedded Platform Service Controler (PSC). External PSC is not supported and it leads into simple SSO topology.

Simplified Certificate Management

Much simpler SSL certificate management. Fewer certificates to manage. For example, vCenter has only two SSL certificates, a Machine SSL certificate, and Certification Authority Certificate. vSphere 7 introduced some vSphere Client UI improvements and also the REST API for certificate management for environments with more vCenters to manage. This is, of course, beneficial for environments implemented based on VMware Validated Designs (VVD) or VMware Cloud Foundation (VCF) environments which is the automated implementation of VVD.

Identity Federation

vCenter is not the key Identity Management System anymore. vSphere Client is using external authentication providers to optimize IDM integration in customer's environments. The first implementation supports only Microsoft Active Directory Federation Services (ADFS), however, VMware SSO still exists, therefore the customer can choose if he will use the brand new Identity Federation or keep existing AD/LDAP authentication through VMware SSO.

vSphere Trust Authority (vTA)

In vSphere 7, vCenter is not trusted authority anymore. vSphere 7 introduces vTA, which creates a hardware root of trust using a separate ESXi host cluster.

vSGX - Support of Intel Software Guard Extensions (SGX)

vSphere 7 introduces support of Intel Software Guard Extensions. I was blogging about SGX a few years ago in blog post Intel Software Guard Extensions (SGX) in VMware VM. Intel SGX allows applications to work with hardware to create a secure enclave that cannot be viewed by the guest OS or hypervisor. With SGX, applications can move sensitive logic and storage into this enclave. SGX is the Intel-only feature. AMD has SEV, which is a different approach.

vSphere 7 Configuration Maximums

Hosts per single vCenter: 2,500
Powered-on VMs on single vCenter: 30,000

Hosts per SSO domain (vCenters in linked mode): 15,000
Powered-on VMs per SSO domain (vCenters in linked mode): 150,000

vCenter Server Latency - vCenter <-> vCenter: 150 ms
vCenter Server Latency - vCenter <-> ESXi: 150 ms
vCenter Server Latency - vSphere Client (web browser) <-> vCenter: 100 ms

The improvements between vSphere 6.7 and 7 are clearly visible in figure below.

For further configuration maximums, look at https://configmax.vmware.com/

Skyline Health for vSphere 7

Skyline Health for vSphere 7 is the unified health check tool for vSphere which works exactly as Skyline Health for vSAN available since vSphere 6.7 U3. It brings into infrastructure operations similar approach developers are doing in agile development methods - automated testing. You can think about it as a set of tests (health check tests) continually testing everything works as expected.

NVMe over Fabric

In vSphere 7, VMware added support for shared NVMe storage using NVMeoF. For external connectivity, NVMe over Fibre Channel and NVMe over RDMA (RoCE v2) are supported.



vSphere 7 is another major vSphere Release. For those who work with VMware virtual infrastructures for ages (see old ESX 3i below), it is amazing where the VMware virtualization platform (vSphere 7, ESXi 7) evolved and what is possible nowadays.

Old good ESXi from Virtual Infrastructure 3 from 2006-ish year :-)
Nowadays, there are totally different reasons to upgrade to the latest vSphere version in comparison to the old days of server consolidation, TCO reduction, and better manageability. Top reasons to upgrade to vSphere 7 are
  • Scalability: The fastest path to the Hybrid/Multi-Cloud and increase scalability through leveraging HCI (Hyper-Converged Infrastructure) 
  • Security: Infrastructure security, secure audits, and account management
  • Performance: maximize performance and efficiency
  • Manageability: Reduce complexity, simplify software patching and hardware upgrades, proactive support technology and services
VMware vSphere 7 new features and incorporation of containers (Kubernetes) into the single platform is another step into VMware's vision to run any app on any cloud. On vSphere 7, you can run
  • monster workloads such as SAP HANA
  • traditional applications in virtual machines
  • modern distributed applications (Cloud Native Applications, CNA) containerized and orchestrated by Kubernetes
This is a great message to all of us, who invested a lot of time (years) to learn, test, design, implement and operate VMware technologies. I can honestly say, ... I LOVE VMWARE ...

Thursday, March 19, 2020

Home Lab 2019/2020

First thing first. Why I have the home lab(s)?

Well, I really need at least one home lab to test and demonstrate VMware vSphere, vSAN, NSX and other components of VMware SDDC stack.

The other reason is, that from time to time I have discussions with other VMware folks discussing our home lab configurations and some of these people have the blog post about their labs. I have never written the blog post about my home lab so far but I realized it is quite useful to document at least some basic information about the lab to quickly show lab details during these discussions and demonstrations. So, here it is.

At the moment, I have two home labs
  1. One in a garage - GARAGE LAB
  2. Second in a flat - APARTMENT LAB
Here are the photos and quick descriptions of my home labs. 


GARAGE LAB vSphere/vSAN Cluster specification
  • 4-node vSphere Cluster (4x ESXi on Dell PE R620) with hybrid vSAN Enabled (4x NVMe 512 GB as cache disks, 8x SATA 500 GB as capacity disks)
  • Each node has 1 CPU Socket (Intel Xeon CPU E5-2620 @ 2.00GHz), 128 Gb RAM, 4x 1Gb Ethernet Port, 1x NVMe 512 GB, 2x SATA 500 GB
GARAGE LAB external storage
  • Flash NAS (NFS) Storage - Synology DS115j , 1x SSD 840 Series 512 GB (465.76 GB SSD)
  • Flash iSCSI Storage - Synology DS115j , 1x SanDisk Ultra II 960 GB (894.3 GB SSD)
  • SATA NAS (NFS) Storage - Synology DS214se, 2x SATA Disk 3TB (2794.52 GB HDD) 

APARTMENT LAB vSphere/vSAN Cluster specification
  • 4-node vSphere Cluster (4x ESXi on Intel NUC) with All-Flash vSAN Enabled (4x SATA M.2 SSD 180 GB as cache disks, 4x SATA SSD 480 GB as capacity disks)
  • Each node is Intel NUC (6i3SYH) having 1 CPU Socket (Intel Core i3-6100U CPU @ 2.30GHz), 32 GB RAM, 1x 1Gb Ethernet Port, 1x SATA M.2 SSD 180 GB, 1x SATA SSD 480 GB
VMware Licensing

As I'm VMware Certified Design Expert, I'm automatically (after application) awarded as VMware vExpert thus entitled to use VMware vSphere Licenses for almost all VMware products. This is IMHO one of the biggest advantages to be VCDX and/or participate in the VMware vExpert program.

Tuesday, February 11, 2020

Host cannot communicate with one or more other nodes in the vSAN enabled cluster

I work as VMware HCI Specialist, therefore I have to do a lot of vSAN testing and demonstrations in my home lab. The only reasonable way how to effectively test and demonstrate different vSAN configurations and topologies is to run vSAN in a nested environment. Thanks to a nested virtualization, I can very easily and quickly build any type of vSAN cluster.

Recently I have experienced the issue in 3-node (nested) vSAN cluster. I have seen vSAN datastore capacity just of a single node instead of three nodes and on hosts was an error message "Host cannot communicate with one or more other nodes in the vSAN enabled cluster".

The first idea was about networking issue but ping between nodes was working ok so it was not a physical network issue. This is the lab environment so all services (mgmt, vMotion, vSAN) are enabled on single VMKNIC (vmknic0) so everything is pretty straight forward.

So what's the problem?

I did some google searching and found that some people were seeing the same error message when experiencing problems with vSAN unicast agents.

Here is the command to list of unicast agents on vSAN node

esxcli vsan cluster unicastagent list

I test it in my environment.
Grrrr. The list is empty!!!! It is empty on all ESXi hosts in my 3 nodes vSAN cluster.

Let's try to configure it manually.

Each vSAN node should have a connection to agents on other vSAN nodes in the cluster.

For example, one vSAN node from 4-node vSAN Cluster should have 3 connections

 [root@n-esx04:~] esxcli vsan cluster unicastagent list  
 NodeUuid               IsWitness Supports Unicast IP Address    Port Iface Name Cert Thumbprint  
 ------------------------------------ --------- ---------------- -------------- ----- ---------- -----------------------------------------------------------  
 5e3ec640-c033-7c7d-888f-00505692f54d     0       true 12321       18:F3:B7:9F:66:C4:C4:3E:0F:7D:69:BB:55:92:BC:A3:AC:E4:DD:5F  
 5df792b0-f49f-6d76-45af-005056a89963     0       true 12321       20:4C:C1:48:F5:2D:04:16:55:F1:D3:F1:4C:26:B5:C4:23:E5:B4:12  
 5e3e467a-1c1b-f803-3d0f-00505692ddc7     0       true 12321       53:99:00:B8:9D:1A:97:42:C0:10:C0:AF:8C:AD:91:59:22:8E:C9:79  

We need the get local UUID of the cluster node.

 [root@n-esx08:~] esxcli vsan cluster get  
 Cluster Information  
   Enabled: true  
   Current Local Time: 2020-02-11T08:32:55Z  
   Local Node UUID: 5df792b0-f49f-6d76-45af-005056a89963  
   Local Node Type: NORMAL  
   Local Node State: MASTER  
   Local Node Health State: HEALTHY  
   Sub-Cluster Master UUID: 5df792b0-f49f-6d76-45af-005056a89963  
   Sub-Cluster Backup UUID:  
   Sub-Cluster UUID: 52c99c6b-6b7a-3e67-4430-4c0aeb96f3f4  
   Sub-Cluster Membership Entry Revision: 0  
   Sub-Cluster Member Count: 1  
   Sub-Cluster Member UUIDs: 5df792b0-f49f-6d76-45af-005056a89963  
   Sub-Cluster Member HostNames: n-esx08.home.uw.cz  
   Sub-Cluster Membership UUID: f8d4415e-aca5-a597-636d-005056997c1d  
   Unicast Mode Enabled: true  
   Maintenance Mode State: ON  
   Config Generation: 7ef88f9d-a402-48e3-8d3f-2c33f951fce1 6 2020-02-10T21:58:16.349  

So here are my nodes
n-esx08 - - 5df792b0-f49f-6d76-45af-005056a89963
n-esx09 - - 5df792b0-f49f-6d76-45af-005056a89963
n-esx10 - - 5df792b0-f49f-6d76-45af-005056a89963

And now the problem is clear. All vSAN nodes have the same UUID.
Why?  Let's check ESXi system UUIDs on each ESXi host.

 [root@n-esx08:~] esxcli system uuid get  

 [root@n-esx09:~] esxcli system uuid get  

 [root@n-esx10:~] esxcli system uuid get  

Note: if you want to check UUID of all ESXi hosts, use following PowerCLI

 Get-VMHost | Select Name,  
   @{N='HW BIOS Uuid';E={$_.Extensiondata.Hardware.SystemInfo.Uuid}},  
   @{N='ESXi System UUid';E={(Get-Esxcli -VMHost $_).system.uuid.get()}}  

So the root cause is obvious.
I use nested ESXi hosts to test vSAN and I forgot to regenerate system UUID after the clone. 
The solution is easy. Just delete UUID from /etc/vmware/esx.conf and restart ESXi hosts.

ESXi system UUID in /etc/vmware/esx.conf

You can do it from command line as well

sed -i 's/system\/uuid.*//' /etc/vmware/esx.conf

So we have identified the problem and we are done. After ESXi hosts restart vSAN Cluster Nodes UUIDs are changed automatically and vSAN unicastagents are automatically configured on vSAN nodes as well.

However, if you are interested in how to manually add a connection to a unicast agent on a particular node, you would execute the following command

esxcli vsan cluster unicastagent add –a [ip address unicast agent] –U [supports unicast] –u [Local UUID] -t [type]

Anyway, such a manual configuration should not be necessary and you should do it only when instructed by VMware support.

Hope this helps someone else in VMware community.