Thursday, May 16, 2019

The SPECTRE story continues ... now it is MDS

Last year (2018) started with shocked Intel CPU vulnerabilities Spectre and Meltdown and two days ago was published another SPECTRE variant know as Microarchitectural Data Sampling or MDS. It was obvious from the beginning, that this is just a start and other vulnerabilities will be found over time by security experts and researchers. All these vulnerabilities are collectively known as Speculative Executions aka SPECTRE variants.

Here is the timeline of particular SPECTRE variant vulnerabilities along with VMware Security Advisories.

2018-01-03 - Spectre (speculative execution by performing a bounds-check bypass) / Meltdown (speculative execution by utilizing branch target injection) - VMSA-2018-0002.3 
2018-05-21 - Speculative Store Bypass (SSB) - VMSA-2018-0012.1
2018-08-14 - L1 Terminal Fault - VMSA-2018-0020
2019-05-14 - Microarchitectural Data Sampling (MDS) - VMSA-2019-0008

I published several blog posts about SPECTRE topics in the past

The last two vulnerabilities "L1 Terminal Fault (aka L1TF)" and "Microarchitectural Data Sampling (aka MDS)" are related to Intel CPU Hyper-threading. As per statement here AMD is not vulnerable.

When we are talking about L1TF and MDS, a typical question of my customers having Intel CPUs is if they are safe when Hyper-Threading is disabled in the BIOS. The answer is yes but you would have to power cycle the physical system to reconfigure BIOS settings which can be pretty annoying and time-consuming in larger environments. That's' why VMware recommends leveraging SDDC concept and set it by software change - ESXi hypervisor advanced setting. It is obviously much easier to change two ESXi advanced settings VMkernel.Boot.hyperthreadingMitigation and VMkernel.Boot.hyperthreadingMitigationIntraVM to the value true and disable hyperthreading in ESXi CPU scheduler without a need of physical server power cycle. You can do it by PowerCLI one-liner in a few minutes which is much more flexible than BIOS changes.

So that's it from the security point of view but what about performance?

It is simple and obvious. When hyper-threading is disabled you will obviously lose the CPU performance benefit of Hyper-Threading technology which can be somewhere between 5 - 20% and heavily depends on the type of particular workload. Let's be absolutely clear here. Until the issue is addressed inside the CPU hardware architecture it will be always the tradeoff between security and performance. If I understand Intel messaging correctly, the first hardware solution for their Hyper-Threading is implemented in Cascade Lake family. You can double check it by yourself here ...
Side Channel Mitigation by Product CPU Model
https://www.intel.com/content/www/us/en/architecture-and-technology/engineering-new-protections-into-hardware.html

You can get hyperthreading performance back but only in VMware vSphere 6.7 U2. VMware vSphere 6.7 U2 includes new scheduler options that secure it from the L1TF vulnerability, while also retaining as much performance as possible. This new scheduler has introduced ESXi advanced setting
VMkernel.Boot.hyperthreadingMitigationIntraVM which allows you to set it to FALSE (this is the default) and leverage HyperThreading benefits within Virtual Machine but still do isolation between VMs when VMkernel.Boot.hyperthreadingMitigation is set to TRUE. This possibility is not available in older ESXi hypervisors and there are no plans to backport it. For further info read paper "Performance of vSphere 6.7 Scheduling Options".

By the way, last year I have spent a significant time to test the performance impact of SPECTRE and MELTDOWN vulnerabilities remediations. If you want to check the results of the performance tests of Spectre/Meltdown 2018 variants along with the conclusion, you can read my document published on SlideShare. It would be cool to perform the same tests for L1TF and MDS but it would require additional time effort. I'm not going to do so until sponsored by some of my customers. But anybody can do it by himself as a test plan is described in the document below.



Friday, May 03, 2019

Storage and Fabric latencies - difference in order of magnitude

It is well known, that the storage industry is in a big transformation. SSD's based on Flash is changing the old storage paradigma and supporting fast computing required nowadays in modern applications supporting digital transformation projects.

So the Flash is great but it is also about the bus and the protocol over which the Flash is connected.
We have traditional storage protocols SCSI, SATA, and SAS but these interface protocols were invented for magnetic disks, that's the reason why Flash over these legacy interface protocols cannot leverage the full potential of Flash technology. That's why NVMe (new storage interface protocol over PCI) or even 3D XPoint memory (Intel Optane).

It is all about latency and available bandwidth. Total throughput depends on I/O size and achievable transaction (IOPS). IOPS on storage systems below can be achieved on particular storage media by a single worker with random access, 100% read, 4 KB I/O size workload. Multiple workers can achieve higher performance but with higher latency.

Latencies order of magnitude:
  • ms - miliseconds - 0.001 of second = 10−3
  • μs - microseconds - 0.000001 of second = 10−6
  • ns - nanoseconds - 0.000000001 of second = 10−9
Storage Latencies

SATA - magnetic disk 7.2k RPM ~= 80 I/O per second (IOPS) = 1,000ms / 80 = 12ms
SAS - magnetic disk 15k RPM ~= 200 I/O per second (IOPS) = 1,000ms / 200 = 5 ms
SAS - Solid State Disk (SSD) Mixed use SFF ~= 4,000 I/O per second (IOPS) = 1,000ms / 4,000 = 0.25 ms = 250 μs.
NVMe over RoCE - Solid State Disk (SSD) ~= TBT I/O per second (IOPS) = 1,000ms / ??? = 0.100 ms =  100 μs
NVMe - Solid State Disk (SSD) ~= TBT I/O per second (IOPS) = 1,000ms / ??? = 0.080 ms =  80 μs
DIMM - 3D XPoint memory (Intel Optane) ~=   the latency less than 500 ns (0.5 μs)

Ethernet Fabric Latencies

Gigabit Ethernet - 125 MB/s ~= 25 ~ 65 μs
10G Ethernet - 1.25 GB/s ~=  μs (sockets application) / 1.3 μs (RDMA application)
40G Ethernet - 5 GB/s ~= μs (sockets application) / 1.3 μs (RDMA application)

InfiniBand and Omni-Path Fabrics Latencies

10Gb/s SDR - 1 GB/s  ~=  2.6 μs (Mellanox InfiniHost III)
20Gb/s DDR - 2 GB/s  ~=  2.6 μs (Mellanox InfiniHost III)
40Gb/s QDR - 4 GB/s  ~=  1.07 μs (Mellanox ConnectX-3)
40Gb/s FDR-10 - 5.16 GB/s  ~=  1.07 μs (Mellanox ConnectX-3)
56Gb/s FDR-10 - 6.82 GB/s  ~=  1.07 μs (Mellanox ConnectX-3)
100Gb/s EDR-10 - 12.08 GB/s  ~=  1.01 μs (Mellanox ConnectX-4)
100Gb/s Omni-Path - 12.36 GB/s  ~=  1.04 μs (Intel 100G Omni-Path)

RAM Latency

DIMM - DDR4 SDRAM ~=  75 ns (local NUMA access) - 120 ns (remote NUMA access)

Conclusion

It is good to realize what latencies we should expect on different infrastructure subsystems 
  • RAM ~= 100 ns
  • 3D XPoint memory ~= 500 ns
  • Modern Fabrics ~= 1-4 μs
  • NVMe ~= 80 μs
  • NVMe over RoCE ~= 100 μs
  • SAS SSD ~= 250 μs
  • SAS magnetic disks ~= 5-12 ms
The latency order of magnitude is important for several reasons. Let's focus on one of them - latency monitoring. It was always a challenge to monitor traditional storage systems as 5 minutes or even 1-minute is simply too large interval for ms (milisecond) latency and the average does not tell you anything about microbursts. However, in lower latency (μs or even ns) systems is 5-minute interval like an eternity. Average, Min and Max of 5-minute interval might not help you to understand what is really happening there. Much deeper mathematical statistics would be needed to have real and valuable visibility into telemetry data. Percentiles are good but Histograms can help even more ...
Wavefront links above are talking mainly about application monitoring but do we have such telemetry granularity in hardware? Mellanox Spectrum claims Real-time Network Visibility but it seems to me as an exception. Intel had an open source project "The Snap Telemetry Framework", however, it seems that it was discontinued by Intel. And what about other components? To be honest, I do not know and it seems to me that real-time visibility is not a big priority for the Infrastructure Industry, however, Operating Systems, Hypervisors and Software Defined Storages could help here. VMware vSphere Performance Manager available via vCenter SOAP API can provide "real-time" monitoring. I'm quoting "real-time" into brackets because it can provide 20-second samples (min, max, average) for metrics in leaf objects. Is it good enough? Well, not really. It is better than a 5-minute or 1-minute sample but still very long for sub-millisecond latencies. Minimum, maximum and average do not have enough information value for some decisions. The histograms could help here. ESXi has an old good tool vscsiStats supporting histograms latency of IOs in Microseconds (us) for virtual machine. Unfortunately, there is no officially supported vCenter API for this tool so it is usually used for short-term manual performance troubleshooting and not for continuous latency monitoring.  William Lam has published a blog post and scripts on how to leverage ESXi API to get vscsiStats histograms. It would be great to be able to get histograms for some objects through vCenter in a supported way and expose such information to external monitoring tools. #FEATURE-REQUEST

Hope this is informative and educational.

Other sources:
Performance Characteristics of Common Network Fabrics: https://www.microway.com/knowledge-center-articles/performance-characteristics-of-common-network-fabrics/
Real-time Network Visibility: http://www.mellanox.com/related-docs/whitepapers/WP_Real-time_Network_Visibility.pdf
Johan van Amersfoort and Frank Denneman present a NUMA deep dive: https://youtu.be/VnfFk1W1MqE
Cormac Hogan : GETTING STARTED WITH VSCSISTATS: https://cormachogan.com/2013/07/10/getting-started-with-vscsistats/
Wiliam Lam : Retrieving vscsiStats Using the vSphere 5.1 API

Tuesday, April 09, 2019

What NSX-T Manager appliance size is good for your environment?

NSX-T 2.4 has NSX Manager and NSX Controller still logically separated but physically integrated within a single virtual appliance which can be clustered as a 3-node management/controller cluster. So the first typical question during NSX-T design workshop or before NSX-T implementation is what NSX-T Manager appliance size is good for my environment.

In NSX-T 2.4 documentation (NSX Manager VM System Requirements) are documented following NSX Manager Appliance sizes.

Appliance Size
Memory
vCPU
Disk Space
VM Hardware Version
NSX Manager Extra Small
8 GB
2
200 GB
10 or later
NSX Manager Small VM
16 GB
4
200 GB
10 or later
NSX Manager Medium VM
24 GB
6
200 GB
10 or later
NSX Manager Large VM
48 GB
12
200 GB
10 or later
In the above documentation section is written that
  • The NSX Manager Extra Small VM resource requirements apply only to the Cloud Service Manager.
  • The NSX Manager Small VM appliance size is suitable for lab and proof-of-concept deployments.
So for NSX-T on-prem production usage, you can use Medium and Large size. But which one? The NSX-T documentation section (NSX Manager VM System Requirements) has no more info to support your design or implementation decision. However, in another part of the documentation (Overview of NSX-T Data Center) is written that
  • The NSX Manager Medium appliance is targeted for deployments up to 64 hosts
  • The NSX Manager Large appliance for larger-scale environments.

Conclusion

Long story short, only Medium and Large sizes are targeted to On-Prem NSX-T production usage. The Medium size should be used in an environment up to 64 ESXi hosts. For larger environments, the Large size is the way to go.

Hope this helps to your NSX-T Plan, Design, and Implement exercise.

Friday, April 05, 2019

vSAN : Number of required ESXi hosts

As you have found this article, I would assume that you know what vSAN is. For those who are new to vSAN, below is the definition from https://searchvmware.techtarget.com/definition/VMware-VSAN-VMware-Virtual-SAN
VMware vSAN (formerly Virtual SAN) is a hyper-converged, software-defined storage (SDS) product developed by VMware that pools together direct-attached storage devices across a VMware vSphere cluster to create a distributed, shared data store. The user defines the storage requirements, such as performance and availability, for virtual machines (VMs) on a VMware vSAN cluster and vSAN ensures that these policies are administered and maintained.
VMware vSAN aggregates local or direct-attached data storage devices to create a single storage pool shared across all ESXi hosts in the vSAN (aka vSphere) cluster. vSAN eliminates the need for external shared storage and simplifies storage configuration and virtual machine provisioning. Data are protected across ESXi hosts. To be more accurate across failure domains, but let's assume we stick with the vSAN default failure domain, which is ESXi host.

vSAN is policy-based storage and policy dictates how data will be redundant, distributed, reserved, etc. You can treat a policy as a set of requirements you can define and storage system will try to deploy and operate the storage object in compliance with these requirements. If it cannot satisfy requirements defined in a policy, the object cannot be deployed or, if already deployed, it becomes in the non-compliant state, therefore at risk.

vSAN is object storage, therefore each object is composed of multiple components.

Let's start with RAID-1. For RAID-1, components can be replicas or witnesses.
Replicas are components containing the data.
Witnesses are components containing just metadata used to avoid split-brain scenario.

Objects components are depicted on the screenshot below where you can see three objects
  1. VM Home 
  2. VM Swap
  3. VM Disk
where each object has two components (data replicas) and one witness (component containing just metadata). 
vSAN Components

The key concept of data redundancy is FTT.  FTT is the number of failures to tolerate. To tolerate failures, vSAN supports two methods of data distribution across vSAN nodes (actually ESXi hosts). It is often referenced as an FTM (Failure Tolerance Method). FTM can be
  • RAID-1 (aka Mirroring)
  • RAID-5/6 (aka Erasure Coding)
As data are distributed across nodes to achieve redundancy and not disks, I'd rather call it RAIN than RAID. Anyway, vSAN terminology uses RAID, so let stick with RAID.

In the table below, you can see how many hosts you need to achieve particular FTT for FTM RAID-1 (Mirroring):

FTTReplicasWitness componentsMinimum # of hosts
0101
1213
2325
3437

In the table below, you can see how many hosts you need to achieve particular FTT for FTM RAID-5/6 (Erasure Coding):
FTTErasure codingRedundancyMinimum # of hosts
0NoneNo redundancy1
1RAID-53D+1P4
2RAID-64D+2P6
3N/AN/AN/A

Design consideration: 
The above number of ESXi hosts are minimal. What does it mean? In case of longer ESXi host maintenance or long-time server failure, vSAN will not be able to rebuild components from affected ESXi node somewhere else. That's the reason why at least one additional ESXi host is highly recommended. Without one additional ESXi host, there can be situations, your data are not redundant, therefore unprotected. 

I have written this article mainly for myself to use it as a quick reference during conversations with customers. Hope you will find it useful as well.

Friday, March 22, 2019

VMware SSO domain design and operational management

Before we will deep dive into VMware SOO management, it is good to understand its architecture and discuss some design considerations. I highly recommend watching the following video


If you have not watched the video yet, do NOT continue and watch it.

The video is great but it is worth to mention that vSphere 6.7 and 6.7U1 come up with few significant improvements in terms of PSC. You can read more about it in the article "External Platform Services Controller, A Thing of the Past". The overall concept stays the same but following enhancements were released:
  • vSphere 6.7 and vSphere 6.5 Update 2 introduced enhanced linked mode support for embedded PSC deployments.
  • The converge utility in vSphere 6.7 Update 1 allows customers with an external PSC deployment to migrate to an embedded PSC deployment. 
  • In vSphere 6.7 is the repoint tool. A stand-alone embedded deployment can join or leave a vSphere SSO Domain. Domain repoint is a feature available in vSphere 6.7 using the cmsso-util CLI command. You can repoint an external vCenter Server across a vSphere SSO domain. New in vSphere 6.7 Update 1 is support for embedded deployment domain repoint. 
So now you should understand VMware architectural basics and we can deep dive into common management operations which can be used also for design verifications.

What is my SSO Domain Name?

It is good to know what is the SSO Domain Name. If I'm logged in PSC (or VCSA /w embedded PSC), following command will show me what is the SSO domain of this particular domain controller (aka PSC)
/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

The output in my home lab is following

 root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost  
 uw.cz  

So my SSO domain is uw.cz

Where my Lookup Service is running?

VCSA command
/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost
show the location of lookup service.

The output in my home lab is following

 root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost  
 https://vc01.home.uw.cz/lookupservice/sdk  

So my lookup service is located at  https://vc01.home.uw.cz/lookupservice/sdk  

What is the SSO Site Name?

VCSA command
/usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name --server-name localhost
show the site name where particular domain controller (aka PSC) is located.

The output in my home lab is following

 root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name --server-name localhost  
 ledcice  

So my PSC is on site ledcice which is the village name where my home lab is located.

Domain replication agreements

If I have more PSCs in SSO Domain, I can determine replication agreements and status by command vdcrepadmin as shown below.

cd /usr/lib/vmware-vmdir/bin
./vdcrepadmin

Examples:
./vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w Administrator_Password
./vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password
./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w Administrator_Password
./vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u ./administrator -w Administrator_Password
./vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN \
-H PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password


These procedures are documented in VMware KB "Determining replication agreements and status with the Platform Services Controller 6.x (2127057)" available at https://kb.vmware.com/kb/2127057

Domain repoint

Domain repoint is a feature available since vSphere 6.5 using the cmsso-util CLI command. You can repoint an external vCenter Server from one PSC to another PSC within the same vSphere SSO domain. Data migration for such repointing is not necessary as all data are replicated across all PSC's within SSO domain. vSphere 6.7 U1 also supports repointing across different SSO domains along with data migrations.

With cmssso-util you can do the following operations
See the cmsso-util CLI command help in the screenshot below

 root@vc01 [ ~ ]# cmsso-util  
 usage: cmsso-util [-h] {unregister,reconfigure,repoint,domain-repoint} ...  
 Tool for orchestrating unregister of a node from LS, reconfiguring a vCenter Server with embedded PSC and repointing a vCenter Server to an external  
 PSC in same as well as different domain.  
 positional arguments:  
  {unregister,reconfigure,repoint,domain-repoint}  
   unregister     Unregister node. Passing --node-pnid will unregister solution users, computer account and service endpoints. Passing --hostId  
             will unregister only service endpoints and solution users.  
   reconfigure     Reconfigure a vCenter with an embedded Platform Services Controller(PSC) to a vCenter Server. Then it repoints to the provided  
             external PSC node.  
   repoint       Repoints a vCenter with an external Platform Services Controller(PSC) to the provided external PSC node.  
   domain-repoint   Repoint Embedded vCenter Server from one vCenter Server to another given domain. The repoint operation will migrate Tags,  
             Authorization, License data to another Embedded node.  
 optional arguments:  
  -h, --help      show this help message and exit  

Command to unregister system vc02.home.uw.cz would look like
cmsso-util unregister --node-pnid vc02.home.uw.cz --username administrator --passwd VMware1! 

How to decommission/remove a PSC from SSO domain?

You should use cmsso-util unregister command to unregister the Platform Services Controller, however sometimes you can get the error, therefore there is another way how to unregister failed PSCs from the SSO database. You can use the command
/usr/lib/vmware-vmdir/bin/vdcleavefed -h hostname -u administrator -w PASSWORD
where hostname is the hostname of the PSC that must be removed.

Usage: vdcleavefed [ -h ] -u [-w ]
        implying offline mode if is provided, and the server must have been down.

        implying online mode if is not provided

It actually alters SSO configuration and removes federation.

How to converge VMware SSO domain topology?

Before vSphere 6.7 U1, there was no way how to converge existing SSO topology, however, vSphere 6.7 U1 allows such convergence. If you have deployed or installed a vCenter Server instance with an external Platform Services Controller, you can convert it to a vCenter Server instance with an embedded Platform Services Controller using the converge utility vcsa-util. You can locate the vcsa-util utility in the vcsa-converge-cli directory in vCenter installation media (DVD).

With vcsa-coverge-cli you can do the following operations


For further practical information and examples, you can read following blog posts


Conclusion

I prefer simplicity over complexity, therefore I personally like all improvements vSphere 6.7 U1 brings into the table. I'm always trying to keep SSO topology as simple as possible. However, in large environments with multiple sites across multiple regions, there can be requirements leading to more complex SSO topologies.

Hope this blog post is useful at least for one other person than me. If you know some other commands or ways how to manage VMware SSO domain, please leave the comment below this blog post. 

Thursday, March 14, 2019

How to transfer large ISO files to ESXi Datastore with USB disk?

I'm participating in one VMware virtualization PoC and we had a need to transfer large ISO file to VMFS datastore on standalone ESXi host. Normally you would upload ISO files over the network but PoC network was only 100Mbps so we would like to use USB disk to transfer ISOs to ESXi host.

There is William Lam blog post "Copying files from a USB (FAT32 or NTFS) device to ESXi" describing how you can use USB with FAT or NTFS filesystem to transfer ISOs but it did not work for me, therefore I wanted to use VMFS filesystem for ISO files transfer. I have VMware Fusion on my MacOSX laptop so it is very easy to spin up VM with ESXi 6.7 and have network access (local within a laptop) to ESXi. I use USB stick connected to the laptop and passed through to VM with ESXi. USB disk is recognized by ESXi but the only challenge is to create VMFS datastore because web management (HTML5 Client) does not allow create new VMFS datastore on USB disks.

Som, the only way is to create it from the command line.

By the way, all credits go to the blog post "Creating A VMFS Datastore On A USB Drive" and here is a quick installation procedure based on the mentioned blog post.

STOP USB Arbitrator

/etc/init.d/usbarbitrator status
/etc/init.d/usbarbitrator stop
/etc/init.d/usbarbitrator status

Find USB disk name

vdq -q
esxcfg-scsidevs -l

MYDISK="/vmfs/devices/disks/t10.SanDisk00Ultra00000000000000000000004C530001161026114003"
echo $MYDISK

Create 10GB VMFS datastore on USB disk

partedUtil getptbl $MYDISK
partedUtil mklabel $MYDISK gpt
partedUtil showGuids
partedUtil setptbl $MYDISK gpt "1 2048 20000000 AA31E02A400F11DB9590000C2911D1B8 0"
vmkfstools -C vmfs6 -S E2USB-ISO-Datastore ${MYDISK}:1

So datastore E2USB-ISO-Datastore is created and you can use upload ISO files to datastore and it goes over the virtual network within laptop computer so it is pretty fast.

Datastore usage on real ESXi host

When ISO files are on USB datastore, you can gracefully shutdown virtual ESXi, remove USB disk from a laptop and connect it to physical ESXi system. USB Arbitrator on physical ESXi system must be temporarily disabled by command ...

/etc/init.d/usbarbitrator stop 

... otherwise, the disk would not be usable within ESXi host as a USB device would be ready for USB passthrough, which you do not want in this particular case. After data transfer to non USB datastore, you can remove USB disk and start USB arbitrator ...

/etc/init.d/usbarbitrator start 

Hope this procedure helps at least one other person in VMware virtual community.

What motherboard chipset is used in VMware Virtual Hardware?

Today I have been asked by one of my customers what motherboard chipset is used in VMware Virtual Hardware. The answer is clearly visible from the screenshot below ...

Motherboard chipset

Motherboard chpset is Intel 440BX (https://en.wikipedia.org/wiki/Intel_440BX). This chipset was released by Intel in April 1998. In the same year, VMware Inc. was founded.

The screenshot above was done in Windows 10 running as Guest OS in VM hardware version 13 but the same chipset is used for VM hardware version 14 so I would assume all VM hardware versions use the same chipset and difference among VM hardware versions are additional features like the maximum amount of RAM, number of NIC adapters, CPU features exposed from physical CPU to virtual CPU, etc.

On two pictures below you can see VM hardware difference between ESXi 3.5 and ESXi 4.0

ESXi 4.0

ESXi 3.5





Friday, March 01, 2019

VMware vSphere Memory Hot Add scalability limitation

VMware vSphere Hot Add CPU/Memory feature has specific requirements and limits. To mention some
  • Virtual machines minimum hardware is version 7.
  • It is not compatible with Fault Tolerance
  • vSphere Enterprise Plus license
  • Hot Remove is not supported
  • Hot-Add/Hot-plug must be supported by the Guest operating system (check at http://vmware.com/go/hcl)
  • Guest-OS technical and licensing limitations had to be taken into consideration.
However, it is good to know about another scalability limitation.

VMware has set a maximum value for hot add memory. By default, this value is 16 times the amount of memory assigned to the virtual machine. For example, if the virtual machine memory is 2 GB, the maximum value for hot add memory is 32GB (2x16).

Actually, this is a good safety mechanism and here is the reason for such restriction ...

When hot memory is enabled, the guest operating system uses a huge amount of kernel memory space in the PFN database. Windows operating system does not have dynamic PFN allocation. When adding memory to the virtual machine, to make it visible to the guest operating system, the PFN database needs to be dynamic as Windows lacks this feature.

Do you want to know more about "Page Frame Number (PFN) database"? Read this article.

This topic is documented in VMware KB https://kb.vmware.com/kb/2020846

Now there is another question. Does this limitation apply only to MS Windows or it applies to Linux OS as well? The short answer is yes it applies to Linux as well. However, for Linux OS there is another limitation. If you are running WM with Linux OS having less then 3GB RAM you can change the memory only up to 3GB RAM in total. If you need more. You have to power off VM, increase memory to for example 4 GB RAM and power on again. When you are running linux with more than 3GB you can use hot memory add but again with a limit to increasing it maximally 16 times.

Hope this is informative.

Memory Hot Add related VMware KBs: