Friday, March 22, 2019

VMware SSO domain design and operational management

Before we will deep dive into VMware SOO management, it is good to understand its architecture and discuss some design considerations. I highly recommend watching the following video

If you have not watched the video yet, do NOT continue and watch it.

The video is great but it is worth to mention that vSphere 6.7 and 6.7U1 come up with few significant improvements in terms of PSC. You can read more about it in the article "External Platform Services Controller, A Thing of the Past". The overall concept stays the same but following enhancements were released:
  • vSphere 6.7 and vSphere 6.5 Update 2 introduced enhanced linked mode support for embedded PSC deployments.
  • The converge utility in vSphere 6.7 Update 1 allows customers with an external PSC deployment to migrate to an embedded PSC deployment. 
  • In vSphere 6.7 is the repoint tool. A stand-alone embedded deployment can join or leave a vSphere SSO Domain. Domain repoint is a feature available in vSphere 6.7 using the cmsso-util CLI command. You can repoint an external vCenter Server across a vSphere SSO domain. New in vSphere 6.7 Update 1 is support for embedded deployment domain repoint. 
So now you should understand VMware architectural basics and we can deep dive into common management operations which can be used also for design verifications.

What is my SSO Domain Name?

It is good to know what is the SSO Domain Name. If I'm logged in PSC (or VCSA /w embedded PSC), following command will show me what is the SSO domain of this particular domain controller (aka PSC)
/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

The output in my home lab is following

 root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost  

So my SSO domain is

Where my Lookup Service is running?

VCSA command
/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost
show the location of lookup service.

The output in my home lab is following

 root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost  

So my lookup service is located at  

What is the SSO Site Name?

VCSA command
/usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name --server-name localhost
show the site name where particular domain controller (aka PSC) is located.

The output in my home lab is following

 root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-site-name --server-name localhost  

So my PSC is on site ledcice which is the village name where my home lab is located.

Domain replication agreements

If I have more PSCs in SSO Domain, I can determine replication agreements and status by command vdcrepadmin as shown below.

cd /usr/lib/vmware-vmdir/bin

./vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w Administrator_Password
./vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password
./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w Administrator_Password
./vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u ./administrator -w Administrator_Password
./vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN \
-H PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password

These procedures are documented in VMware KB "Determining replication agreements and status with the Platform Services Controller 6.x (2127057)" available at

Domain repoint

Domain repoint is a feature available since vSphere 6.5 using the cmsso-util CLI command. You can repoint an external vCenter Server from one PSC to another PSC within the same vSphere SSO domain. Data migration for such repointing is not necessary as all data are replicated across all PSC's within SSO domain. vSphere 6.7 U1 also supports repointing across different SSO domains along with data migrations.

With cmssso-util you can do the following operations
See the cmsso-util CLI command help in the screenshot below

 root@vc01 [ ~ ]# cmsso-util  
 usage: cmsso-util [-h] {unregister,reconfigure,repoint,domain-repoint} ...  
 Tool for orchestrating unregister of a node from LS, reconfiguring a vCenter Server with embedded PSC and repointing a vCenter Server to an external  
 PSC in same as well as different domain.  
 positional arguments:  
   unregister     Unregister node. Passing --node-pnid will unregister solution users, computer account and service endpoints. Passing --hostId  
             will unregister only service endpoints and solution users.  
   reconfigure     Reconfigure a vCenter with an embedded Platform Services Controller(PSC) to a vCenter Server. Then it repoints to the provided  
             external PSC node.  
   repoint       Repoints a vCenter with an external Platform Services Controller(PSC) to the provided external PSC node.  
   domain-repoint   Repoint Embedded vCenter Server from one vCenter Server to another given domain. The repoint operation will migrate Tags,  
             Authorization, License data to another Embedded node.  
 optional arguments:  
  -h, --help      show this help message and exit  

Command to unregister system would look like
cmsso-util unregister --node-pnid --username administrator --passwd VMware1! 

How to decommission/remove a PSC from SSO domain?

You should use cmsso-util unregister command to unregister the Platform Services Controller, however sometimes you can get the error, therefore there is another way how to unregister failed PSCs from the SSO database. You can use the command
/usr/lib/vmware-vmdir/bin/vdcleavefed -h hostname -u administrator -w PASSWORD
where hostname is the hostname of the PSC that must be removed.

Usage: vdcleavefed [ -h ] -u [-w ]
        implying offline mode if is provided, and the server must have been down.

        implying online mode if is not provided

It actually alters SSO configuration and removes federation.
How to List of Services Registered with Single Sign-On
For vSphere 6.x
/usr/lib/vmidentity/tools/scripts/ --list 
For vSphere 7.x
/usr/lib/vmware-lookupsvc/tools/ list --url http://localhost:7090/lookupservice/sdk

How to converge VMware SSO domain topology?

Before vSphere 6.7 U1, there was no way how to converge existing SSO topology, however, vSphere 6.7 U1 allows such convergence. If you have deployed or installed a vCenter Server instance with an external Platform Services Controller, you can convert it to a vCenter Server instance with an embedded Platform Services Controller using the converge utility vcsa-util. You can locate the vcsa-util utility in the vcsa-converge-cli directory in vCenter installation media (DVD).

With vcsa-coverge-cli you can do the following operations

For further practical information and examples, you can read following blog posts


I prefer simplicity over complexity, therefore I personally like all improvements vSphere 6.7 U1 brings into the table. I'm always trying to keep SSO topology as simple as possible. However, in large environments with multiple sites across multiple regions, there can be requirements leading to more complex SSO topologies.
Update 2021/05/07: I have just been told about very useful tool (lsdoctor) to address potential issues with data stored in the PSC database. See. VMware KB "Using the 'lsdoctor' Tool"

Hope this blog post is useful at least for one other person than me. If you know some other commands or ways how to manage VMware SSO domain, please leave the comment below this blog post. 

Thursday, March 14, 2019

How to transfer large ISO files to ESXi Datastore with USB disk?

I'm participating in one VMware virtualization PoC and we had a need to transfer large ISO file to VMFS datastore on standalone ESXi host. Normally you would upload ISO files over the network but PoC network was only 100Mbps so we would like to use USB disk to transfer ISOs to ESXi host.

There is William Lam blog post "Copying files from a USB (FAT32 or NTFS) device to ESXi" describing how you can use USB with FAT or NTFS filesystem to transfer ISOs but it did not work for me, therefore I wanted to use VMFS filesystem for ISO files transfer. I have VMware Fusion on my MacOSX laptop so it is very easy to spin up VM with ESXi 6.7 and have network access (local within a laptop) to ESXi. I use USB stick connected to the laptop and passed through to VM with ESXi. USB disk is recognized by ESXi but the only challenge is to create VMFS datastore because web management (HTML5 Client) does not allow create new VMFS datastore on USB disks.

Som, the only way is to create it from the command line.

By the way, all credits go to the blog post "Creating A VMFS Datastore On A USB Drive" and here is a quick installation procedure based on the mentioned blog post.

STOP USB Arbitrator

/etc/init.d/usbarbitrator status
/etc/init.d/usbarbitrator stop
/etc/init.d/usbarbitrator status

Find USB disk name

vdq -q
esxcfg-scsidevs -l

echo $MYDISK

Create 10GB VMFS datastore on USB disk

partedUtil getptbl $MYDISK
partedUtil mklabel $MYDISK gpt
partedUtil showGuids
partedUtil setptbl $MYDISK gpt "1 2048 20000000 AA31E02A400F11DB9590000C2911D1B8 0"
vmkfstools -C vmfs6 -S E2USB-ISO-Datastore ${MYDISK}:1

So datastore E2USB-ISO-Datastore is created and you can use upload ISO files to datastore and it goes over the virtual network within laptop computer so it is pretty fast.

Datastore usage on real ESXi host

When ISO files are on USB datastore, you can gracefully shutdown virtual ESXi, remove USB disk from a laptop and connect it to physical ESXi system. USB Arbitrator on physical ESXi system must be temporarily disabled by command ...

/etc/init.d/usbarbitrator stop 

... otherwise, the disk would not be usable within ESXi host as a USB device would be ready for USB passthrough, which you do not want in this particular case. After data transfer to non USB datastore, you can remove USB disk and start USB arbitrator ...

/etc/init.d/usbarbitrator start 

Hope this procedure helps at least one other person in VMware virtual community.

What motherboard chipset is used in VMware Virtual Hardware?

Today I have been asked by one of my customers what motherboard chipset is used in VMware Virtual Hardware. The answer is clearly visible from the screenshot below ...

Motherboard chipset

Motherboard chpset is Intel 440BX ( This chipset was released by Intel in April 1998. In the same year, VMware Inc. was founded.

The screenshot above was done in Windows 10 running as Guest OS in VM hardware version 13 but the same chipset is used for VM hardware version 14 so I would assume all VM hardware versions use the same chipset and difference among VM hardware versions are additional features like the maximum amount of RAM, number of NIC adapters, CPU features exposed from physical CPU to virtual CPU, etc.

On two pictures below you can see VM hardware difference between ESXi 3.5 and ESXi 4.0

ESXi 4.0

ESXi 3.5

Friday, March 01, 2019

VMware vSphere Memory Hot Add scalability limitation

VMware vSphere Hot Add CPU/Memory feature has specific requirements and limits. To mention some
  • Virtual machines minimum hardware is version 7.
  • It is not compatible with Fault Tolerance
  • vSphere Enterprise Plus license
  • Hot Remove is not supported
  • Hot-Add/Hot-plug must be supported by the Guest operating system (check at
  • Guest-OS technical and licensing limitations had to be taken into consideration.
However, it is good to know about another scalability limitation.

VMware has set a maximum value for hot add memory. By default, this value is 16 times the amount of memory assigned to the virtual machine. For example, if the virtual machine memory is 2 GB, the maximum value for hot add memory is 32GB (2x16).

Actually, this is a good safety mechanism and here is the reason for such restriction ...

When hot memory is enabled, the guest operating system uses a huge amount of kernel memory space in the PFN database. Windows operating system does not have dynamic PFN allocation. When adding memory to the virtual machine, to make it visible to the guest operating system, the PFN database needs to be dynamic as Windows lacks this feature.

Do you want to know more about "Page Frame Number (PFN) database"? Read this article.

This topic is documented in VMware KB

Now there is another question. Does this limitation apply only to MS Windows or it applies to Linux OS as well? The short answer is yes it applies to Linux as well. However, for Linux OS there is another limitation. If you are running WM with Linux OS having less then 3GB RAM you can change the memory only up to 3GB RAM in total. If you need more. You have to power off VM, increase memory to for example 4 GB RAM and power on again. When you are running linux with more than 3GB you can use hot memory add but again with a limit to increasing it maximally 16 times.

Hope this is informative.

Memory Hot Add related VMware KBs: