Monday, November 16, 2015

Creating a Capacity & Performance Management Dashboard in vRealize Operations 6.x

I'm long time proponent of performance SLAs in modern virtual datacenters. Performance SLAs is nothing else than mutual agreement between service provider and service consumer. Agreement describes what performance of particular resource consumer can expect and provider should guarantee. The performance SLA is important mainly on shared resources. On dedicated resources consumer knows exactly what to expect from performance point of view. In modern virtualized datacenters is almost everything shared therefore performance SLAs are a must and all service consumers should require it.

The most important shared resources on virtualized infrastructures having significant impact on application performance are CPU and Disk. The rest infrastructure resources - Memory and Network - are important as well but CPU and Disk performance was typical final root cause of any performance troubleshootings I did over several years. In VMware vSphere we can typically identify CPU Contention by CPU %RDY metric and Storage Contention based on disk response time of normalized I/O's. We can identify such issues during troubleshooting when infrastructure consumers are complaining about application performance. We call it reactive approach.  But more mature approach is to identify potential performance issues before application is affected.  We call it proactive approach. And that's where performance SLA's and threshold monitoring come in to play.

Infrastructure performance SLA can looks like

  • CPU RDY is below 3% (notification threshold 2%)
  • If # of vDisk IOPS < 1000 then vDISK Response Time is below 10ms (notification threshold 7ms)

Simple right? These two bullets above should be clearly articulated, explained and agreed between infrastructure service provider and infrastructure consumer building and providing application services on top of infrastructure.

So now how to monitor these performance metrics? I have just found Sunny Dua and Iwan Rahabok blog post covering this topic and step by step problem solution with vRealize Operations 6.x. Sunny and Iwan prepared and shared with community customized vROps supermetrics, views and dashboards for performance capacity planning.  To be honest I do not have big experience with vROps so far but it seems to me as very helpful tool for anybody using vRealize Operations as monitoring platform.

Let's try to build and provide mature IT with clearly articulated SLA's and with agreed expectations between service providers and service consumers.

Friday, November 06, 2015

VMware Tools 10 and "shared productLocker"

VMware tools (aka VM tools, vmtools) were always distributed together with ESXi image however it changed with VMware Tools 10. VMware is now shipping VM tools also outside of the vSphere releases. For more information look at this blog post.

Where can I get VMware Tools?

Option 1/ VMware Tools 10 can be downloaded from More specifically from this direct URL. Please be aware, that you must be logged in to before direct link works.

Option 2/ VMware Tools can be also downloaded from here without any login credentials required. The latest version (10.0.6 at the moment of writing this blog post) is available here. Benefit of option (2) is that there are binaries which can be use directly within guest operating systems.

Option 3/ Open-vm-tools. This option is available for linux based and FreeBSD operating systems. You can read more about it here , in SourceForge or in GitHub. Optimally the open-vm-tools should be available through standard package manager of your unix-like operating system.

It is worth to mention that
  • VMware Tools 10 are backward compatible and are independent on ESXi version.
  • You can share VMware Tools 10 packages with application/OS owners and they can update VMware Tools by them selves during OS update procedure. But even your OS owners will do VMtools update by them selves it is still worth to have VMware tools available in ESXi to have ability of VMtools comparison from vSphere point of view.
VMtools versions

VMtools reports version as a number. For example version number 9536 is version 9.10.0. You can map VMtools version number to human readable version by leveraging this file.


Ok, so what? If you update your VMware Tools in old way (together with ESXi image) you - or VMware Update Manager - have to upload VMware tools to each ESXi host with following impacts
  1. It takes significantly more time especially in bigger environments.
  2. You can potentially end-up with different VM tools version in different ESXi hosts in your datacenter. It can be reported as outdated VM tools after vMotion of particular VM across different ESXi hosts.
The thing is that VMware Tools 10 and above doesn't need to be updated automatically with ESXi update on each ESXi host. You can update ESXi hosts without VMware Tools and later update VMware tools bundle just on single shared place - in shared productLocker location.

So what actually the "productLocker" is? The "productLocker" is essentially VMware Tools directory. This directory is on each ESXi host by default however it can be reconfigured and pointed to directory on shared datastore. Such configuration is called "shared productLocker" and it enables us to do centralize update of VM tools. It is worth to mention that all your ESXi hosts must be reconfigured to use this shared location.

Reconfiguration has to be done via ESXi host advanced configuration option UserVars.ProductLockerLocation. So it has to be changed manually on each host, you can change it automatically via custom PowerCLI script or if you have Enterprise Plus Edition you can leverage Host Profiles. The last option works for me perfectly.

Below is screenshot showing /productLocker directory structure and content on ESXi 6 host ...

/productLocker directory structure and content 
If you use central location for VMware Tools then you don't need update ESXi hosts with full ESXi image but only with ESXi image without VMware Tools. See example of different profiles in ESXi 6 Update 2 image below.
[root@esx01:~] esxcli software sources profile list -d /vmfs/volumes/NFS-SYNOLOGY-SATA/ISO/
Name                              Vendor        Acceptance Level
--------------------------------  ------------  ----------------
ESXi-6.0.0-20160301001s-no-tools  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20160302001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20160301001s-standard  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20160302001-no-tools   VMware, Inc.  PartnerSupported
Profile names with postfix no-tools can be used for ESXi update without updating VMware Tools to each ESXi host. For further details how to update ESXi hosts with particular profile see my other post - How to update ESXi via CLI.

Current ESXi host product locker location can be displayed by esxcli command
esxcli system settings advanced list -o /UserVars/ProductLockerLocation 
and the output should looks like ...
[root@esx01:~] esxcli system settings advanced list -o /UserVars/ProductLockerLocation   
Path: /UserVars/ProductLockerLocation   
   Type: string 
   Int Value: 0
   Default Int Value: 0
   Min Value: 0   
   Max Value: 0   
   String Value: /locker/packages/6.0.0   
   Default String Value: /locker/packages/6.0.0   
   Valid Characters: *   
   Description: Path to VMware Tools repository
To change location you can use following esxcli command
esxcli system settings advanced set -o /UserVars/ProductLockerLocation --string-value "/vmfs/volumes/NFS-SYNOLOGY-SATA/VMtools/latest
And you can verify that setting was changed ...
[root@esx02:~] esxcli system settings advanced list -o /UserVars/ProductLockerLocation
   Path: /UserVars/ProductLockerLocation
   Type: string
   Int Value: 0
   Default Int Value: 0
   Min Value: 0
   Max Value: 0
   String Value: /vmfs/volumes/NFS-SYNOLOGY-SATA/VMtools/latest   
   Default String Value: /locker/packages/6.0.0   
   Valid Characters: *   
   Description: Path to VMware Tools repository
ESXi host has to be rebooted to activate new Product Locker Location.

Hope this helps other folks in VMware community to simplify operations with VMware Tools.

UPDATE 2018-02-05: I have just been told about very nice PowerCLI command-lets allowing to manage VMtools.  Command-let Update-VMToolsImageLocation updates the link /productLocker in ESXi host directly to avoid host reboot and command-let Invoke-VMToolsUpgradeInVMs in the combination with shared productLocker is a very nice way how to automatically update VMtools.
All VMtools management command-lets are available on GitHub here

UPDATE 2019-04-04: New VMware blog post about this topic has been published in January 2019. The blog post is here Configuring a VMware Tools Repository in vSphere 6.7U1.
In comments is the link to PowerCLI code leveraging vSphere API to change the productLocker location. The link to the code is here. The code is clear and simple ...

 $esxName = 'MyEsx'  
 $dsName = 'MyDS'  
 $dsFolder = 'ToolsRepo'  
 $esx = Get-VMHost -Name $esxName  
 $ds = Get-Datastore -Name $dsName  
 $oldLocation = $esx.ExtensionData.QueryProductLockerLocation()  
 $location = "/$($ds.ExtensionData.Info.Url.TrimStart('ds:/'))$dsFolder"  
 Write-Host "Tools repository moved from"  
 Write-Host $oldLocation  
 Write-Host "to"  
 Write-Host $location