Tuesday, October 18, 2016

vSphere 6.5 announced so what is coming?

vSphere 6.5 has been announced on VMworld 2016 so you can ask yourself what it brings and why consider upgrade or at least upgrade plan.

It is obvious and expected that almost all vSphere 6.5 scalability limits will be increased. Configuration maximums like hosts per vCenter, powered on VMs per vCenter, hosts per cluster, VMs per cluster, vCenters in linked mode, etc are expected to increase. Theses limits are no longer limits for me but if you need it, just wait for vSphere 6.5 GA and double check well known document vsphere-65-configuration-maximums.pdf

However, vSphere users are usually looking for new features. So here they are ...

vCenter features
  • vCenter Server Appliance (aka VCSA) will be recommended as "First Choice" because the coolest new features are available just in VCSA. 
  • Platform Service Controller (PSC) will have out-of-the-box high availability for VCSA using PSC's. You will be able to achieve PSC RTO lower then 5 minutes. << UPDATE: unfortunately, this feature was not released in vSphere 6.5 release so let's hope it will be released in future vSphere 6.5 Updates.  
  • VCSA supports native High Availability support of vCenter service with RTO lower then 15 minutes.
  • VCSA has embedded vCenter Appliance Monitoring and Management to gain visibility into VCSA performance and capacity management including embedded vPostgreSQL database service.
  • VMware Update Manager (VUM) is be fully integrated into VCSA.
  • File level vCenter Server Backup and Restore is complementary backup method to existing VDP image backup. It will be possible to restore vCenter file level backup to fresh VCSA.
  • Content Libraries in vSphere 6.5 have additional features including the option to mount an ISO from a Content Library, update existing templates, and apply guest OS Customization Specifications during VM deployments. If Content Libraries reside on VCSA then you can also make use of vCenter HA, and native Backup and Restore, both new features to vSphere 6.5 mentioned above.
vSphere HA Cluster features
  • vSphere HA cluster wide restart ordering ability with intra-app dependencies during failover. It allows multi-tier application consistency during VM fail-overs.  It is also known as "vSphere HA Orchestrated Restart" because you can create VM to VM dependencies which will force specified VMs to perform HA restarts before others. You can also choose in your vSphere HA settings, when the next VM should begin restarting. At the power-on initiated command, when resources allocated, VMware Tools heartbeats,  etc. You can also set additional timeouts and delays if needed.
  • vSphere HA Admission Control - default Admission Control policy has changed from Slot Policy (Default until 6.5), to ‘Cluster Resource Percentage’. Any time you add or remove a host from the cluster, the failover capacity percentages will update, and the amount of resources required on each host will also be updated automatically.
  • Proactive HA - it integrates with the Server vendor’s monitoring software, via a Web Client plugin, which will pass detailed server health status/alerts to DRS, and DRS will react based on the health state of the host’s hardware. Yes, even the name is "Proactive HA" it is DRS functionality. Confusing? The name was chosen because it has positive impact on availability.
vSphere DRS Cluster features
  • Predictive DRS - it integrates DRS with vROps to provide placement and balancing decisions.
  • Network-Aware DRS - DRS takes physical NIC utilization in to consideration. Once a target host has been chosen for placement/load-balancing, DRS will then check to see if that host’s network is saturated (default is 80% utilization of connected uplinks, but can be configured with ‘NetworkAwareDrsSaturationThresholdPercent’. If the host is considered saturated, it will use a different target host
  • DRS Additional Option : VM Distribution - even distribution of VMs across cluster
  • DRS Additional Option : Memory Metric for Load Balancing - usage of active versus consumed memory for DRS recommendations
  • DRS Additional Option : CPU Over-Commitment - limit the number of vCPUs per pCPU in particular DRS cluster. Specific vCPU:pCPU ratio is set as advanced DRS option MaxVcpusPerClusterPct.
ESXi features and improvements
  • ESXi is pretty stable and best in class hypervisor. However even in this component you can expect some improvements.  For example I/O improvements because of RDMA / PVRDMA. PVRDMA (para-virtualized RDMA) is industry first virtualized RDMA and it allows virtualization of applications which require ultra low latency. And it supports live vMotion which SR-IOV does not.
  • ESXi core storage improvements - Support for 4K Native Drives in 512e mode, SE Sparse Default for VMFS, Automatic Space Reclamation, Support for 1024 devices and 4096 paths (versus 256 and 1024 in the previous versions)
vSphere management features
  • Auto Deploy and Image Builder will be full integrated into WebClient and Host profiles will be improved to smoothly support auto deploy.
  • vSphere Web Client usability and performance will be improved again. It is pretty important because C# client is not available for vSphere 6.5 so vSphere admins will rely on web client. HTML5-based vSphere Client should be included in 6.5 release.
  • Content library improvements - mount ISO directly from content library, customization during VM deployment, improved scale and performance, high availability along with VCSA
  • vSphere 6.5 introduces new REST-based APIs for VM Management
Storage related features
  • VVOLs 2.0 will bring data protection and replication along with support for MSCS, Oracle RAC, NFS 4.1 and SMP-FT.
  • VSAN - Virtual SAN iSCSI Service
  • VSAN - 2-Node Direct Connect with witness Traffic Separation for ROBO
  • VSAN - 512e drive support (still waiting for 4K native support)
Security related features
  • VM Encryption will be new feature to protect your VM data with tenants keys. It enables encryption on a per VM as well as per VMDK basis. It can be integrated with 3rd party Key Management Servers (KMS).
  • vSphere 6.5 also delivers enhanced audit-quality logging capabilities that provide more forensic information about user actions.
WebClient related features
  • In vSphere 6.5, the vSphere Web Client will have no dependency on Client Integration Plug-in (as it exists before).  For the Use Window Session Authentication functionality, you will need the new slimmed down Enhanced Authentication Plug-in, but the other functions (File upload/download, Deploy OVA/OVF) are replicated without CIP.

vSphere 6 is already very mature virtualization platform but vSphere 6.5 brings some very interesting enterprise features if you ask me. The most interesting features for me personally are
  • VCSA and PSC high availability
  • VVOLs 2.0
  • VM Encryption
  • REST-based APIs for vSphere Management
but all other features are cool and very handy as well. 

It is very common practice to wait for Update 1 before upgrading production environments but our labs and test environments are good candidates for vSphere 6.5 release when available. I'm eagerly waiting for GA.

Other related blog posts and resources:

Monday, October 17, 2016

VMware SIOC quick configuration in datacenter scale

I'm currently troubleshooting one weird high kernel latency (KAVG) issue and there is a suspicion that the issue can be somehow related to VMware SIOC which is widely use in customer's environment. To confirm or disprove the issue is really related to SIOC we can simply disable SIOC on all datastores and observe if it has positive impact on kernel latency.

Customer has lot of production datastores grouped in datastore clusters so following PowerCLI one liners can help with quick configuration and validation of SIOC settings across whole datacenter.

SIOC current state for all datastores in datastore clusters
 Get-DatastoreCluster | Get-Datastore | select-object name,type,StorageIOControlEnabled | Format-List -Property *  

Disable SIOC for all datastores in datastore clusters
 Set-Datastore (Get-DatastoreCluster | Get-Datastore) -StorageIOControlEnabled $false | select-object name,type,StorageIOControlEnabled

Enable SIOC for all datastores in datastore clusters
 Set-Datastore (Get-DatastoreCluster | Get-Datastore) -StorageIOControlEnabled $true | select-object name,type,StorageIOControlEnabled  

Thanks PowerCLI!

Thursday, October 13, 2016

Metro Cluster High Availability or SRM Disaster Recovery?

Several years I continuously try to explain my customers that metro cluster is not disaster recovery. I have finally found some time and summarize my thoughts into slide deck which I published on SlideShare. I'm planning to present it at Czech VMUG local meeting on 6 December this year. More info about this particular Czech VMUG event is here.

The goal of my presentation is to explain the difference between multi site high availability (aka metro cluster) and disaster recovery. General concepts are same for any products but presentation is obviously more tailored for specific VMware products and technologies.

You can look at presentation here on SlideShare ...

It would be great to see you at the event if you will be in the town. But in the meantime don't hesitate to write any comment or feedback here and we can have good discussion as there are still two months till the event.

BTW: Kudos to Stanislav Jurena @stan_jurena who already did several reviews and gave me some comments and feedback before first public release.  

Sunday, October 02, 2016

Leveraging VMware LogInsight for VM hardware inventory

There is no doubt that VMware LogInsight is a must for any properly managed vSphere environment. I'm explaining LogInsight benefits to all my customers. The main use case for LogInsight is troubleshooting but there are infinite number of other use cases where LogInsight can help.

During last LogInsight presentation to one of my customers I have got an interesting question if LogInsight can be used for change management. Change management is very broad term but all VM hardware changes should be in log files therefore there is potential to prepare specific interactive analytics to track VM hardware changes.

You can very simply find reconfiguration tasks for particular VMs logged by vCenter or ESXi hosts.

In ESXi logs reconfiguration log entries looks like

 2016-10-01T21:01:27.126Z esx01.home.uw.cz Hostd: info hostd[31C40B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/d22648f3-b9848863/test_cmdb/test_cmdb.vmx opID=7c3fe7f0-976b-4afa-a182-6d9f074a6446-45173-ngc-25-78-fc98 user=vpxuser:UW.CZ\cdave] Send config update invoked  

And in vCenter tasks it looks like
 2016-10-01 21:01:27.434 vc01.home.uw.cz vcenter-server: Reconfigured test_cmdb on esx01.home.uw.cz in home  

The problem is that even we know there were some reconfigurations, there are no VM reconfiguration details like CPU, RAM, disks and NICs. This is the current state in vSphere 6.0 U2 and it would be great if logging of such details would be improved in future vSphere version but this is what it is at the moment.

However, lot of VM hardware details should be in vmware.log which is located in home directory of each VM. The problem is that these logs are placed on VMFS filesystem and not forwarded to syslog so LogInsight cannot be used for log analysis.

Fortunately, there is the blog of the best VMware virtualization blogger William Lam and more specifically his blog post "A Hidden vSphere 5.1 Gem – Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1". William very nicely explains how to redirect VM logs also to syslog.

You have to change following two advanced settings for each Virtual Machine you would like to redirect VM hardware logs to syslog server.
And when VM hardware logs are available in LogInsight you can prepare custom analytics to visualize VM hardware changes over time and put it on to dashboard for future use. You can see my dashboard on screenshot below.

VMware LogInsight dashboard - VM Hardware Inventory 

I have exported my VM Hardware Inventory dashboard as LogInsight content pack and shared it here on github so you can download and import it in to your LogInsight and test it.

Content Pack source code from GitHub is embedded below ...
[ Download Content Pack from GitHub ]

And as always, any comments are more then welcome.