Friday, January 12, 2018

VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown)

Since January 3, 2017, the whole IT industry is mitigating the impact of SPECTRE and MELTDOWN vulnerabilities and administrators are updating their infrastructures.

Three different CVEs have been identified related to the media described issues:
  • CVE-2017-5753 (Spectre - Variant 1) - Branch target injection
  • CVE-2017-5715 (Spectre - Variant 2) - Bounds check bypass
  • CVE-2017-5754 (Meltdown - Variant 3) - Rogue data cache load
All three security CVEs (common vulnerabilities and exposures) are abusing multi-pipeline architecture (using multiple CPU pipelines) and performance optimization (speculative execution) used in all modern CPUs. Simplified but IMHO very nice explanation is here. If I understand the issue and resolution correctly, it all boils down to new CPU microcodes which must expose special CPU instructions to operating systems which has to incorporate it into their kernels.

VMware infrastructure is, obviously, impacted as well. It is very important to patch the whole infrastructure stack. All subsystems in the infrastructure stack described below must be patched or upgraded for effective mitigation. The virtual infrastructure stack is composed of
  • Operating systems (OS)
  • Virtual Machines / Virtual Appliances
  • Hypervisors
  • CPU microcode (usually part of BIOS)
For guest OS you should obtain patches from your OS software vendors. However, guest operating systems are using new CPU instructions (capabilities IBRS, IBPB, STIBP) to mitigate Spectre/Meltdown vulnerabilities. These capabilities are exposed from physical CPU microcode, therefore, patches from top to down (OS-HYPERVISOR-CPU microcode) is necessary for successful vulnerability mitigation. To expose those new CPU capabilities to guest OS, Virtual Machine hardware 9 and newer must be used otherwise those new capabilities are masked by VM hardware and Guest OS kernel would not be able to use it.

Another consideration should be given to VMware Virtual Appliances. VMware officially released a new Knowledge Base article regarding the investigation of the impact on the VMware Virtual Appliances (KB 52264) and there is also VMware Security Advisory VMSA-2018-0007.

Hypervisor remediation can be classified into the two following categories:
  • Hypervisor-Specific Remediation (documented in VMSA-2018-0002) - Result of exploitation may allow for information disclosure from one Virtual Machine to another Virtual Machine that is running on the same host. 
  • Hypervisor-Assisted Guest Remediation (documented in VMSA-2018-0004) - This issue may allow for information disclosure between processes within the VM.
For Hypervisor-Specific remediation, all relevant patches already exist and are described in security advisory VMSA-2018-0002. VMware recommends applying described patches.

For Hypervisor-Assisted Guest Remediation, all relevant patches exist and are described in security advisory VMSA-2018-0004.

Rogue Data Cache Load or “Meltdown” (CVE-2017-5754) does not affect ESXi, Workstation, and Fusion because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides.

The virtualization specific performance impact of these mitigations should be negligible but performance impact of workloads inside guest OS is expected but workload dependent. There are rumors it might be somewhere between 10% and 30%.

The VMware performance team evaluated the performance costs of the Meltdown/Spectre mitigations for vSphere and official messaging of performance impact is published in VMware KB 52337.

Customers having vROps 6.6.0 and above can leverage specific dashboards to assess performance impact of Spectre & Meltdown patches using vRealize Operations. For more information read this blog post.

Please note, the update order is important.

To enable hardware support for branch target mitigation in vSphere, you should apply these steps, in the order shown below:
  1. Ensure you have EVC enabled on vSphere clusters
  2. For vCenter 6.x, in case you have external PSC, update all external PSCs to the latest patches of PSC 5.5 – 6.5 
  3. Update to the latest patches of vCenter 5.5 – 6.5 
  4. Apply the latest ESXi 5.5 – 6.5 patches
  5. Apply the CPU Microcode by BIOS or alternatively by VMware patch mentioned in VMSA-2018-0004 where VMware distributes CPU Microcode. Please note, CPU Microcode distributed by BIOS is highly recommended.

An ESXi host that is running both a patched CPU microcode and a patched hypervisor will see new CPU features that were not previously available. These new features will be exposed to all Virtual Hardware Version 9+ VMs that are powered-on by that host. Because these VMs now see additional CPU features, vMotion to ESXi without the microcode or hypervisor patches applied will be prevented. There are three new CPU Features
  • IBRS
  • IBPB
These three new CPU capabilities are not known to existing EVC baselines, therefore it seems to ESXi hosts as newer CPU model and prevents vMotion migration. EVC baselines are defined in vCenter and that's the reason why vCenter patch must be applied before ESXi patch and CPU microcode update. Patched EVC on the latest vCenter Server is leveraged to temporarily mask new CPU Features until all hosts in the cluster have new CPU Microcode. This allows vMotion of VMs from unpatched to patched ESXi hosts.


You should verify it in different layers.

Hypervisor level
If you want to know if you have Intel microcode updated you can run following shell command on your ESXi host
if [ `vsish -e get /hardware/msr/pcpu/0/addr/0x00000048 2&>1 > /dev/null ;echo $?` -eq 0 ]; then echo -e "\nIntel Security Microcode Updated\n";else echo -e "\nIntel Security Microcode NOT Updated\n";fi 
On Intel machine with microcode update, you get the output "Intel Security Microcode Updated".
On Intel machine with microcode update, you get the output "Intel Security Microcode NOT Updated".

If you want to do the same for AMD microcode following command should work
if [ $(($((`vsish -e get /hardware/cpu/cpuList/0 | grep -i EBX | head -5 | tail -1 | awk -F: '{print $2}'` & 0x00001000)) >> 12)) -eq 1 ]; then echo -e "\nAMD Security Microcode Updated\n";else echo -e "\nAMD Security Microcode NOT Updated\n";fi 
Guest OS - Linux
  • Very nice blog post with the shell script is here “How to check Linux for Spectre and Meltdown vulnerability". Please note, that if you run the shell script inside guest OS on top of update ESXi host, the script will report that CPU microcode is updated even the physical CPU microcode is vulnerable. However, the information if Guest OS is patched is still valuable.
Guest OS - Microsoft Windows


Intel official response is available here
Intel Analysis of Speculative Execution Side Channels is available here

DELL official responseMeltdown and Spectre Vulnerabilities
DELL KB article with technical information:

HP response I have found so far: Resources to help mitigate Speculative Execution vulnerability in Intel and other processors
HP Support Customer Bulletina00039267en_us
Hewlett Packard Enterprise Product Security Vulnerability Alerts

RedHat: Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715

FreeBSD: How to patch Meltdown and Spectre vulnerabilities on FreeBSD

If you know links to other or better server vendors responses, please leave it in comments.

Hope this article is helpful. And as always, if you see some wrong information or you just have a question or some other experience, do not hesitate to write the comment. I will try to update this article in case something new is available.



Darren said...

The link in "Check VMware KB 52345." is actually just "http://www.processors impacted by intel microcode sightings"

David Pasek said...

Thanks Daren. The link should be fixed now.