|Modern Storage System Architecture - Disk Pool|
How to design your infrastructure to keep your storage performance predictable? The short answer is to implement QoS. Of course, there is the longer answer.
Storage QoSvSphere Storage QoS is called Storage I/O Control or simply SIOC. Storage-based per LUN QoS will not help you on VMFS because multiple VM virtual disks (vDisks) are accomodated on single LUN (datastore), however, all modern storages support Virtual Volumes (aka VVols) which is VMware framework how to manage vDisks directly on storage but the implementation is always specific to particular storage vendor. Nevertheless, in general, VVols can do QoS per vDisk (aka VMDK) as each vDisk is represented on storage as independent volume (aka LUN or better say sub-LUN or micro-LUN).
vSphere Storage QoS (SIOC) supports
- IOPS Limits
- IOPS Reservations
- IOPS Shares
vSphere SIOC V1In case you want to use vSphere SIOC V1 storage QoS, it is good to know, at least some, SIOC V1 "IOPS limit" implementation details
- I/Os are normalized at 32KB so 128KB I/O is counted as 4 I/Os. This is vSphere default which can be changed by advanced parameter Disk.SchedCostUnit (default value is 32768 bytes) to satisfy different customer's specific requirements. Disk.SchedCostUnit is configurable and allowable values range between 4K to 256K.
- In the latest ESXi version ESXi 6.5 build 7526125) IOPS limit does not interfere with SVmotion (Storage vMotion). However, this is different behavior than in previous ESXi versions where storage vMotion was billed to the VM.
- If VM has multiple disks with IOPS limits on the same datastore (LUN) then all limits are aggregated and enforced on such datastore per the whole VM. If vDisks of single VM are located on different datastores then limits are enforced independently per each datastore. The behavior is different on NFS datastores. All this behavior is explained in VMware KB "Limiting disk I/O from a specific virtual machine" - https://kb.vmware.com/kb/1038241
vSphere SIOC V2It is worth to mention that SIOC V1 and SIOC V2 can co-exist. SIOC V2 is very different when compared to V1.
SIOC V2 implementation details
- I/Os are NOT normalized at static I/O size like SIOC V1. In other words, SIOC V2 does not have Disk.SchedCostUnit implemented.
- SIOCv2 is implemented using IO Filter framework and is managed by using SPBM Policies.
VVols QoSVVols QoS is out of the scope of this article.
Hope it is informative but as always do not hesitate to contact me for further details or discussions.