I've just found in /var/log/vmkernel.log lot of following storage errors
2012-12-19T01:34:02.010Z cpu2:4098)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x93 (0x412401965f00, 5586) to dev "naa.60060e80102d5f500511c97d000000d4" on path "vmhba2:C0:T0:L2" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32. Act:NONE
2012-12-19T01:34:02.010Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401965f00) 0x93, CmdSN 0xc6fd5 from world 5586 to dev "naa.60060e80102d5f500511c97d000000d4" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32.
The main part of log entry is "failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x96 0x32"
If I understand correctly
D: 0x2 = DEVICE CHECK CONDITION
Sense code 0x5 = ILLEGAL REQUEST
What is it? What doe's it mean?
I have ESXi 5.0 build 768111, storage HDS AMS 2300, CISCO UCS blade system, CISCO FC switches.
I've thought more about the root cause ... important detail is that it is happen when storage vMotion or other data migration is happening. So I've a hypotheses that it is related to VAAI. Storage is VAAI enabled and VAAI is supported. However disk block size is different on datastores (we are just in the middle of migration from VMFS-3 to VMFS-5).
So I've to do deeper diagnostic and root cause troubleshooting.
Solved, VAAI primitives must be enabled also on HDS Host Masking. For more information check