Jumbo Frames must be configured end-to-end, therefore we should start the configuration in the network core on Physical Switches, then continue to Virtual Switches and finish on VMkernel ports (vmk). These three configuration places are depicted on schema below.
Jumbo Frames on physical switches can be configured per the whole switch or per switch ports. It depends on a particular physical switch but my Force10 switch supports configuration only per switch ports as shown on the screenshot below. The configuration per the whole switch would be easier with less configuration and as far as I know, some Cisco switches support it.
If you have more physical switches, all ports in the path must be configured for Jumbo Frames.
On the screenshot below you can see the Jumbo Frame configuration on my VMware Virtual Distributed Switch.
And last but not least, the configuration on VMkernel port, in this case, the vmk interface used for vSAN traffic.
After any implementation, we should do the test that implementation was successful and all is working as expected. We should log in to ESXi host via ssh and use following ping command
vmkping -I vmk5 -s 8972 -d 192.168.26.122
-d set DF bit (IPv4) or disable fragmentation (IPv6)
The default is 56, which translates to a 64 byte
ICMP frame when added to the 8 byte ICMP header.
(Note: these sizes does not include the IP header).
and here is the result in case everything is configured correctly.
In case the message is longer than configured MTU we would see the following ...
You can ask why we use size 8972 and not 9000?
The reason for the 8972 on *nix devices is that the ICMP/ping implementation doesn’t encapsulate the 28 byte ICMP (8) + IP (20) (ping + standard internet protocol packet) header – thus we must take the 9000 and subtract 28 = 8972. [source & credits for the answer]
Hope this helps.