Friday, November 21, 2014

ISCSI Best Practices

General ISCSI Best Practices
  • Separate VLAN for iSCSI traffic. 
  • Two separate networks or VLANs for multipathed iSCSI. 
  • Two separate IP subnets for the separate networks or VLANs in multipathed iSCSI. 
  • Gigabit Full Duplex connectivity between Storage Center and all local Servers 
  • Auto-Negotiate for all switches that will correctly negotiate at Gigabit Full Duplex 
  • Gigabit Full Duplex hard set for all iSCSI ports, for both Storage Center and Servers for switches that do not correctly negotiate 
  • Bi-Directional Flow Control enabled for all Switch Ports that servers or controllers are using for iSCSI traffic. 
  • Bi-Directional Flow Control enabled for all Server Ports used for iSCSI (Storage Center and QLogic HBA's automatically enable it). 
  • Bi-Directional Flow Control enabled for all ports that handle iSCSI traffic. This includes all devices in between two sites that are used for replication. 
  • Unicast storm control disabled on every switch that handles iSCSI traffic. 
  • Multicast disabled at the switch level for any iSCSI VLANs. 
  • Multicast storm control enabled (if available) when multicast cannot disabled. 
  • Broadcast disabled at the switch level for any iSCSI VLANs. 
  • Broadcast storm control enabled (if available) when broadcast cannot disabled. 
  • Routing disabled between regular network and iSCSI VLANs. 
  • Do not use Spanning Tree (STP or RSTP) on ports which connect directly to end nodes (the server or Compellent controllers iSCSI ports.) If you must use it, enable the Cisco PortFast option or equivalent on these ports so that they are configured as edge ports. 
  • Ensure that any switches used for iSCSI are of a non-blocking design. 
  • When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments. 
For Jumbo Frame Support
  • Some switches have limited buffer sizes and can only support Flow Control or Jumbo Frames, but not both at the same time. Compellent strongly recommends choosing Flow Control. 
  • All devices connected through iSCSI need to support 9k jumbo frames. 
  • All devices used to connect iSCSI devices need to support it. 
  • This means every switch, router, WAN Accelerator and any other network device that will handle iSCSI traffic needs to support 9k Jumbo Frames. 
  • If is not 100% positive that every device in iSCSI network supports 9k Jumbo Frames, then NOT turn on Jumbo Frames. 
  • Because devices on both sides (server and SAN) need Jumbo Frames enabled, change disable to enable Jumbo Frames is recommended during a maintenance window. If servers have it enabled first, the Storage System will not understand their packets. If Storage System enables it first, servers will not understand its packets.

VMware ESXi iSCSI tunning

  • Disabling "TCP Delayed ACK" (esxcli iscsi adapter param set -A vmhba33 -k DelayedAck -v 0 - command not tested)
  • Adjust iSCSI Login Timeout (esxcli iscsi adapter param set -A vmhba33 -k LoginTimeout -v 60)
  • Disable large receive offload (LRO) (esxcli system settings advanced set -o /Net/TcpipDefLROEnabled 0 or esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabled)
  • Set up Jumbo Frames is configured end to end (esxcli network vswitch standard set -m 9000 -v vSwitch2 and  esxcli network ip interface set -m 9000 -i vmk1)
  • Set up appropriate multi pathing based on iSCSI storage system
  • FlowControl is enabled on ESXi by default. To display FlowControl settings use ethtool --show-pause vmnic0 or esxcli system module parameters list --module e1000 | grep "FlowControl"
If you know about some other best practice, tuning setting or recommendation don't hesitate to leave comment below this blog post. 

No comments: