This is a vSphere on Nimble best practices express guide – for those that are on-the-go or too busy to read our 20+ page best practices guide, this is a quick and dirty express edition. Keep in mind this one is specific to VMDK ONLY. We will have in-guest attached one in the near future.
Here we go, feedback/additional questions welcome.
Let’s start with the how I/O flows from the virtual machine to the Nimble, through the VMware storage stack, down to the NIC, and through the network fabric.
- Guest OS (VM running Windows or Linux): at this layer, ensure the OS partition is 4K aligned. Misaligned OS partition starting offset could cause a write operation to span more than one block on the storage array, causing a single write to incur an extra ‘read-modify-write’ operation. For optimal performance, ensure this is aligned. VMware recommends this as a best practice as well. Read more info on this here
- Virtual SCSI HBA: joint recommendation from both VMware and Nimble is to distribute the VMDKs over multiple virtual SCSI HBAs. For example, if you have a database VM with a dedicated VMDk for OS, DB files, transaction logs, you’d want three vSCSI HBAs (vscsi0, vscsi1 and vscsi2), each with its own VMDK. This will yield slightly better performance without much effort (just a matter of point and click in vSphere Client)
- If using software iSCSI initiator in ESX, ensure you leverage the native MPIO capability of the PSA NMP architecture. Make sure the iSCSI vmhba is bound to multiple vmkernel ports (if using Nimble 1G model, you’d want 4 vmnics dedicated to iSCSI with one vmkernel port each; if 10G, then two vmnics)
The sw/iscsi vmhba should look something like this:
Or if you are an esxcli fan, then use the following command to verify:
#esxcli iscsi networkportal list | grep vmk*
The output should look something like this:
Ensure the following rule exists in the /etc/vmware/esx.conf file (purpose is to assign PSP_RR as the default multipathing PSP for all Nimble volumes)
/storage/plugin/NMP/config[VMW_SATP_ALUA]/defaultpsp = “VMW_PSP_RR”
# esxcli storage nmp satp rule list | grep Nimble
Your output should look something like the following:
VMW_SATP_ALUA Nimble user VMW_PSP_RR
If the rule or option line above doesn’t appear, execute the following line in ESXi shell to add it:
#esxcli storage nmp satp rule add –psp=VMW_PSP_RR –satp=VMW_SATP_ALUA –vendor=Nimble
Path Switching Criteria
It’s best to configure iops=0 &bytes=0 settings for PSP_RR path switching criteria. The value ‘0’ throws off a lot of people. We have seen very positive performance test results from these settings, and a good number of our customers have reported better IOPS numbers through this tweak as well. Here’s why we recommend ‘0’ for both iops & bytes (Thanks Jay Wang from Nimble Engineering):
Setting iops=0 and bytes=0 when policy=rr indicates that it disables the path switching based on commands and blocks. The VMW_PSP_RR will simply round robin the path disregard of io counts or bytes on each path. If the iops is set to 10 (iops=10) when the policy=rr, it means that the PSP will wait until a path accumulated to 10 ios before switching path. Therefore, setting iops=0 and bytes=0 can utilize more paths at the same time than setting iops=10.
4. ESX NIC dedicated for iSCSI should have flow control enabled
5. Network Configuration
- Ensure flow control is enabled on:
- vmnics that are bound to sw/iscsi initiator
- switch ports where the vmnics are connected to
- switch ports where the nimble data interfaces are connected to
- Turn OFF STP
- Disable Unicast Storm Control
- Stack switches method is preferred over ISL
6. Nimble Side
- performance policy is set to “VMware ESX or ESX5” (defaults to 4KB block size with cache enabled)
- choose “vCenter Server” as snapshot synchronization if you have applications that need VSS quiescing
- make sure you limit access to only the group of ESX initiators that need access to the volume(s), and check the box for multi-initiator support, as multiple ESX servers will share access to the same VMFS volume(s)
That’s it for now – express edition of what you need to consider when implementing vSphere on Nimble.