VMware vSphere on Nimble Best Practice Express Edition

This is a vSphere on Nimble best practices express guide – for those that are on-the-go or too busy to read our 20+ page best practices guide, this is a quick and dirty express edition.  Keep in mind this one is specific to VMDK ONLY.  We will have in-guest attached one in the near future.

Here we go, feedback/additional questions welcome.

Let’s start with the how I/O flows from the virtual machine to the Nimble, through the VMware storage stack, down to the NIC, and through the network fabric.

  1. Guest OS (VM running Windows or Linux): at this layer, ensure the OS partition is 4K aligned.  Misaligned OS partition starting offset could cause a write operation to span more than one block on the storage array, causing a single write to incur an extra ‘read-modify-write’ operation.  For optimal performance, ensure this is aligned.  VMware recommends this as a best practice as well.  Read more info on this here
  2. Virtual SCSI HBA: joint recommendation from both VMware and Nimble is to distribute the VMDKs over multiple virtual SCSI HBAs.  For example, if you have a database VM with a dedicated VMDk for OS, DB files, transaction logs, you’d want three vSCSI HBAs (vscsi0, vscsi1 and vscsi2), each with its own VMDK.  This will yield slightly better performance without much effort (just a matter of point and click in vSphere Client)
  3. If using software iSCSI initiator in ESX, ensure you leverage the native MPIO capability of the PSA NMP architecture.  Make sure the iSCSI vmhba is bound to multiple vmkernel ports (if using Nimble 1G model, you’d want 4 vmnics dedicated to iSCSI with one vmkernel port each; if 10G, then two vmnics)

The sw/iscsi vmhba should look something like this:

 

 

 

 

 

Or if you are an esxcli fan, then use the following command to verify:

#esxcli iscsi networkportal list | grep vmk*

The output should look something like this:

vmhba34

Adapter: vmhba34

Vmknic: vmk1

vmhba34

Adapter: vmhba34

Vmknic: vmk2

Ensure the following rule exists in the /etc/vmware/esx.conf file (purpose is to assign PSP_RR as the default multipathing PSP for all Nimble volumes)

/storage/plugin/NMP/config[VMW_SATP_ALUA]/defaultpsp = “VMW_PSP_RR”

OR

# esxcli storage nmp satp rule list  | grep Nimble

Your output should look something like the following:

VMW_SATP_ALUA                Nimble                                                                    user                                             VMW_PSP_RR

If the rule or option line above doesn’t appear, execute the following line in ESXi shell to add it:

#esxcli storage nmp satp rule add –psp=VMW_PSP_RR –satp=VMW_SATP_ALUA –vendor=Nimble

Path Switching Criteria

It’s best to configure iops=0 &bytes=0 settings for PSP_RR path switching criteria.  The value ‘0’ throws off a lot of people.  We have seen very positive performance test results from these settings, and a good number of our customers have reported better IOPS numbers through this tweak as well.  Here’s why we recommend ‘0’ for both iops & bytes (Thanks Jay Wang from Nimble Engineering):

Setting iops=0 and bytes=0 when policy=rr indicates that it disables the path switching based on commands and blocks.  The VMW_PSP_RR will simply round robin the path disregard of io counts or bytes on each path. If the iops is set to 10 (iops=10) when the policy=rr, it means that the PSP will wait until a path accumulated to 10 ios before switching path. Therefore, setting iops=0 and bytes=0 can utilize more paths at the same time than setting iops=10.

4.       ESX NIC dedicated for iSCSI should have flow control enabled

5.       Network Configuration

  • Ensure flow control is enabled on:
    • vmnics that are bound to sw/iscsi initiator
    • switch ports where the vmnics are connected to
    • switch ports where the nimble data interfaces are connected to
  • Turn OFF STP
  • Disable Unicast Storm Control
  • Stack switches method is preferred over ISL

6.       Nimble Side

  • performance policy is set to “VMware ESX or ESX5” (defaults to 4KB block size with cache enabled)
  • choose “vCenter Server” as snapshot synchronization if you have applications that need VSS quiescing
  • make sure you limit access to only the group of ESX initiators that need access to the volume(s), and check the box for multi-initiator support, as multiple ESX servers will share access to the same VMFS volume(s)

That’s it for now – express edition of what you need to consider when implementing vSphere on Nimble.

9 thoughts on “VMware vSphere on Nimble Best Practice Express Edition

  1. you can use the following commands:
    esxcli storage nmp psp roundrobin deviceconfig set –type=iops –iops 0 –device=
    esxcli storage nmp psp roundrobin deviceconfig set –type=bytes –bytes 0 –device=

    or, alternatively, you could use a single command but with the generic psp in place of rr, but it does the job as well:
    esxcli storage nmp psp generic deviceconfig set -d $i –config policy=rr;iops=0;bytes=0

    remember this needs to be set for every device you have provisioned from Nimble.

  2. Thanks, very helpful. Your command was missing a –

    here it is.

    storage nmp psp roundrobin deviceconfig set –type=iops –iops=0 –device=eui.xxxx

  3. Sorry for multiple post

    Nimble KB-000059 only states to do this with IOPS. Does this KB need updating?

    How can customers search for KB articles from there support logon?

    Also from the above KB article here is how you do it for all storage volumes so you dont have to do it so many times.

    Bytes=

    for i in `esxcli storage nmp device list | awk ‘/Nimble iSCSI Disk/{print $7}’ | sed -e ‘s/(//’ -e ‘s/)//’`; do esxcli storage nmp psp roundrobin deviceconfig set –type “bytes” –bytes=0 –device=$i; done

    Iops =

    for i in `esxcli storage nmp device list | awk ‘/NimbleiSCSIDisk/{print $7}’ | sed -e ‘s/(//’ -e ‘s/)//’`; do esxcli storage nmp psp roundrobin deviceconfig set –type “iops” –iops=0 –device=$i; done

    To verify the change

    esxcli storage nmp device list

  4. Hey Wen,

    I was speaking with Vmware storage support and mentioned this.

    Here is what they said:

    Its one or the other. (ions to 0 or bytes to 0)

    If you do iops = 0 then your PSP policy becomes IOPS and the Bytes =0 doesn’t matter because its ignored. And vice versa

    They actually don’t recommend setting the PSP policy to be bytes and setting bytes to 0.

    To see your policy that in effect, bytes or IOPS

    esxcli storage nmp device list

    1. Thanks! I double checked with our engineering team – you are correct – it is either one or the other, and setting one takes precedence over another. I am working on an updated post with our performance validation results. In the meantime, I’ll go ahead and correct any public mentioning of setting both. Thanks much again!

Leave a Reply

Your email address will not be published. Required fields are marked *