Archive for March, 2012

Setting up a VMware ESXi 4.1 Environment with a Dell PS4000x SAN using Software iSCSI, Multiple Paths and Jumbo Frames

Posted in Blog, Tech, Tutorial on March 8th, 2012 by Matt – Be the first to comment

Yea, that's a jumble of a title. This will be a brain dump of all the configuration I've done on our SAN environment. Looking back it seems so easy to configure, and I doubt that it would take me more than an hour or two to set up a whole new SAN environment from scratch. But since this was my first time, it took me a lot of reading and research. Here's a dump of what I've done:

Our environment:

Two Dell r710 servers: Each with four onboard NIC ports and two add-in Broadcom 5709s with two ports each. Each server has two internal hard disks in a RAID 1 (mirrored) array to boot ESXi off of. Booting directly from the SAN is an option I will explore at a later date. In our environment I already had one server hosting a few VMs off of internal storage including a VSphere server. This process was an upgrade to add a SAN and another server. A note: If I could do this over again I would NOT have purchased Broadcom 5709s in our servers. The main reason I got them was to offload all iSCSI traffic to the cards and use the hardware iSCSI initiators in VMware. Apparently 5709s support offloading of iSCSI but not with jumbo frames. After reading this article which benchmarks VMware with hardware iSCSI without jumbo frames against software iSCSI with jumbo frames I decided I would use the software iSCSI initiators.

One Dell PS4000x SAN: The SAN has two controllers each with three ports. One port is for management and is on our LAN. The other two are for SAN traffic. The controllers work in a Active/Standby configuration. The secondary controller simply mirrors the primary controllers configuration and sits idle until the primary controller fails, then the secondary takes over.

Two Cisco 3750G Switches: The pair are in a stacked configuration using Cisco's StackWise interconnects. These switches will ONLY be used for the SAN. Doing so saves a lot of VLAN configuration. At the moment I didn't even set VLAN information.

My process:

The following is how I set all this up. I used tons of various articles from official configuration guides to forum and blog posts. I'll try to link to the actual documents used wherever possible.

I rack mounted everything. I decided to go with the switches top most. Below that (with some space) our two servers. Below that (with some space) our SAN, Below that (with some space) two UPSs.

Install ESXi:

As stated earlier my environment already had one ESXi server running with VMs hosted out of internal storage including a VM hosting VSphere. I simply installed ESXi on the new server, wired it into the SAN/LAN and added it to VSphere. From there I managed everything. If I were doing this fresh (without a VSphere install) I'd install ESXi on both servers, connect all the SAN/LAN connections then connect to each one via a VSphere client  independently to configure the SAN connections then create a VSphere VM to manage them both.

Network Connections:

For redundancy each server should have at least two completely independent network routes to the SAN and LAN. In my situation each of my Dell servers have three independent network controllers. The two Broadcom cards and the onboard units. On each server I'm using one port on each of the two Broadcom cards for SAN connectivity. The SAN has two controller cards. Each card has two SAN ports and a management port. One SAN port from each card is connected to one switch and the Management ports are connected to the LAN.

Should any one network card on a server fail the sever will still have SAN and LAN access. Should any one of the switches fail the servers will have SAN and LAN access. Should any one of the SAN controllers fail the servers will have SAN and LAN access.

IP addresses for each device are in the range. The Dell SAN requires three IP address. One for each network port (.1 and .2) on a member and a group IP address (.10). The servers have two SAN IPs each (.3 .4 and .5 .6).

SAN Topology Diagram

SAN Topology Diagram: Blue is SAN connections. Green is LAN connections.

Configuring the Cisco 3750g Switches:

Following the Appendix F of the Dell EqualLogic Configuration Guide "Cisco IOS based Switch Configuration" I enabled portfast, enabled flow control, disabled unicast storm, and enabled jumbo frames. Once again I'm setting these options on all ports because these are dedicated SAN only switches. Also because my switches are stacked I can configure them both from the same connection. Your situation may differ.

Enabled portfast on all ports:

Switch# config terminal
Switch(config)# spanning-tree portfast default
Switch(config)# end
Switch# copy running-config startup-config

Enabled flow control:

Switch> enable
Switch# configure terminal
Switch(config)# interface range gi1/0/1 - 24
Switch(config-if)# flowcontrol receive desired
Switch(config-if)# exit
Switch(config)# interface range gi2/0/1 - 24
Switch(config-if)# flowcontrol receive desired
Switch(config-if)# exit
Switch(config)# exit
Switch# copy running-config startup-config

Disabled unicast storm control:

Switch> enable
Switch# configure terminal
Switch(config)# interface range gi1/0/1 - 24
Switch(config-if)# no storm-control unicast level
Switch(config-if)# exit
Switch(config)# interface range gi2/0/1 - 24
Switch(config-if)# no storm-control unicast level
Switch(config-if)# exit
Switch(config)# exit
Switch# copy running-config startup-config

Enabled Jumbo Frames (notice this requires a reload... don't do this if you're on production equipment!):

Switch> enable
Switch# config terminal
Switch(config)# system mtu jumbo 9000
Switch(config)# exit
Switch# copy running-config startup-config
Switch# reload

Then I verified all my settings:

Switch# show spanning-tree interface gi1/0/1
Switch# show flowcontrol interface gi1/0/1
Switch# show storm-control gi1/0/1 unicast
Switch# configure t
Switch(config)# show system mtu
Switch(config)# exit
Switch# show interface gigabitethernet1/0/1

The last command is to confirm and MTU of 9000 on the port. Repeat the previous commands on all ports to ensure they have the necessary settings.

Initialize the SAN:

First time use of the PS Series SANs require you to connect to the unit with a serial cable and run a initial configuration wizard. The PS4000 Installation and Setup Manual covers all these steps in more detail.

Hook up the serial cable and open a connection to whatever COM port you used. I used Putty for this. Hit enter and you should get a prompt. Type "setup" to start the configuration wizard. First time setup uses "grpadmin" for both the username and password. From here you'll br prompted to set up a member name, group name, IP address, password and a few other things. After this process completes you can connect in a web browser to the units management interface via the IP address you set. From there you can setup the RAID level and volumes. At this time I've chosen RAID 5, but since I'm currently not production with this unit I plan on reformatting it to RAID 6 and RAID 50 to test performance levels. A note: Once you've chosen a particular RAID level you can only change to similar RAID levels.

RAID-10: Can only be changed to RAID-50, RAID-6, or RAID-5
RAID-50: Can only be changed to RAID-6 or RAID-5
RAID-6: can only be changed to RAID-5

Should you want to make an unsupported RAID level change you need to pretty much start over with the SAN member. In a one member environment this means also starting over with the group. See this post by an EqualLogic employee.

After completing the initial setup, I connected to the SAN's LAN IP address and created a temporary volume. Having a volume ready to connect to makes it easier to know you've set up your ESXi iSCSI connections correctly.

Install the Dell MEM (Multipath Extension Module):

This module is an alternative to the built in VMware iSCSI path selectors. It's configured to better support the PS series controllers. Installation consists of downloading the Dell Multipath Extension Module then using one of three ways to install it to your ESX hosts. All of the following is taken from the Installation and User Guide that comes in the MEM download package. Note: No matter which method you choose your ESX host will need to be in maintenance mode.

I chose to install the module via the vSphere CLI that's available here. After installing the vSphere CLI, I extracted the contents of the Dell MEM download to the /bin directory in the CLI install. Then after launching the CLI and changing directory to the /bin directory ("cd bin") I ran the following:

esxcli --server=esxi1 software vib install --depot

Then repeated the command for my other ESX server and took both servers out of maintenance mode.

Configuring ESXi SAN Connections with Jumbo Frames:

Most of the following I got from the VMware iSCSI SAN Configuration Guide however that guide doesn't cover jumbo frames. For Jumbo frames replace the commands in the previous guide with the commands from the Knowledge Base article: iSCSI and Jumbo Frames configuration on ESX/ESXi.

All of the guides say to do the steps straight from the terminal. However all of the steps can be accomplished remotely using the VMware CLI. The VMware CLI is a series of Perl scripts. According to the documentation you can simply type the scripts name then switches to launch them. However on my computer I needed to change to the /bin subdirectory of the CLI install (where the scripts are actually located) and call each script with its .pl extension. Since the guides commands all assume your at the console the "--server=servername" switch will needed to be added to all commands in the guides.

Create a vSwitch: --server=esxi1 -a vSwitch-iSCSI

Change the MTU to 9000 on the vSwitch: --server=esxi1 -m 9000 vSwitch-iSCSI

Create two VM kernal port groups for iSCSI traffic (because I have two SAN connections on this server): --server=esxi1 -A iSCSI1 vSwitch-iSCSI --server=esxi1 -A iSCSI2 vSwitch-iSCSI

Create two VM kernal connections (one for each port group) with jumbo frames: --server=esxi1 -a -i -n -m 9000 iSCSI1 --server=esxi1 -a -i -n -m 9000 iSCSI2

Make a note of the port names assigned to these connections when you create them. The port name will be in the format of "vmk#". You can view the port name from the networking section under the configuration tab. In my case iSCSI1 was vmk1 and iSCSI2 was vmk2

Each VMKernal Port needs to be assigned to one physical adapter. By default since all ports on a switch can use all adapters on the switch. We need to change each port to ONLY use a particular adapter. To do so select Properties on the vSwitch we created. Select the first iSCSI port listed and hit the Edit button. On the NIC Teaming tab move all but one of the vmnic adapters to the Unused Adapters section. Repeat the process for the other iSCSI ports on the switch so that each iSCSI port is bound to only one Active Adapter.

Failover Override

Each iSCSI port must be assigned only one Active Adapter

Enable the Software iSCSI adapter now if you haven't. Log in to vSphere client, select your server, configuration tab, Storage Adapters, select the Software iSCSI Adapter (and make a note of its port name, e.g. vmhba#), click configure, select Enable. In my case the Software iSCSI Adapter was vmhba41.

Bind the iSCSI ports to the iSCSI adapters:

esxcli --server=esxi1 swiscsi nic add -n vmk1 -d vmhba41
esxcli --server=esxi1 swiscsi nic add -n vmk2 -d vmhba41

The last step is to set up our discovery address and scan the bus.  Under the Storage Adapter section of the Configuration tab select the Software iSCSI adapter, select Properties, select the Dynamic Discovery tab, select the Add button and input the IP address of our Dell SAN group, not the individual member, but the group. In our case the group IP was Select OK. When you hit close on the iSCSI Initiator Properties window you will be prompted to scan the bus. Select yes. If everything worked out correctly a new entry should populate under the iSCSI Storage Adapter details. Also if we navigate to the the Storage section we should see the volume we created earlier available for use.

An alternative to running each of the previous commands individually is to use the script provided by Dell. We copied it over earlier to our vSphere CLI installation bin folder from the Dell MEM installation. Using the script consists of launching the vSphere CLI, changing into the /bin directory (only because thats where we extracted it earlier), and running the following: --configure --server=esxi1

The script will then prompt your for all the parameters we used earlier and setup everything automatically. It makes setup very easy. Have a look:

Dell's Setup Script

Dell's Setup Script - Click to Enlarge

Confirming Jumbo Frames are Being Used:

To confirm you are in fact using Jumbo frames and multiple paths you can connect to the Dell SAN's management interface via web browser, login, and go to the Monitoring section and select the Events list. You should see "iSCSI log on to target... stuff here... successfull, using Jumbo frame length."


Now you're up and running. If this was a fresh install I'd create myself a vSphere server VM and go to town.