In an earlier article VMDirectPath I/O was discussed and how it can be used to virtualize a storage lab.  Based on some of the comments in that article and more information received from Sunny LiYu Zhang, I wanted to add the following points:

• The VT-d specification states that all conventional PCI devices behind a PCIe-to-PCI bridge have to be assigned to the same domain.
• PCIe devices do not have this restriction.

Notes from Sunny:

Please see the VMDirect IO configed on my ESXi 5.0 server.

My mainboard contain “5520 I/O HUB PCI Express root port”. ASIC PI7C21P100 as a translator, bridge PCI-E resource to PCI-X.  Each of this ASIC has two channel. It means with one “PCI Express root port”, only two PCI-X device can be used.

VMDirectPath I/O has a little bit limitation. Example, If you assign some device in the red box selected (09:xx:xx) to one VM:
1: at least one sub-device can be assign to one VM.
2: if one sub-device has been assign to one VM, other can’t be used by other VM. Because of root-tree ASIC (PI7C21P100) has been used by VM DirectI/O.

What this means is that with PCI-e cards you will have much more flexibility but no doubt at a higher cost.  It is important that if you are going to use PCI-X cards, that you take into consideration the limitations that PCI bridges place on VMDirectPath I/O at this time.

After a long delay I have finally made some progress on my VMware lab.  I have successfully installed ESXi 4.1 on both servers, as well as vCenter Server, VMware Site Recovery Manager and many other goodies!  The Synology DS1010+ has been great.  I use it to share out a Z: partition to most of my servers and also have it running FTP.  I created a 100GB iSCSI LUN for my ISO’s and source files and a 2TB iSCSI LUN for my Virtual Machines.  I actually created the original iSCSI LUN as 1TB, but growing it with Synology and vCenter Server was literally just a few clicks.  I even grew the vmdk’s for my vCenter Servers from 40GB to 80GB and then “extended” the file systems in Windows 2008 with just a few simple clicks and no wizardry!

Just about all my VM’s are Thin Provisioned.  I am using Thin Provisioning on my Synology as well as on the VM’s as well.  I can say it takes considerable amount of time to stand up a lab even as small as this one.  Spinning disks, doing Windows updates, initial configurations, took a lot of time.  In the process I have continued to learn so much about Windows, Active Directory, SQL, VMware, EMC, NetApp………I can definitely say the lab is fulfilling its purpose!  Here is a highlight of what I have cooking on the lab right now:

Virtual Machines

You can see many of the usual suspects: Celerra VSA’s (running storage on SSD!), NetApp VSA’s, a couple of Domain Controllers.  My DC’s are Windows 2003 Enterprise 64-bit.  My VCenter Servers are Windows 2008 Enterprise 64-bit, and My SRM Servers are Windows 2008 Enterprise 64-bit running SQL Server 2008 Enterprise 64-bit.  You can also see that I installed a vMA which is great for using with ESXi.  One thing I dislike about vMA is you have to type out the vCenter server and the ESX host in almost every command.  Then you have to authenticate.  What would be nice, is that if you could authenticate to the vCenter, and then change context to like an ESX host, and then just issue commands.  A contextual way of managing, almost how it is done on Cisco ASA’s.

Being a voice guy I also stood up a CUCM 7.1.3 and a Unity Connection 7.1.3.  I did 7.x on purpose, so that I could practice doing the upgrade to 8.x  Definitely learned some lessons there about the importance of having good NTP servers.  When you add a subscriber, CUCM is basically doing database replication, as well as numerous security credential checks.  I learned my lesson with Digital Certificates and the importance of time sources back in studying for my CCIE Storage.  CUCM is very picky, and just as the documentation says you should not slave NTP off the ESX host!!!  Instead its much better to point all UC VM’s to a central time server that slaves off say time.nist.gov, or have them all directly point to time.nist.gov.

Also you will see “Titanium 5.1(2)”.  Yes this is the N7K Simulator.  It took me a long time to get this, and please don’t ask for it as I can’t give it out, but it’s definitely useful to have (even when you have multiple Nexus 7000′s at your disposal as I do).  I have not yet truly fired it up and done some connectivity tests, but I did boot it and already find out some of it’s peculiarities.

Currently I am working my way through Nicholas Weaver’s UBER SRM Video Guide, which is way cool.  He put a lot of work into both the guides and the video.  He has moved so fast with the development of the VSA you have to be careful about which documentation you’re reading because much of it is outdated but you can still learn a lot from it.  I should have the complete SRM lab completed in a few days (If I can get the time).

The only annoyance in my lab so far is an error I get on my ESXi hosts because of the fans.  I get a “Host hardware fan status” error event due to one of the fan ports not being used on the Supermicro X8SIL motherboard.  This is a known issue and really doesn’t seem to have any big problem other than being annoying.  Other’s on the net have talked about how to fix this but I am not a big fan of either solution so will just wait for a BIOS fix.  The reality is, even though the Supermicro X8SIL-F is not on the HCL, it is a fantastic motherboard, with super features, and I still can’t think of a better motherboard to use.  Incidently, I had never heard of the HP Proliant Microservers, but now that I have built my lab with Lian Li V352B, they look awfully similar!  I am sure mine is more powerful and a lot cheaper

I finally got my VMware servers built and my Synology DS-1010+ array up and running and over to the lab.  I had to renumber a few things, IP address all the devices, lace the cables and check everything.  What is done now is less than remarkable.  I have the devices connected, cabled and reachable.  ESXi 4.1 is installed on both boxes and a RAID-5 Volume is built on the Synology

There is still a lot of work to do.  I have been very busy at work and just have not had time to do much in the virtualization lab as I would like.  I spent a day on the weekend doing all the cabling.  I hope to set aside a weekend day to at least install vCenter Server, Celerra VSA’s and NetApp VSA’s.  I plan to run through the entire VMware FastTrack lab book again as well.

But first things first.  I have put a lot of study time (nights what little time I have and weekends) into my EMC SI2 TA studies and I need to just knock those tests out.  I also believe the NAS TA will not be much of an issue.  They have however updated the CLARiiON TA to include FLARE 30, which is not something I have the time to work with that much, so I am concerned about that and also concerned that EMC could make further changes to the TA tests so I need to take them while the knowledge I have is still relatively current!

All of my equipment for my VMware vSphere 4 lab has arrived!  I popped the 5 drives into the Synology DS1010, and one of them is DoA.  So I have already submitted the RMA to Newegg and hopefully in the next week or so I get that fixed.  I didn’t get to work on it that much yet.  What I have done is disassembled the Lian Li case, which is a very well built case, I am impressed.  The motherboard comes out like a platter and makes maintenance very easy.  I popped the X3450 into the motherboard, it was a bit scary locking down the lever on the CPU as it required more pressure than I anticipated.  I snapped the memory in and also put in the 80mm GELID silent fan.  The Seasonic X Series 400W came in all this fancy packaging with velvet like pouches, it looks like a nice power supply it sure was pricey enough.

The fun begins!

Unfortunately I am pretty busy and can’t get this all put together at least until the weekend of January 1.  I am looking forward to it.  I am going to run the Synology DS1010 in RAID6.  I could run RAID5 with no hot spare and that would give me an extra 2TB, but honestly I just don’t trust the workmanship of todays hard drives that much, so I am going with RAID6.

So the plan is to assemble all this stuff, build a volume on the Synology (probably will take 5+ hours to build the RAID group), upgrade all of the firmware of the BIOS, BMC, etc.  I plan to load ESXi 4.1 on some fast 4GB USB thumb drives, which will install nicely on the internal USB slot on the Supermicro X8SIL-F motherboard.  So far I feel really good about my equipment selection.  I am glad I spent the time doing a lot of the research.  This lab is definitely not cheap, but I plan to get a lot of use out of it.  In addition to just about every VMware application, Nexus 1000, etc, I plan to run Dynamips, a full voice lab, dual Celerra VSA’s, Dual NetApp VSA’s, and a ton of other stuff………I will definitely get my money out of this.

There is some re-tooling required on my current lab switch to move some things around to make room for the 1GB interfaces needed for these ESXi boxes (iSCSI, vMotion, etc).

One of the last things I purchased for my VMware lab is the memory for my Supermicro X8SIL-F motherboard.  This was mostly due to the fact that they make the memory information regarding the motherboard a bit confusing.  At first I thought I would get 1333mhz memory.  Then I realized that 1333mhz memory is a pricey premium.  It makes much more sense to go with 1066mhz memory.  Supermicro has two specific pages that are helpful regarding memory selection for this motherboard.  The first is the Tested Memory List which is linked off the motherboards homepage.  This allows you to select either 1066mhz or 1333mhz unbuffered or registered memory and select various sizes.  More importantly however is the following chart which you can on the motherboard page under the System Memory section:

One of the key takeaways from the above is that you want to populate the memory in pairs to take advantage of interleaved memory.  There are two types of memory supported, Unbuffered (UDIMM) and Registered (RDIMM).  If you want to get the most amount of memory on the board, us 2Gb x8 DRAM’s.  You can check with the manufacturer to see if the memory meets specifications.  I personally went with Registered Dual Rank memory.  I went with Crucial CT51272BB1067, and got a decent price at Buy.com.  You will probably find that a lot of the memory out there is Quad Rank memory.  This includes the recommended memory list when you search off Crucial’s website.  Take a look at the chart below:

The issue is that if you use 1066mhz RDIMMs and you populate 2 DIMMs per channel (to get a total of 16GB’s), you will dumb down your motherboard to an 800mhz memory bus!  This is not obvious and I have seen a lot of people use the 1066mhz Quad Rank memory on this board.  This is why I opted for the Dual Rank memory.  If single rank memory exists it’s likely expensive as the chips are more costly. Just like with compromising with 1066mhz memory over 1333mhz memory I decided to go for value.

I have not yet received the memory or had time to test, but I will post a follow-up.  The only advantage to going with Quad Rank memory is that you could populate the board with 32GB of memory, but at the sacrifice of memory speed.  16GB is plenty, as I plan to run two ESXi hosts.