In an earlier article VMDirectPath I/O was discussed and how it can be used to virtualize a storage lab.  Based on some of the comments in that article and more information received from Sunny LiYu Zhang, I wanted to add the following points:

• The VT-d specification states that all conventional PCI devices behind a PCIe-to-PCI bridge have to be assigned to the same domain.
• PCIe devices do not have this restriction.

Notes from Sunny:

Please see the VMDirect IO configed on my ESXi 5.0 server.

My mainboard contain “5520 I/O HUB PCI Express root port”. ASIC PI7C21P100 as a translator, bridge PCI-E resource to PCI-X.  Each of this ASIC has two channel. It means with one “PCI Express root port”, only two PCI-X device can be used.

VMDirectPath I/O has a little bit limitation. Example, If you assign some device in the red box selected (09:xx:xx) to one VM:
1: at least one sub-device can be assign to one VM.
2: if one sub-device has been assign to one VM, other can’t be used by other VM. Because of root-tree ASIC (PI7C21P100) has been used by VM DirectI/O.

What this means is that with PCI-e cards you will have much more flexibility but no doubt at a higher cost.  It is important that if you are going to use PCI-X cards, that you take into consideration the limitations that PCI bridges place on VMDirectPath I/O at this time.

I am running an ASA 7.2(5)/ASDM 5.2(5), this is the ASA at my house.  And I am not sure when it started, but I cannot get traceroutes to work from my mac to the Internet.  I can traceroute on my LAN, but not going through the ASA, I just get * * *.  Here are the related parts of my ASA config:

So I have found a bit about why this wasn’t working, and interested to get some opinions from the ASA gurus on here:

The ASA is sending the time-exceeded’s to 172.16.5.30, which is my Synology:

08 packets captured

   1: 01:52:42.269090 802.1Q vlan#1 P0 172.16.5.229 > 8.8.8.8: icmp: echo request 

   2: 01:52:42.309355 802.1Q vlan#1 P0 64.233.174.2 > 172.16.5.30: icmp: time exceeded in-transit 

   3: 01:52:47.268159 802.1Q vlan#1 P0 172.16.5.229 > 8.8.8.8: icmp: echo request 

   4: 01:52:47.311141 802.1Q vlan#1 P0 66.249.94.22 > 172.16.5.30: icmp: time exceeded in-transit 

   5: 01:52:52.268906 802.1Q vlan#1 P0 172.16.5.229 > 8.8.8.8: icmp: echo request 

   6: 01:52:52.313368 802.1Q vlan#1 P0 66.249.94.22 > 172.16.5.30: icmp: time exceeded in-transit 

My ASA is very basic, I just PAT everything out the outside interface.  however I do have one static:

static (inside,outside) interface 172.16.5.30 netmask 255.255.255.255 dns 

So my thought with this, is anything coming into the ASA, that does not belong to some other xlate would goto 172.16.5.30.  there are maybe a handfull of ports I send its way via the outside ACL:

access-list outsideIn extended permit tcp any interface outside eq www 

access-list outsideIn extended permit tcp any interface outside eq 5001 

access-list outsideIn extended permit tcp any interface outside eq 5432 

access-list outsideIn extended permit udp any interface outside eq 5432 

access-list outsideIn extended permit tcp any interface outside eq 6881 

access-list outsideIn extended permit tcp any interface outside eq 9997 

access-list outsideIn extended permit tcp any interface outside eq 9998 

access-list outsideIn extended permit tcp any interface outside eq 9999 

access-list outsideIn extended permit tcp any interface outside eq ftp-data 

access-list outsideIn extended permit tcp any interface outside eq ftp 

access-list outsideIn extended permit tcp any interface outside eq 5006 

access-list outsideIn extended permit tcp any interface outside eq 7001 

access-list outsideIn extended permit tcp any interface outside eq 6900 

But it should not take a time-exceeded packet and send it to 172.16.5.30.  The ASA should be smart enough to know that it was in response to a UDP packet sent from 172.16.5.229 (my laptop).  It seems like it could be a bug.  I only have 1 IP address, it’s dynamically assigned, so I static my Synology to it and I also NAT everything else to it:

global (outside) 1 interface

nat (inside) 1 0.0.0.0 0.0.0.0

I am pretty sure this is allowed.  The only other way I could do it would be to make a ton of “statics” that were port based, one for each of the ports in the ACL……….just would be more administrative work, but I wonder if one way or the other is truly correct.  You can see the xlate is clearly there:

home-asa# show xlate | inc ICMP

4 in use, 234 most used

Global 174.61.8.28 Local 172.16.5.30

PAT Global 174.61.8.28(28476) Local 172.16.5.229 ICMP id 41382 

As a test, I got rid of my single “catch all” static and replaced it with static PAT’s:

global (outside) 1 interface

nat (inside) 1 0.0.0.0 0.0.0.0

static (inside,outside) tcp interface www 172.16.5.30 www netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 5001 172.16.5.30 5001 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 5432 172.16.5.30 5432 netmask 255.255.255.255  dns 

static (inside,outside) udp interface 5432 172.16.5.30 5432 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 6881 172.16.5.30 6881 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 9997 172.16.5.30 9997 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 9998 172.16.5.30 9998 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 9999 172.16.5.30 9999 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface ftp-data 172.16.5.30 ftp-data netmask 255.255.255.255  dns 

static (inside,outside) tcp interface ftp 172.16.5.30 ftp netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 5006 172.16.5.30 5006 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 7001 172.16.5.30 7001 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 6900 172.16.5.30 6900 netmask 255.255.255.255  dns 

This “fixed” the issue.  But now I remember why I did the single static.  DNS Doctoring doesn’t work with static PAT, despite that it allows you to configure the “dns” keyword.  The only way to actually have it work is with a static NAT.  So I guess I am going to just lose the ability to have traceroute (and god know what else) working, in exchange for working DNS Doctoring.  I never noticed anything wasn’t working until now anyways.

Still, I call it a bug, because my laptop should create a XLATE, that the ASA can tie to the time-exceeded coming back…….I don’t know, it just doesn’t seem like the normal way an ASA would work.  At least now its working, I will just have to add a static entry every time I open a port to match my ACL.

Here you can see the Wireshark capture of an ICMP packet going from 8.8.8.8 to the outside of the ASA:

 You can see that under the ICMP section of the packet, there is a IPv4 section, that shows the packet came from the ASA (was NAT’ed) and the destination was 8.8.8.8.  This capture was made on the “outside” interface of the ASA.

Similarly, you can see below is a Wireshark capture of the same packet, captured on the “inside” interface of the ASA:

In this capture you can see that the IPv4 header shows a destination of 172.16.5.30, which is my Synology which has the “catch all” static NAT.  This is incorrect, this is in response to a traceroute ran on my Mac, which his 172.16.5.229.  You can see in the ICMP header, there is a IPv4 subsection that shows correctly that the original source of the packet was my Mac 172.16.5.229.  So the ASA obviously had the information maintained in it’s state tables, but did not correctly fixup and route this packet.

If anyone know’s if this is a configuration problem or indeed a bug please let me know.  With only a single IP address to share at my house, and the need for so many ports to goto my Synology, and the need for DNS Doctoring, I figured what I was originally doing was quite fine, and in fact everything that I could tell seemed to be working except traceroute.

Back when I built my storage lab, I had to build separate physical servers (WIN1, WIN2, WIN3 and MGMT) each with their own fibre channel cards and I/O paths.  Today, VMware has VMDirectPath I/O, which allows you to take multiple I/O resources within a server and tie them to various guests.  What this means is that you can put say 3 guests into a physical ESXi server, 3 four-port FC cards in the server, and then map specific FC ports to specific guests, allowing you to virtualize your FC servers in a storage lab.

I looked at doing this myself when VMDirectPath I/O came out.  However, VMDirectPath I/O will not work on just any server.  You need specific hardware requirements, mainly the CPU has to support it (VT-d) and the motherboard chipset.  The obstacle this presents is that you need modern server hardware, which means its going to take newer FC cards (PCI-e, etc), and so you will pay more for your FC cards as well.  The advantage is you will have one single server that can do all your FC initiator needs, saving you space, power and cooling.

When I built my lab, I used cheap HP DL360G2/G3′s, and cheap FC cards that were about $20/each on eBay.  However, some of you may have modern servers in  your labs (now of days I do too), and if you have VMDirectPath I/O, then virtualizing your initiator servers is rather low hanging fruit.

One person has done just this, Sunny LiYu Zhang.  Not only did he virtualize the servers in the lab, but he virtualized the storage as well!

First let’s look at the servers, here is what Sunny is using in his single physical server which is virtualizing multiple servers using fibre channel.

Mainboard: Supermicro X8DA6
CPU: Intel Xeon E5606 * 1
Memory: 16GB DDR3 RAM with ECC
Storage: 500G SATA HDD
Fibre Channel Cards: Qlogic QLA2344 * 2 (Win1,Win2)
Qlogic QLA2532 * 1 (Win3)

The software being used is ESXi stand alone without vCenter.

For the storage side of things, most people assembling a lab will use four storage arrays, usually JBOD’s.  Some who have read my article on partitioning a JBOD can get by with two storage arrays.  The reality is, any arrays will typically do so long as they present a FC loop either public or private.  With VMDirectPath I/O you can get rid of all storage arrays.  You can basically do similar as to what is described above for virtualizing the servers.  The challenge is, a server with a FC card in it does not make a storage array.  That is your normal O/S running Linux, Windows, etc. does not present a loop of hard drives with WWN’s out its FC port as FC targets.  You need special software to do this.  In fact, the software even makes it better, as you do not need 24 hard drives each with a WWN, the software can virtualize the hard disks and WWN’s, so its essentially using files or file space and presenting them as hard drives with WWN’s.  This is a huge plus, as typically in a CCIE storage lab you build for yourself you will have 4 JBOD’s each with 6 drives, which means 24 drives total.

Building a virtual storage platform with VMDirectPath I/O will save you lots of space, power and cooling, it seems like a no brainer and is even more compelling then virtualizing the servers.  The issue is, the software to do this magic is not cheap.  One such piece of software, the software Sunny chose was SanBlaze from www.sanblaze.com.  This is an amazing piece of software for doing FC Target Emulation.  However the software is not cheap. It’s possible you may have it where you work or you may be able to get ahold of a demo license as Sunny did, which will eventually expire, but at least you can try it out to see if it suits your needs.

There is a SourceForge project which looks promising that may be able to offer SCSI Target Emulation as well called SCST.  A google search of “SCSI Target Emulation” will turn up multiple companies and products, almost all of which cost money.  Other companies that may offer such software are www.open-e.com or www.datacore.com.

Sunny ran his SanBlaze on a Cisco C210 M2 server with 2 LSI7404EP-LC fibre channel cards.  Obviously you will need to check the product requirements for whatever Target Emulation software you choose to make sure the fibre channel cards you choose are compatible.

So in closing this has allowed Sunny to use just two servers, one to handle the SCSI initiators and one to handle the SCSI targets.  The secret sauce is a combination of VMDirectPath I/O and SCSI Target Emulation software.  If you are successfully using any VMDirectPath I/O in your lab, please chime in.  I am especially interested in hearing any success stories using open source SCSI Target Emulation such as SCST.

Thanks to Sunny Liyu Zhang for letting me talk a bit about his lab.  Sunny is a Customer Support Engineer with Cisco Systems in Beijing, China.  He should be taking his lab attempt in the next few months and I wish him the best of luck!

With the new year, comes new resolutions.  For me, there are many: Losing weight, completing my first year or graduate school, learn Hebrew, journal more, and read more.  With reading, I have taken on the challenge of Charles W. Eliot, L.L.D., to read 15 minutes a day from the Harvard Classics.  I am using scanned copies of the volumes which can be found online in the public domain.  I am also continuing to search for a complete 51 volume set so that I have the original texts.

The first reading assignment for the year is from the Autobiography of Benjamin Franklin.  It’s quite fitting for the start of a new year, as the section is “Franklin’s Advice for the New Year”.  Here are the virtues Franklin prescribes to be taken to habit:

1. TEMPERANCE.
Eat not to dullness; drink not to elevation.

2. SILENCE.
Speak not but what may benefit others or yourself; avoid trifling conversation.

3. ORDER.
Let all your things have their places; let each part of your business have its time.

4. RESOLUTION.
Resolve to perform what you ought; perform without fail what you resolve.

5. FRUGALITY.
Make no expense but to do good to others or yourself; i. e., waste nothing.

6. INDUSTRY.
Lose no time; be always employ’d in something useful; cut off all unnecessary actions.

7. SINCERITY.
Use no hurtful deceit; think innocently and justly, and, if you speak, speak accordingly.

8. JUSTICE.
Wrong none by doing injuries, or omitting the benefits that are your duty.

9. MODERATION.
Avoid extreams; forbear resenting injuries so much as you think they deserve.

10. CLEANLINESS.
Tolerate no uncleanliness in body, cloaths, or habitation.

11. TRANQUILLITY.
Be not disturbed at trifles, or at accidents common or unavoidable.

12. CHASTITY.
Rarely use venery but for health or offspring, never to dulness, weakness, or the injury of your own or another’s peace or reputation.

13. HUMILITY.
Imitate Jesus and Socrates.