Back when I built my storage lab, I had to build separate physical servers (WIN1, WIN2, WIN3 and MGMT) each with their own fibre channel cards and I/O paths.  Today, VMware has VMDirectPath I/O, which allows you to take multiple I/O resources within a server and tie them to various guests.  What this means is that you can put say 3 guests into a physical ESXi server, 3 four-port FC cards in the server, and then map specific FC ports to specific guests, allowing you to virtualize your FC servers in a storage lab.

I looked at doing this myself when VMDirectPath I/O came out.  However, VMDirectPath I/O will not work on just any server.  You need specific hardware requirements, mainly the CPU has to support it (VT-d) and the motherboard chipset.  The obstacle this presents is that you need modern server hardware, which means its going to take newer FC cards (PCI-e, etc), and so you will pay more for your FC cards as well.  The advantage is you will have one single server that can do all your FC initiator needs, saving you space, power and cooling.

When I built my lab, I used cheap HP DL360G2/G3′s, and cheap FC cards that were about $20/each on eBay.  However, some of you may have modern servers in  your labs (now of days I do too), and if you have VMDirectPath I/O, then virtualizing your initiator servers is rather low hanging fruit.

One person has done just this, Sunny LiYu Zhang.  Not only did he virtualize the servers in the lab, but he virtualized the storage as well!

First let’s look at the servers, here is what Sunny is using in his single physical server which is virtualizing multiple servers using fibre channel.

Mainboard: Supermicro X8DA6
CPU: Intel Xeon E5606 * 1
Memory: 16GB DDR3 RAM with ECC
Storage: 500G SATA HDD
Fibre Channel Cards: Qlogic QLA2344 * 2 (Win1,Win2)
Qlogic QLA2532 * 1 (Win3)

The software being used is ESXi stand alone without vCenter.

For the storage side of things, most people assembling a lab will use four storage arrays, usually JBOD’s.  Some who have read my article on partitioning a JBOD can get by with two storage arrays.  The reality is, any arrays will typically do so long as they present a FC loop either public or private.  With VMDirectPath I/O you can get rid of all storage arrays.  You can basically do similar as to what is described above for virtualizing the servers.  The challenge is, a server with a FC card in it does not make a storage array.  That is your normal O/S running Linux, Windows, etc. does not present a loop of hard drives with WWN’s out its FC port as FC targets.  You need special software to do this.  In fact, the software even makes it better, as you do not need 24 hard drives each with a WWN, the software can virtualize the hard disks and WWN’s, so its essentially using files or file space and presenting them as hard drives with WWN’s.  This is a huge plus, as typically in a CCIE storage lab you build for yourself you will have 4 JBOD’s each with 6 drives, which means 24 drives total.

Building a virtual storage platform with VMDirectPath I/O will save you lots of space, power and cooling, it seems like a no brainer and is even more compelling then virtualizing the servers.  The issue is, the software to do this magic is not cheap.  One such piece of software, the software Sunny chose was SanBlaze from www.sanblaze.com.  This is an amazing piece of software for doing FC Target Emulation.  However the software is not cheap. It’s possible you may have it where you work or you may be able to get ahold of a demo license as Sunny did, which will eventually expire, but at least you can try it out to see if it suits your needs.

There is a SourceForge project which looks promising that may be able to offer SCSI Target Emulation as well called SCST.  A google search of “SCSI Target Emulation” will turn up multiple companies and products, almost all of which cost money.  Other companies that may offer such software are www.open-e.com or www.datacore.com.

Sunny ran his SanBlaze on a Cisco C210 M2 server with 2 LSI7404EP-LC fibre channel cards.  Obviously you will need to check the product requirements for whatever Target Emulation software you choose to make sure the fibre channel cards you choose are compatible.

So in closing this has allowed Sunny to use just two servers, one to handle the SCSI initiators and one to handle the SCSI targets.  The secret sauce is a combination of VMDirectPath I/O and SCSI Target Emulation software.  If you are successfully using any VMDirectPath I/O in your lab, please chime in.  I am especially interested in hearing any success stories using open source SCSI Target Emulation such as SCST.

Thanks to Sunny Liyu Zhang for letting me talk a bit about his lab.  Sunny is a Customer Support Engineer with Cisco Systems in Beijing, China.  He should be taking his lab attempt in the next few months and I wish him the best of luck!

With the new year, comes new resolutions.  For me, there are many: Losing weight, completing my first year or graduate school, learn Hebrew, journal more, and read more.  With reading, I have taken on the challenge of Charles W. Eliot, L.L.D., to read 15 minutes a day from the Harvard Classics.  I am using scanned copies of the volumes which can be found online in the public domain.  I am also continuing to search for a complete 51 volume set so that I have the original texts.

The first reading assignment for the year is from the Autobiography of Benjamin Franklin.  It’s quite fitting for the start of a new year, as the section is “Franklin’s Advice for the New Year”.  Here are the virtues Franklin prescribes to be taken to habit:

1. TEMPERANCE.
Eat not to dullness; drink not to elevation.

2. SILENCE.
Speak not but what may benefit others or yourself; avoid trifling conversation.

3. ORDER.
Let all your things have their places; let each part of your business have its time.

4. RESOLUTION.
Resolve to perform what you ought; perform without fail what you resolve.

5. FRUGALITY.
Make no expense but to do good to others or yourself; i. e., waste nothing.

6. INDUSTRY.
Lose no time; be always employ’d in something useful; cut off all unnecessary actions.

7. SINCERITY.
Use no hurtful deceit; think innocently and justly, and, if you speak, speak accordingly.

8. JUSTICE.
Wrong none by doing injuries, or omitting the benefits that are your duty.

9. MODERATION.
Avoid extreams; forbear resenting injuries so much as you think they deserve.

10. CLEANLINESS.
Tolerate no uncleanliness in body, cloaths, or habitation.

11. TRANQUILLITY.
Be not disturbed at trifles, or at accidents common or unavoidable.

12. CHASTITY.
Rarely use venery but for health or offspring, never to dulness, weakness, or the injury of your own or another’s peace or reputation.

13. HUMILITY.
Imitate Jesus and Socrates.

Learned about a new feature today, thanks to a co-worker, that I never knew about…..IOS quiet-mode.  The command reference explains it fairly well starting with the “login block-for” command here.  Basically what this allows you to do, is define a maximum login attempts made on the VTY.  When these attempts are exceeded, IOS will enter quiet mode.  IOS will put an auto-generated access-class in called sl_def_acl that will prevent telnet, ssh and www access.  You can also define your own ACL to go into effect when IOS enters quiet mode.  Consider the following:

ip access-list extended sshAccess
 permit tcp 10.0.0.0 0.255.255.255 eq 22 log
 permit tcp 172.16.0.0 0.0.31.255 eq 22 log
 permit tcp 192.168.0.0 0.0.255.255 eq 22 log
 deny ip any any log

ip access-list extended quiet-sshAccess
 permit tcp 10.1.1.1 0.0.0.0 eq 22 log
 deny ip any any log

login block-for 360 attempts 6 within 100
login quiet-mode access-class quiet-sshAccess

The login block-for command has to be entered before you enter the login quiet-mode command.  In this example, you would apply the ACL sshAccess to your VTY as normal using an access-class command.  Then, you enter the above login commands and quiet-sshAccess ACL, and after 6 attempts within 100 seconds are made, IOS will enter quiet mode for 360 seconds. During the quiet-mode the quiet-sshAccess ACL is in place, so only host 10.1.1.1 can ssh to the device.  It’s very simple.  With today’s brute force hacking botnets, something like this is very useful.  Another good command to combine with it is the login delay command which you can read about here.  This allows you to put a delay between login attempts to further hamper brute force login attempts.

When EMC re-did their group/role mappings for Celerra Administrative Roles, back in 2008 or so (When DART 5.6 was released), they had a chance to create a new set of group/roles that totally make sense.  And for the most part they do, but does anyone else see something wrong with this picture?

So with the security in Celerra, Roles and Groups have a One to One relationship.  You can see that the fullnas group is mapped to the Nasadmin role.  The nasadmin group is mapped to the Operator role.  ?!?!?!??!  To me, it would have made a lot more sense to create an operator group and map the Operator role to that.  Maybe I am just being a bit OCD about this, but it just bothers me that the entire scheme looks relatively clean, and they had an opportunity to make it just so perfect, but left in this confusing point.

Now, why are some of the Role Names capitalized and others not?  I have no idea.  But I must say this.  EMC Education does a hell of a job cranking out a great amount of material.  So sometimes typo’s exist and things are actually correct(ed) in the OS, and other times they are just the messenger and have nothing to do with the design of the system (actually, that’s probably most cases).

I have been impressed in watching the advancements of the Celerra from a few years ago until now morphing into the VNX.  Things have always improved greatly.  I am not a heavy user of RBAC, simply because I look at it more like there are two options:  Those that should have access and those who should not .  Obviously we design things for customers based on their requirements but I like to have an educated group who have access, and then not have to worry about those that don’t.  When I say educated, I don’t mean they are the Grand Master at all things, Celerra in this case, but that they understand enough to know there are things they should touch and things they should not.

If you don’t know much about Celerra, you shouldn’t be doing something like following commands that start off with you doing “export NAS_DB_DEBUG=1″.

NetApp DS14mk2

One of the questions I am asked many times is about what type of disk storage (JBOD) can be used for CCIE Storage studies.  There are many that can be used.  I prefer to use something that is public loop and has SFP interfaces.  I also prefer to use something that supports multiple loops and allows the partitioning of those loops so you can effectively have one box do the job of two.

I have written about how I like the Xyratex RS-1600-FC2 boxes.  These are in fact what NetApp OEM’ed for the DS14mk2 shelves and you can find them on ebay, craigslist, and other places.  Now I must caveat that I have not used one of these in this capacity myself, however, it’s in fact a JBOD so there is no reason it should not work.

One of the things that the NetApp Filers do however is they do write some custom information to the first sector of the disks.  You will want to zap that information.  Easiest way is to just attach the shelf to a linux fibre card or you could do this with Windows if that is what your comfortable with.  Using Linux, follow the guide here:

http://cuddletech.com/articles/netapp/netapp-evms.html

Which explains nicely how to setup a basic Linux FC card, zap the first part of the disk using dd, and also has some great information on  using the Linux LVM.

I am a huge fan of Linux, and I definitely like it more than Windows.  However, I must say that during my lab studies all my initiators were Windows.  This just makes sense to me, as the windows has easy to download iSCSI initiators, FC stacks/drivers, tools, RADIUS and TACACS tools, etc.  Sure, you can track all this down on Linux but likely the books your studying with are going to assume you are using Windows.  Now, that aside, I do prefer to do low level maintenance tasks using linux, such as zapping the drives as described by cuddletech.com.

I do not know if the LRC’s in the NetApp shelves are as feature rich as you would find on the RS-1600-FC2.  On the DS14mk2 you may find them with ESH modules or the newer ESH2 modules.  The ESH2 modules support auto-terminating FCAL.  Other than that I am not sure if there are any important differences for someone trying to use it as a JBOD, but personally I would look for a DS14mk2 with ESH2.

What about the DS14 (non-mk2)?  Well, these should work just fine as well.  These have copper interfaces, and so you would need to use a MIA, but I don’t see why in the end you would not be able to get it to work just fine, and you can likely find it cheap.

I do not know much about whether any of the DS14 (mk1 or mk2) have the DIP switches that allow you to break into multiple loops and set various tasks.  My thought is no they don’t.  Just looking at the ESH’s they look to be single loop devices (1 in, 1 out), but if you find out otherwise, please post here.  Good Luck!