This post is to highlight something that is already mentioned on cisco.com, but you may not be aware of.  That is the number of port indexes consumed by various MDS modules.

Module	Port Indexes Consumed
DS-X9016	16
DS-X9032	32
DS-X9032-SSM	32
DS-X9304-SMIP	16**
DS-X9308-SMIP	32**
DS-X9302-14K9	22**
DS-X9112	12
DS-X9124	24
DS-X9148	48
DS-X9704	4

* A Port Channel can be considered a logical port but it does not consume a port index rather the interfaces that comprise the Port Channel consume them.
** Each port IP services physical interface supports up to three FCIP tunnels. Therefore every IP services physical interface is assigned four index IDs, there for FCIP and one for iSCSI.

You can also get access to this information from the CLI by using the show port index-allocation command:

MDS1# show port index-allocation
Module index distribution:
------------------------------------------------------+
Slot | Allowed |      Allotted indices info           |
     |  range* | Total |          Index values        |
-----|---------|-------|------------------------------|
 1   |   0-  31|   16  | 0-15                         |
 2   |  32-  63|   32  | 32-63                        |
 SUP | 253-255 |   3   | 253-255                      |

*Allowed range applicable only for Generation-1 modules

Take careful note of the IPS-4, IPS-8 and 14/2 modules. These modules have Gigabit Ethernet ports onboard which can be used for either iSCSI or FCIP. FCIP has a limit of three tunnels per port. In addition to three tunnels of FCIP, each port can have iSCSI/iSLB configured. Each of these takes up a port index, so each gigabit port reserves 4 port indexes. As noted in the above chart, port-channels do not consume port-indexes separate from those already consumed by their members.

It’s also important to realize that port indexes are allocated in the order the modules power up. Modules power up based on the slots they are in, starting with the first slots. Generation 1 modules have to start on an even boundary of 32 (0, 32, 64, etc). This is not a restriction on Generation 2 modules. For this reason Generation 1 modules need to be installed before Generation 2 modules, otherwise there could be an issue in port index allocation and the module will not come up. To correct this you need to power off the generation 2 modules, power on the generation 1 modules, and then power on the generation 2 modules. This can be done using the poweroff module command.

Cisco has already announced the new Supervisor-2B which advertises support for FCoE on the Cisco MDS.  But what does that mean exactly?  I don’t know for sure, because there really isn’t much information on Cisco to document the FCoE capabilities of this supervisor.  The tech note eludes to an “FCoE module” of some sort that will be required for FCoE functionality.

It is pretty clear that the MDS will not support FCoE initiators to connect directly to it.  That job is already handled very well by the Nexus 5k/2k.  The Nexus switches can also terminate FCoE targets as well, so I am not sure what the advantage would be to having some sort of FCoE target module since those could just be connected to the Nexus.  A capability that would seem most likely is that the MDS will be able to share its FC storage via FCoE.  This type of functionality already exists on the MDS for iSCSI.

The main benefits of FCoE in a typical network are on the initiator side.  That is wear 80% or more of the benefits are.  The risk/difficulty today in an FCoE world is the lack of pervasive DCB switching and storage supporting FCoE directly.  By terminating initiators using CNA’s into Nexus switches, but leaving your storage on FC, you gain almost all the benefits and take on very little obstacles that one may face when trying to deploy an end-to-end converged network.  This is also where the savings are from a financial perspective.  For this reason I can see the MDS FCoE module as a useful option.  On the flip side is the fact that there are many companies producing front end units that will share out heterogenous storage using FCoE.

All we can do for now is wait and see what Cisco releases on the order of documentation and hardware to support FCoE on the MDS.  It is unfortunate that a new Supervisor seems to be required and likely an additional module, hopefully there are some good features and functionality that will come out of its use.

Title: Designing Storage Networks
Author: Tom Clark
ISBN-10: 0321136500
ISBN-13: 978-0321136503
Available from Amazon

When I first started learning about storage it was hard to find the right place to start.  I came from a Cisco-centric world, and I knew I wanted to learn the MDS switches, so I started with documentation there.  The problem with this approach is that you skip alot of fundamentals along the way.  I decided I need to get a book to use as a primer.  I actually ended up getting several.  Among all the books I got, I liked this one the best.  I really like Tom’s writing style and the order in which he covers topics.

The book takes you through learning about the various standards, SCSI protocol, and IP protocols.  It does have some dated parts, after all the book was written in 2003.  However, I find it still the best thing out there to learn about what it teaches.

Especially helpful were things such as principal switch selection, FC-AL initialization, FLOGI/PLOGI, public/private loops, and many other relatively elementary concepts.  Later on when I would go on to more advanced studies using The Fibre Channel Bench Reference and other texts, it all made sense thanks to a careful and complete introduction to technologies made possible by this excellent book.

For what it’s worth, I also own Tom Clark’s other book IP SANS: A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area Networks. I did not really care for it.  Its not necessary if you own this book as there is much overlap, and I found it a alot more basic, and I was looking for more of an experts introduction.

I recommend this book for someone who has limited experience in storage networks, and is trying to pursue storage networking.  I you have read this book and have any comments please feel free to contribute!

When putting together a storage lab for study, one of the important components is the disk arrays.  For the purpose of CCIE Storage Study, the arrays may seem to be rather generic, and from the MDS perspective they pretty much are.  You don’t need anything fancy.  You can use public or private loop for example.  I prefer public loop simply because its fabric aware, and TL ports are not supported on Gen 2 modules, so I want to future proof my investment, especially since public loop JBOD’s can be had for so little.

There are reasons however for getting a better JBOD, it may in fact save you money, even though it may seem to cost more.  For example, it’s no secret that one of my favorite JBOD’s is the Xyratex RS-1600-FC2.  These are basically what NetApp uses as their DS14mk2 disk shelves.  These are public loop and seemingly basic, however they have some nice options for lab study, particularly the ability to be paritioned. There are many other JBOD’s that support this type of partitioning functionality.  For example the Sun StorEdge A5200 (and most later models) support this functionality.  I prefer the Xyratex unit over the Sun however, since its more readily available, less heavy, less noisy, uses less power, and produces less heat.  This concept of splitting a JBOD can be applied to many different disk arrays however.

The RS-1600-FC2 has the ability to hold 16 drives.  These can be configured as a single 16 drive loop, or as two 8 drive loops.  When your using two loops, they are really completely separate from the MDS’s perspective.  It effectively views these two separate loops as two JBOD’s each with 8 drives.  Because the RS-1600-FC2 has two LRC modules, you end up with two ports for each loop to plug into your MDS’s, just as you would if you had two physically separate JBOD’s.  There is much advantage to this.  When your building a SAN lab, power, space, cooling and noise can really become an issue.  It’s not the type of thing you easily build in your house.  The biggest culprits of these environmentals are in fact the disk shelves, followed by the MDS’s, and lastly the servers and other support equipment.  Being able to reduce the number of power circuits to half, and not introduce any additional power supplies and fans is a big plus.  Also you don’t have to use the redundant power on the JBOD’s, they will work just fine with single power supplies plugged in, you can silence any alarms using the configuration screen or jumpers.  The net result, is that instead of purchasing say four separate JBOD’s for study, you can get away with 2 and have pretty much the same result.

Here is a look at the way this particular JBOD is laid out:

You can see that the JBOD has redundant LRC’s, redundant power supplies, and redundant fans.  Notice how the drives are laid out left to right.  The drives can be partitioned so that drives 0 through 7 can bee on one loop and drives 8 through 15 can be on a second loop.

To the left are the two different loop configurations supported on this particular JBOD.  Configuring these is very simple, via the use of a thumbwheel and set of jumpers on the back of the unit.  Xyratex refers to the 1×16 drive setup as Mode 1, and the 2×8 drive setup as Mode 2.  A decent setup would be to populate the array with at least 12 drives, 6 in each loop.  I find 6 to be a very useful number of drives for a JBOD and studying storage.  You could populate all the drives, but each drive does use more power and generate more heat, so something to take into consideration.  With our setup of 12 drives, we would populate drive slots 0-5 and drive slots 8-13.  Each loop would be setup in the same way.  I prefer to use small drives and typically finding drives that are 7200 or 10000 RPM would be preferable to 15000 RPM, since usually the faster the drives the more heat and power that is used.  There is not much benefit in using larger drives if its just for lab.  You can get small good drives for probably less than $10 each on ebay.  An actual RS-1600-FC2 based shelf unit can sometimes be had for as little as $100-$150.

You can see that the switch settings clearly show its a breeze to partition the JBOD.  Simply set switch 1 to “Off” and the JBOD will use a split 2×8 configuration the next time it is powered on.  Don’t worry that this is not the factory recommended setting, after all why would Xyratex want you to get two for the price of one .  There are other settings you may want to configure such as the speed or the drive addressing.   Realize that on most JBOD’s like the RS-1600-FC2, there is no auto-detection between 1Gb/s and 2Gb/s, you have to set this manually.  You have to make sure your GBIC’s at both end support the speed as well, which with 1-2GB this usually is not a problem.

So this is one step toward making your study lab more affordable and more easier to manage.  It is also possible to virtualize the servers.  I am not talking about using NPIV however, as that would not be as appropriate in a SAN lab, but rather multiple physical adapters in a beefy server running ESX for example.  You could say run 3-4 hosts on a box like this, and put 3 dual port HBA’s in it, and have literally 1 server and 2 JBOD’s doing the work of 4 servers and 4 JBOD’s.  Unfortunately there is no way to virtualize the MDS’s or third party switches such as the McData and Brocades you will need.

If you are using a JBOD that supports this type of partitioning please comment.  I can tell you that the JMR Fortra arrays I use do not.  But I have the Sun A5200 and Xyratex arrays which do.  Also to note is there is a difference between the Xyratex RS-1600-FC and Xyratex RS-1600-FC2.  You want the FC2.  There are also some other models which may be RAID, and what you want in a CCIE Storage Lab is JBOD mode, not RAID.  So do careful shopping, ask questions and make some smart purchases.  Good luck in building your lab!

The Cisco MDS switches support many different ways you can connect them with both fibre channel and ethernet. Some modules do not support all of the features, but they are available in the architecture. For example the MDS 9216i and 14/2 modules do not support Ethernet Port Channels.

Consider the case where you want multiple links to be used with FCIP. You have at least two different ways you can handle this:

Scenario A – An FCIP tunnel made of Ethernet Port Channels. Say you have two ethernet links, g1/1 and g1/2. You wish to combine them. You can make them into an ethernet port channel and then run an FCIP tunnel over this. This has a few disadvantages:

1. Ethernet Port Channels must be made using adjacent ports on an IPS module. For example g1/1 and g1/2 can be made into an ethernet Port Channel, but not g1/1 and g1/3.

2. Because all of the FCIP traffic is going to a specific IP destination/port, it will only take one path within the Ethernet Channel (transmit). The return path may or may not take the same link, this can be very inefficient.

3. Ethernet Port Channels are limited to two members.

4. Both Ethernet Port Channel members must goto the same switch.

5. The Ethernet Port Channel must be terminated on the adjacent ethernet switch, it cannot transit other ethernet switches.

Here is this scenario illustrated:

There is another way this could be configured:

Scenario B – An FC Port Channel made up of FCIP links. In this scenario you place each Gigabit Ethernet link into its own FCIP tunnel. These can be located anywhere in the MDS. Multiple FCIP tunnels can be aggregated into a single FC Port Channel. This has the following advantages:

1. You can have up to 16 members in a FC Port Channel. They can be from any ports on the MDS.

2. All modules support FC Port Channels.

3. Up to 128 FC Port Channels when mixing Gen-1 and Gen-2 modules, up to 256 Port Channels when no Gen-1 modules are present.

4. FSPF sees this as one big single pipe. If a member breaks it does not change the FSPF state.

5. Although the logical FC Port Channel must be from one MDS to another, the individual FCIP links may transit multiple ethernet switches and multiple separate paths.

6. MDS uses the load balancing parameter set under the VSAN. Each exchange or flow can use a different link in the FC Port Channel, making use of the entire aggregate. Each exchange or flow would be limited to a single FCIP pipe size at one time.

Here is this illustrated:

I believe that Cisco is phasing out Ethernet Port Channel support in the MDS. It really is not as useful of a feature. There are really almost no instances where I can think of that ethernet port channels make sense. Certainly when we are talking about FCIP, it makes a lot more sense to make a FC Port Channel out of FCIP links. One may argue that in the case of iSCSI making a ethernet port channel can give you 2GB/s. This may or may not be the case. If you have setup iSCSI to use a new iSCSI connection per request, then yes this is possible. But you could also accomplish the same thing using iSLB with VRRP, and at the same time add a lot more fault tolerance, easier administration and scalable aggregate bandwidth beyond 2GB.

You can do some crazy things with ethernet port channels, FCIP and FC Port channels. You can keep stacking them up making FCIP tunnels out of port channels that are made up of FCIP tunnels or port channels. It really lets you go crazy making some very confusing stuff that actually works There is of course really no need to do this, but it probably would not be out of place to see something that crazy in the CCIE lab as its known for creating complex scenarios that don’t exactly mimic real world but are rather designed to test expert level knowledge.

If you have a good use for Ethernet Port Channels on the MDS post some comments!