One issue to be mindful of when configuring Cisco MDS switches with Brocade switches is that Brocades Per VC Flow Control must be disabled.  Cisco MDS 9000 switches do not support Brocades Per VC Flow Control.  The flow control will interfere with an ISL being established between the Cisco and Brocade switches, even if other parameters are set correctly.

An essential document for successful configuration of Cisco to Brocade interoperability is the Cisco MDS 9000 Family Switch-to-Switch Interoperability Guide.

Here is a review of the requirements for Brocade Interop:

• R_A_TOV must be the same on both sides
•  E_D_TOV must be the same on both sides
• B2B Credits must be the same on both sides
• core.pid on the brocade must match the interop mode on the MDS; pidFormat:0 = interop 2, pidFormat:1 = interop 3
• per vc flow control must be off on the brocade

Verifying R_A_TOV and E_D_TOV

Brocade

SINO-300:admin> configshow | grep E_
fabric.ops.E_D_TOV:2000
SINO-300:admin> configshow | grep R_
fabric.ops.R_A_TOV:10000
SINO-300:admin>

Cisco

MDS1(config-if)# do show fctimer
F_S_TOV   D_S_TOV   E_D_TOV   R_A_TOV
—————————————-
5000 ms   5000 ms   2000 ms   10000 ms
MDS1(config-if)#

Setting B2B Credits on MDS to match Brocade

Brocade

SINO-300:admin> configshow | grep BB
fabric.ops.BBCredit:16
fcAL.useAltBBCredit:0
flannel.ops.openBBCredit:4
SINO-300:admin>

Cisco

MDS1(config-if)# do sh run int fc1/6
version 3.3(5a)

interface fc1/6
no shutdown
switchport fcrxbbcredit 16

Verify core.pid matches the Cisco Interop mode

Brocade

SINO-300:admin>
SINO-300:admin> configshow | grep inter
cer.internal_port_code:1
switch.interopMode:0
SINO-300:admin> configshow | grep pid
fabric.ops.mode.pidFormat:1
SINO-300:admin>

Cisco

MDS1(config-if)# do sh run | in interop
vsan 20 name “BLUE” interop 3 loadbalancing src-dst-id

If you do all of the above, you still may have the issue of Per VC flow control being enabled on the Brocade switch.  You may experience errors in the logs such as the following:

MDS1(config-if)# 2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:28 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:28 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:29 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:29 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:11 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:11 MDS1 %PORT-5-IF_DOWN_ELP_FAILURE_ISOLATION_UNKNOWN_FLOW_CTL_PARAM: %$VSAN 20%$ Interface fc1/6 is down(Isolation due to ELP failure: invalid flow control param)
2012/05/15-07:22:30, [FABR-1001], 6215, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:22:43, [FABR-1001], 6216, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:22:56, [FABR-1001], 6217, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:23:09, [FABR-1001], 6218, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:23:22, [FABR-1001], 6219, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).

So how do we solve this?  Brocade has a command that can be used to disable the Per VC Flow control behavior.  From the Cisco MDS 9000 Switch-to-Switch Interoperability Guide documentation located here:

ISL Flow Control

Brocade uses a proprietary flow control called Virtual Channel (VC) flow control. VC flow control is used by Brocade for fair traffic distribution during congested fabrics by prioritizing management traffic.

When an ISL between an MDS 9000 switch and Brocade switch comes up, the ISL negotiates for the standards-based buffer-to-buffer flow control during ELP. The MDS 9000 switch rejects the Brocade proprietary VC flow control and responds with the standards-based buffer-to-buffer flow control.

To enable Brocade to accept the standards-based flow control on Brocade firmware versions 3.1.0 and 4.1.1 so that the ISL will complete negotiation, use the portcfgislmode slot/port 1 command on the Brocade switch. Switches running version 2.x firmware do not require this command.

This does not affect any other ISLs in the fabric. All Brocade-to-Brocade ISLs can still be native VC flow control and MDS 9000 switch-to-MDS 9000 switch ISLs can still be TE (trunking or multi-VSAN) ISLs. Having a Brocade-to-MDS 9000 switch ISL using buffer-to-buffer flow control should have no noticeable impact to the fabric.

Once Per VC Flow Control is disabled on the Brocade, the ISL should come up.  Special thanks to Sunny LiYu Zhang for allowing me to use the screen captures from his lab.

Update:  Thanks Joseph Tarin for finding out that the portcfgislmode command likely will require a comma as follows

One of the nice things about SAN-OS (and also NX-OS) is that you can enable features, even if you do not have full licenses installed. What happens is the devices come with a “try before you buy” licensing model where you can try out the features for I believe 120 days. Here is a bit on how this works:

1. This is known as Grace Licensing, think “Grace Period”, and it allows you to use the feature without actually having a license installed.

2. It is limited to 120 days. This means that the countdown starts as soon as the feature is activated. For example, as soon as you enable FCIP, the “SAN Extension” license would start counting down.

3. If you disable a feature, the countdown stops.

4. The countdown only keeps going while the feature is active. So if you power down the device or disable the feature the countdown stops.

What this means is you end up with 120 days of “run time”, not 120 days of “calendar time. This is great news for working in the lab.

One of the issues with buying used equipment, or using lab equipment, is that at some point someone may have used up all the grace licensing. The grace licensing status is stored on the SPROM. And it is possible to actually clear this out. In fact, in earlier versions of SAN-OS doing an “init system” would clear the grace licensing.

“init system” is done at the (boot) prompt. Its very destructive, as it wipes out all code, kickstart, system, and configuration. So no one in their right mind would run this in production, as its very disruptive to do so (every 120 days!). However, Cisco decided to remove this functionality for whatever reason. So if you run out of grace license you are hosed.

Now a 9216a costs maybe $400 (give or take) on ebay. What does licensing cost? Well, if you factor in all the licenses: MAINFRAME, FM SERVER, SAN EXT, ENTERPRISE, etc. Its about $80,000 list. Yes, that’s right, you will pay At least $40,000 or so to turn on all the licensing on your $400 9216a .

There is a way however. A way for those who have labs. Like most of the information on my site, you can’t get this by searching the Internet, or asking TAC, the truth is most TAC engineers are not actually aware of this.

There is a command “clear license sprom 0″ I believe, and that is an interesting command. What is more interesting is actually going into the OS itself, and looking at the scripts/executables which show how the licensing is setup, and how it can be….manipulated. The licensing was in part written by Macrovision (remember those old Cable TV scramblers?). In any case, no need to mess with the shell or sprom. Here is the easy way.

1. Get to the (boot) prompt
2. issue the “init system” command.
3. Once completed reboot
4. You will need to treat this like a MDS with no code, after all you just wiped out everything on it. Download SAN-OS 3.2(1a). Load it onto the MDS.
5. Do the install all procedure to get the kickstart and system images all hooked up and working.
6. Reboot
7. Drop to the (boot) prompt, re-issue “init system”
8. Now reboot and re-install 3.3(5) (or whatever you like)
9. Grace licenses should be reset.

Now I ask that no one do something stupid like try to run this in production. This is for LAB study. This takes time, its not fun to do. However, I will tell you that the average SAN study student will only need to do this at most 1 time. I believe it takes roughly 350-500 hours of on rack time to be fully prepared for the CCIE Storage lab. 120 days of actual runtime is ALOT more than that. So no worries.

If this was helpful please share your comments.

Best of luck!

Recently the topic came up on the Cisco Learning Network on OSM ratio’s for the 18/4 module.  Here is a summary of many of the ratios for various modules as well as some handy notes:

Generation 2 Switches and Modules

9216 does not support Gen-2 modules

If dedicated ports are not using all of their allocated bandwidth, the unused bandwidth is made available for use by all ports configured for shared bandwidth mode.

Performance Buffers are used on 12-port 4GB, 4-port 10GB modules only.

Port Groups

With regards to the 12-port module:

There are 2488 extra buffers available as extended BB_credit buffers after allocating all the default BB_credit buffers for all the ports in ISL mode (5488 – (250 * 12)).

Extended BB_credits are allocated across all ports on the switch. That is, they are not allocated by port group.

With regards to the 4-port module:

The ports in the 4-port 10-Gbps switching module only support 10-Gbps dedicated rate mode. FL port mode and shared rate mode are not supported.

There are 4488 extra buffers available as extended BB_credits after allocating all the default BB_credit buffers for all the ports in ISL mode (5488 – (250 * 4)).

VRRP on the MDS in it’s basic form looks something like this:

Switch 1

interface GigabitEthernet2/1.10
  ip address 192.168.10.2 255.255.255.0
  switchport mtu 3000
  no shutdown
  vrrp 10
    priority 120
    preempt
    address 192.168.10.2
    no shutdown

In the above configuration, 192.168.10.2 is the actual physical interface address and 192.168.10.2 is also the VIP address.  This is common.  The other side may looking something like this:

Switch 2

interface GigabitEthernet2/1.10
  ip address 192.168.10.3 255.255.255.0
  switchport mtu 3000
  no shutdown
  vrrp 10
    priority 100
    preempt
    address 192.168.10.2
    no shutdown

On this side of the link 192.168.10.3 is the physical interface address and 192.168.10.2 is the VIP address.  These two MDS switches Switch 1 and Switch 2 are both members of VR10.  As per the VRRP specification, VRRP VIP addresses are for passing traffic onto real server IP addresses that they front end, they are not for the origination or destination of traffic.  If you try to send traffic to a VRRP VIP, these packets are by default dropped.  Some application’s on the  MDS may need to use the VRRP IP as an actual IP address to terminate traffic on.  For example iSNS (removed from current versions of SAN-OS / NX-OS) and IPSec.  With IPSec you may wish to use the VRRP as a destination of a IPSec tunnel for high availability.  To do this you have to add the “secondary” option like so:

Switch 2

interface GigabitEthernet2/1.10
  ip address 192.168.10.3 255.255.255.0
  switchport mtu 3000
  no shutdown
  vrrp 10
    priority 100
    preempt
    address 192.168.10.2 secondary
    no shutdown

Realize on Switch 1, the primary IP address of G2/1.10 is configured the same as the VIP.  So in this case when Switch 1 is master it will be able to accept traffic terminated to the VIP address.  Switch 2 will not be able to receive traffic destined to the VIP address even with the secondary option configured unless it is the master!  So the key is, the secondary option allows a switch to be able to terminate traffic destined to the VIP, when it’s the master.  If we were terminated an IPSec session on the VIP of Switch 1 and Switch 2, and wanted it to failover, we would need to configure the secondary address for Switch 2.

Also realize this has an effect on the switch VRRP priorities.  A switch that uses the same VRRP address as its interface address automatically has a VRRP priority of 255.  A switch using a different address for its VRRP than it has on its interface has a priority of 100.  This applies to the use of the secondary command as well.  With or without the secondary command the interface will have a default priority of 100 if its interface address does not match the VRRP address.

Also realize, you cannot configure a secondary address to be the same as the interface address, if you try to do so you will get an error:

Switch 1(config-if)# no shut
2010 Oct  9 21:20:04.648 MDS2 %VRRP_ENG-2-INVALID_CONFIG: Cannot
start the VR 10 on the interface GigabitEthernet2/1.10. Invalid
IP configuration. Reason: A secondary VRRP address can't be
configured as the primary IP address of the interface

This post is to highlight something that is already mentioned on cisco.com, but you may not be aware of.  That is the number of port indexes consumed by various MDS modules.

* A Port Channel can be considered a logical port but it does not consume a port index rather the interfaces that comprise the Port Channel consume them.
** Each port IP services physical interface supports up to three FCIP tunnels. Therefore every IP services physical interface is assigned four index IDs, there for FCIP and one for iSCSI.

You can also get access to this information from the CLI by using the show port index-allocation command:

MDS1# show port index-allocation
Module index distribution:
------------------------------------------------------+
Slot | Allowed |      Allotted indices info           |
     |  range* | Total |          Index values        |
-----|---------|-------|------------------------------|
 1   |   0-  31|   16  | 0-15                         |
 2   |  32-  63|   32  | 32-63                        |
 SUP | 253-255 |   3   | 253-255                      |

*Allowed range applicable only for Generation-1 modules

It’s also important to realize that port indexes are allocated in the order the modules power up. Modules power up based on the slots they are in, starting with the first slots. Generation 1 modules have to start on an even boundary of 32 (0, 32, 64, etc). This is not a restriction on Generation 2 modules. For this reason Generation 1 modules need to be installed before Generation 2 modules, otherwise there could be an issue in port index allocation and the module will not come up. To correct this you need to power off the generation 2 modules, power on the generation 1 modules, and then power on the generation 2 modules. This can be done using the poweroff module command.