Cisco MDS and Brocade Interop Issues with Per VC Flow Control

One issue to be mindful of when configuring Cisco MDS switches with Brocade switches is that Brocades Per VC Flow Control must be disabled.  Cisco MDS 9000 switches do not support Brocades Per VC Flow Control.  The flow control will interfere with an ISL being established between the Cisco and Brocade switches, even if other parameters are set correctly.

An essential document for successful configuration of Cisco to Brocade interoperability is the Cisco MDS 9000 Family Switch-to-Switch Interoperability Guide.

Here is a review of the requirements for Brocade Interop:

• R_A_TOV must be the same on both sides
•  E_D_TOV must be the same on both sides
• B2B Credits must be the same on both sides
• core.pid on the brocade must match the interop mode on the MDS; pidFormat:0 = interop 2, pidFormat:1 = interop 3
• per vc flow control must be off on the brocade

Verifying R_A_TOV and E_D_TOV

Brocade

SINO-300:admin> configshow | grep E_
fabric.ops.E_D_TOV:2000
SINO-300:admin> configshow | grep R_
fabric.ops.R_A_TOV:10000
SINO-300:admin>

Cisco

MDS1(config-if)# do show fctimer
F_S_TOV   D_S_TOV   E_D_TOV   R_A_TOV
—————————————-
5000 ms   5000 ms   2000 ms   10000 ms
MDS1(config-if)#

Setting B2B Credits on MDS to match Brocade

Brocade

SINO-300:admin> configshow | grep BB
fabric.ops.BBCredit:16
fcAL.useAltBBCredit:0
flannel.ops.openBBCredit:4
SINO-300:admin>

Cisco

MDS1(config-if)# do sh run int fc1/6
version 3.3(5a)

interface fc1/6
no shutdown
switchport fcrxbbcredit 16

Verify core.pid matches the Cisco Interop mode

Brocade

SINO-300:admin>
SINO-300:admin> configshow | grep inter
cer.internal_port_code:1
switch.interopMode:0
SINO-300:admin> configshow | grep pid
fabric.ops.mode.pidFormat:1
SINO-300:admin>

Cisco

MDS1(config-if)# do sh run | in interop
vsan 20 name “BLUE” interop 3 loadbalancing src-dst-id

If you do all of the above, you still may have the issue of Per VC flow control being enabled on the Brocade switch.  You may experience errors in the logs such as the following:

MDS1(config-if)# 2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:28 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:28 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:29 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:29 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:11 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:11 MDS1 %PORT-5-IF_DOWN_ELP_FAILURE_ISOLATION_UNKNOWN_FLOW_CTL_PARAM: %$VSAN 20%$ Interface fc1/6 is down(Isolation due to ELP failure: invalid flow control param)
2012/05/15-07:22:30, [FABR-1001], 6215, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:22:43, [FABR-1001], 6216, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:22:56, [FABR-1001], 6217, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:23:09, [FABR-1001], 6218, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:23:22, [FABR-1001], 6219, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).

So how do we solve this?  Brocade has a command that can be used to disable the Per VC Flow control behavior.  From the Cisco MDS 9000 Switch-to-Switch Interoperability Guide documentation located here:

ISL Flow Control

Brocade uses a proprietary flow control called Virtual Channel (VC) flow control. VC flow control is used by Brocade for fair traffic distribution during congested fabrics by prioritizing management traffic.

When an ISL between an MDS 9000 switch and Brocade switch comes up, the ISL negotiates for the standards-based buffer-to-buffer flow control during ELP. The MDS 9000 switch rejects the Brocade proprietary VC flow control and responds with the standards-based buffer-to-buffer flow control.

To enable Brocade to accept the standards-based flow control on Brocade firmware versions 3.1.0 and 4.1.1 so that the ISL will complete negotiation, use the portcfgislmode slot/port 1 command on the Brocade switch. Switches running version 2.x firmware do not require this command.

This does not affect any other ISLs in the fabric. All Brocade-to-Brocade ISLs can still be native VC flow control and MDS 9000 switch-to-MDS 9000 switch ISLs can still be TE (trunking or multi-VSAN) ISLs. Having a Brocade-to-MDS 9000 switch ISL using buffer-to-buffer flow control should have no noticeable impact to the fabric.

Once Per VC Flow Control is disabled on the Brocade, the ISL should come up.  Special thanks to Sunny LiYu Zhang for allowing me to use the screen captures from his lab.

Update:  Thanks Joseph Tarin for finding out that the portcfgislmode command likely will require a comma as follows