One issue to be mindful of when configuring Cisco MDS switches with Brocade switches is that Brocades Per VC Flow Control must be disabled.  Cisco MDS 9000 switches do not support Brocades Per VC Flow Control.  The flow control will interfere with an ISL being established between the Cisco and Brocade switches, even if other parameters are set correctly.

An essential document for successful configuration of Cisco to Brocade interoperability is the Cisco MDS 9000 Family Switch-to-Switch Interoperability Guide.

Here is a review of the requirements for Brocade Interop:

• R_A_TOV must be the same on both sides
•  E_D_TOV must be the same on both sides
• B2B Credits must be the same on both sides
• core.pid on the brocade must match the interop mode on the MDS; pidFormat:0 = interop 2, pidFormat:1 = interop 3
• per vc flow control must be off on the brocade

Verifying R_A_TOV and E_D_TOV

Brocade

SINO-300:admin> configshow | grep E_
fabric.ops.E_D_TOV:2000
SINO-300:admin> configshow | grep R_
fabric.ops.R_A_TOV:10000
SINO-300:admin>

Cisco

MDS1(config-if)# do show fctimer
F_S_TOV   D_S_TOV   E_D_TOV   R_A_TOV
—————————————-
5000 ms   5000 ms   2000 ms   10000 ms
MDS1(config-if)#

Setting B2B Credits on MDS to match Brocade

Brocade

SINO-300:admin> configshow | grep BB
fabric.ops.BBCredit:16
fcAL.useAltBBCredit:0
flannel.ops.openBBCredit:4
SINO-300:admin>

Cisco

MDS1(config-if)# do sh run int fc1/6
version 3.3(5a)

interface fc1/6
no shutdown
switchport fcrxbbcredit 16

Verify core.pid matches the Cisco Interop mode

Brocade

SINO-300:admin>
SINO-300:admin> configshow | grep inter
cer.internal_port_code:1
switch.interopMode:0
SINO-300:admin> configshow | grep pid
fabric.ops.mode.pidFormat:1
SINO-300:admin>

Cisco

MDS1(config-if)# do sh run | in interop
vsan 20 name “BLUE” interop 3 loadbalancing src-dst-id

If you do all of the above, you still may have the issue of Per VC flow control being enabled on the Brocade switch.  You may experience errors in the logs such as the following:

MDS1(config-if)# 2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:02 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:15 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:28 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:28 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:29 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:29 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:42 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:13:55 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_OFFLINE: %$VSAN 20%$ Interface fc1/6 is down (Offline)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_NONE: %$VSAN 20%$ Interface fc1/6 is down (None)
2012 May 15 08:14:08 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:11 MDS1 %PORT-5-IF_DOWN_LINK_FAILURE: %$VSAN 20%$ Interface fc1/6 is down (Link failure)
2012 May 15 08:14:11 MDS1 %PORT-5-IF_DOWN_ELP_FAILURE_ISOLATION_UNKNOWN_FLOW_CTL_PARAM: %$VSAN 20%$ Interface fc1/6 is down(Isolation due to ELP failure: invalid flow control param)
2012/05/15-07:22:30, [FABR-1001], 6215, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:22:43, [FABR-1001], 6216, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:22:56, [FABR-1001], 6217, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:23:09, [FABR-1001], 6218, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).
2012/05/15-07:23:22, [FABR-1001], 6219, FID 128, WARNING, SINO-300, port 0, incompatible flow control parameters (3).

So how do we solve this?  Brocade has a command that can be used to disable the Per VC Flow control behavior.  From the Cisco MDS 9000 Switch-to-Switch Interoperability Guide documentation located here:

ISL Flow Control

Brocade uses a proprietary flow control called Virtual Channel (VC) flow control. VC flow control is used by Brocade for fair traffic distribution during congested fabrics by prioritizing management traffic.

When an ISL between an MDS 9000 switch and Brocade switch comes up, the ISL negotiates for the standards-based buffer-to-buffer flow control during ELP. The MDS 9000 switch rejects the Brocade proprietary VC flow control and responds with the standards-based buffer-to-buffer flow control.

To enable Brocade to accept the standards-based flow control on Brocade firmware versions 3.1.0 and 4.1.1 so that the ISL will complete negotiation, use the portcfgislmode slot/port 1 command on the Brocade switch. Switches running version 2.x firmware do not require this command.

This does not affect any other ISLs in the fabric. All Brocade-to-Brocade ISLs can still be native VC flow control and MDS 9000 switch-to-MDS 9000 switch ISLs can still be TE (trunking or multi-VSAN) ISLs. Having a Brocade-to-MDS 9000 switch ISL using buffer-to-buffer flow control should have no noticeable impact to the fabric.

Once Per VC Flow Control is disabled on the Brocade, the ISL should come up.  Special thanks to Sunny LiYu Zhang for allowing me to use the screen captures from his lab.

Update:  Thanks Joseph Tarin for finding out that the portcfgislmode command likely will require a comma as follows

A list of every command possible on NX-OS on the Nexus 7000.  It’s in a bit different form than a CLI command reference may list it, and best of all it includes all the hidden commands as well.

You can view the list here.  I will post one for the Nexus 7000 soon.  If there are any commands  you know of that are not listed, please let me know, but I am pretty sure this is 100%.

A list of every command possible on NX-OS on the Nexus 5000.  It’s in a bit different form than a CLI command reference may list it, and best of all it includes all the hidden commands as well.

You can view the list here.  I will post one for the Nexus 7000 soon.  If there are any commands  you know of that are not listed, please let me know, but I am pretty sure this is 100%.

Anyone who comes from a Fibre Channel background likely has a pretty good understanding of Buffer to Buffer (B2B) Credits.  The Fibre Channel network is designed so that all capacity within the system is carefully distributed and not oversubscribed.  This is one of the many ways FC keeps from dropping frames, something that should never be allowed to happen in a FC world, because the underlying protocol at the FC4 layer, SCSI, will not tolerate it.

With Fibre Channel, each switch keeps track of how many credits it has available over each of it’s links.  If it has credits, it sends data, decrements the credits and stops sending data when it runs out of credits.  As it receives Receiver Ready frames (R_RDY), it replenishes its credits.  At no time does it allow itself to use more capacity than it has agreed upon with the distant end.

802.3 defines a PAUSE frame in order to implement a lossless fabric over ethernet.  With PAUSE, a switch does not keep track of B2B credits, it simply just keeps sending data until the far end tells it to stop.  So long as their are enough buffers in the output queues, this should work fine.  Perhaps the switch was about to send a frame but received a PAUSE, it will wait, the frame will sit in queue, and then it will put it on the wire as soon as it receives the go ahead.  The “go ahead” is in fact another PAUSE frame but with a time of 0, essentially an un-PAUSE.

Some of the benefits of PAUSE over B2B credits, is that with PAUSE and Priority Flow Control (PFC), you can pause individual lanes of the link based on priority.  With B2B you either sent data or you didn’t with no regard for prioritization.  This gives increased control in the system, we can send data that can tolerate a drop in order to maximize the efficiency of the link, and at the same time pause data that cannot handle a drop.

So on the surface it can appear that FCoE and its ability to create a lossless network does just about all we need it to do.  But I would caution that when it comes to a long distance link be careful about trying to run FCoE.  First, it was not designed with long distance links in mind.  This is what FCIP is for.  Another alternative is to run FC and use B2B credits – for example over a DWDM link.  Both of these (FCIP and FC) rely on B2B credits.  The allocation of agreed upon resources at the beginning of the exchange is essential for reliable delivery of data over a long distance.  With such long distances and the size of a FC frame at 2112 bytes, it can take a considerable number of frames in flight to fill a wire.  You don’t want to cross paths, in flight with a PAUSE frame when you have 100+ frames in flight.

A quick review of what Fibre Channel looks like on a wire:

•  At 1Gbps a FC frame is 4km long, at 2Gbps a frame is 2km long, and at 4Gbps a frame is 1km long.
• A 10km cable is 20km round trip.  Round trip must be accounted for since the R_RDY packet reply from the distant end needs to traverse this distance before the source receives it and can continue to send.  The goal is the saturate the link with as much traffic as possible.  In order to do this, you must allocate enough B2B credits, which is essentially the allocation of buffer resources.
• On 10km cable @ 2Gbps we need 10 BB credits, 100km cable @ 1Gbps we need  50 BB credits, etc.

This is basic link engineering with Fibre Channel.  Calculate the amount of B2B credits you need, assign them to the switches, and you should have no worries.  You still may need to worry about latency, and of course you will always have to consider the bandwidth delay product of any link to understand it’s true capacity, but your lossless needs will be handled.

If you try to do this with FCoE you may end up in a bad race condition.  There may be substantial frames in flight by the time the distant end generates a PAUSE.  Ethernet is not a guaranteed delivery medium, so the in flight stream is not going to be buffered by the sender.  Frames could get dropped, and lots of them.  This issue doesn’t really exist on a localized data center converged network, because the speeds are so fast (10GB, typically at least 4GB for FC alone), and the distances so small (meters not kilometers) that things can be handled in relative harmony, a hysteresis does not exist that would leave to the type of adverse condition that may exist on a WAN. I am not saying it will not work, and no doubt some people will try it and may be doing it today.  FCoE is great in the data center, it’s not the right tool for the job on the WAN, use FCIP or FC instead.

I am running an ASA 7.2(5)/ASDM 5.2(5), this is the ASA at my house.  And I am not sure when it started, but I cannot get traceroutes to work from my mac to the Internet.  I can traceroute on my LAN, but not going through the ASA, I just get * * *.  Here are the related parts of my ASA config:

So I have found a bit about why this wasn’t working, and interested to get some opinions from the ASA gurus on here:

The ASA is sending the time-exceeded’s to 172.16.5.30, which is my Synology:

08 packets captured

   1: 01:52:42.269090 802.1Q vlan#1 P0 172.16.5.229 > 8.8.8.8: icmp: echo request 

   2: 01:52:42.309355 802.1Q vlan#1 P0 64.233.174.2 > 172.16.5.30: icmp: time exceeded in-transit 

   3: 01:52:47.268159 802.1Q vlan#1 P0 172.16.5.229 > 8.8.8.8: icmp: echo request 

   4: 01:52:47.311141 802.1Q vlan#1 P0 66.249.94.22 > 172.16.5.30: icmp: time exceeded in-transit 

   5: 01:52:52.268906 802.1Q vlan#1 P0 172.16.5.229 > 8.8.8.8: icmp: echo request 

   6: 01:52:52.313368 802.1Q vlan#1 P0 66.249.94.22 > 172.16.5.30: icmp: time exceeded in-transit 

My ASA is very basic, I just PAT everything out the outside interface.  however I do have one static:

static (inside,outside) interface 172.16.5.30 netmask 255.255.255.255 dns 

So my thought with this, is anything coming into the ASA, that does not belong to some other xlate would goto 172.16.5.30.  there are maybe a handfull of ports I send its way via the outside ACL:

access-list outsideIn extended permit tcp any interface outside eq www 

access-list outsideIn extended permit tcp any interface outside eq 5001 

access-list outsideIn extended permit tcp any interface outside eq 5432 

access-list outsideIn extended permit udp any interface outside eq 5432 

access-list outsideIn extended permit tcp any interface outside eq 6881 

access-list outsideIn extended permit tcp any interface outside eq 9997 

access-list outsideIn extended permit tcp any interface outside eq 9998 

access-list outsideIn extended permit tcp any interface outside eq 9999 

access-list outsideIn extended permit tcp any interface outside eq ftp-data 

access-list outsideIn extended permit tcp any interface outside eq ftp 

access-list outsideIn extended permit tcp any interface outside eq 5006 

access-list outsideIn extended permit tcp any interface outside eq 7001 

access-list outsideIn extended permit tcp any interface outside eq 6900 

But it should not take a time-exceeded packet and send it to 172.16.5.30.  The ASA should be smart enough to know that it was in response to a UDP packet sent from 172.16.5.229 (my laptop).  It seems like it could be a bug.  I only have 1 IP address, it’s dynamically assigned, so I static my Synology to it and I also NAT everything else to it:

global (outside) 1 interface

nat (inside) 1 0.0.0.0 0.0.0.0

I am pretty sure this is allowed.  The only other way I could do it would be to make a ton of “statics” that were port based, one for each of the ports in the ACL……….just would be more administrative work, but I wonder if one way or the other is truly correct.  You can see the xlate is clearly there:

home-asa# show xlate | inc ICMP

4 in use, 234 most used

Global 174.61.8.28 Local 172.16.5.30

PAT Global 174.61.8.28(28476) Local 172.16.5.229 ICMP id 41382 

As a test, I got rid of my single “catch all” static and replaced it with static PAT’s:

global (outside) 1 interface

nat (inside) 1 0.0.0.0 0.0.0.0

static (inside,outside) tcp interface www 172.16.5.30 www netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 5001 172.16.5.30 5001 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 5432 172.16.5.30 5432 netmask 255.255.255.255  dns 

static (inside,outside) udp interface 5432 172.16.5.30 5432 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 6881 172.16.5.30 6881 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 9997 172.16.5.30 9997 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 9998 172.16.5.30 9998 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 9999 172.16.5.30 9999 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface ftp-data 172.16.5.30 ftp-data netmask 255.255.255.255  dns 

static (inside,outside) tcp interface ftp 172.16.5.30 ftp netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 5006 172.16.5.30 5006 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 7001 172.16.5.30 7001 netmask 255.255.255.255  dns 

static (inside,outside) tcp interface 6900 172.16.5.30 6900 netmask 255.255.255.255  dns 

This “fixed” the issue.  But now I remember why I did the single static.  DNS Doctoring doesn’t work with static PAT, despite that it allows you to configure the “dns” keyword.  The only way to actually have it work is with a static NAT.  So I guess I am going to just lose the ability to have traceroute (and god know what else) working, in exchange for working DNS Doctoring.  I never noticed anything wasn’t working until now anyways.

Still, I call it a bug, because my laptop should create a XLATE, that the ASA can tie to the time-exceeded coming back…….I don’t know, it just doesn’t seem like the normal way an ASA would work.  At least now its working, I will just have to add a static entry every time I open a port to match my ACL.

Here you can see the Wireshark capture of an ICMP packet going from 8.8.8.8 to the outside of the ASA:

 You can see that under the ICMP section of the packet, there is a IPv4 section, that shows the packet came from the ASA (was NAT’ed) and the destination was 8.8.8.8.  This capture was made on the “outside” interface of the ASA.

Similarly, you can see below is a Wireshark capture of the same packet, captured on the “inside” interface of the ASA:

In this capture you can see that the IPv4 header shows a destination of 172.16.5.30, which is my Synology which has the “catch all” static NAT.  This is incorrect, this is in response to a traceroute ran on my Mac, which his 172.16.5.229.  You can see in the ICMP header, there is a IPv4 subsection that shows correctly that the original source of the packet was my Mac 172.16.5.229.  So the ASA obviously had the information maintained in it’s state tables, but did not correctly fixup and route this packet.

If anyone know’s if this is a configuration problem or indeed a bug please let me know.  With only a single IP address to share at my house, and the need for so many ports to goto my Synology, and the need for DNS Doctoring, I figured what I was originally doing was quite fine, and in fact everything that I could tell seemed to be working except traceroute.