An FCIP tunnel made of Ethernet Port Channels or a FC Port Channel made of FCIP tunnels, which is better?

The Cisco MDS switches support many different ways you can connect them with both fibre channel and ethernet. Some modules do not support all of the features, but they are available in the architecture. For example the MDS 9216i and 14/2 modules do not support Ethernet Port Channels.

Consider the case where you want multiple links to be used with FCIP. You have at least two different ways you can handle this:

Scenario A – An FCIP tunnel made of Ethernet Port Channels. Say you have two ethernet links, g1/1 and g1/2. You wish to combine them. You can make them into an ethernet port channel and then run an FCIP tunnel over this. This has a few disadvantages:

1. Ethernet Port Channels must be made using adjacent ports on an IPS module. For example g1/1 and g1/2 can be made into an ethernet Port Channel, but not g1/1 and g1/3.

2. Because all of the FCIP traffic is going to a specific IP destination/port, it will only take one path within the Ethernet Channel (transmit). The return path may or may not take the same link, this can be very inefficient.

3. Ethernet Port Channels are limited to two members.

4. Both Ethernet Port Channel members must goto the same switch.

5. The Ethernet Port Channel must be terminated on the adjacent ethernet switch, it cannot transit other ethernet switches.

Here is this scenario illustrated:

There is another way this could be configured:

Scenario B – An FC Port Channel made up of FCIP links. In this scenario you place each Gigabit Ethernet link into its own FCIP tunnel. These can be located anywhere in the MDS. Multiple FCIP tunnels can be aggregated into a single FC Port Channel. This has the following advantages:

1. You can have up to 16 members in a FC Port Channel. They can be from any ports on the MDS.

2. All modules support FC Port Channels.

3. Up to 128 FC Port Channels when mixing Gen-1 and Gen-2 modules, up to 256 Port Channels when no Gen-1 modules are present.

4. FSPF sees this as one big single pipe. If a member breaks it does not change the FSPF state.

5. Although the logical FC Port Channel must be from one MDS to another, the individual FCIP links may transit multiple ethernet switches and multiple separate paths.

6. MDS uses the load balancing parameter set under the VSAN. Each exchange or flow can use a different link in the FC Port Channel, making use of the entire aggregate. Each exchange or flow would be limited to a single FCIP pipe size at one time.

Here is this illustrated:

I believe that Cisco is phasing out Ethernet Port Channel support in the MDS. It really is not as useful of a feature. There are really almost no instances where I can think of that ethernet port channels make sense. Certainly when we are talking about FCIP, it makes a lot more sense to make a FC Port Channel out of FCIP links. One may argue that in the case of iSCSI making a ethernet port channel can give you 2GB/s. This may or may not be the case. If you have setup iSCSI to use a new iSCSI connection per request, then yes this is possible. But you could also accomplish the same thing using iSLB with VRRP, and at the same time add a lot more fault tolerance, easier administration and scalable aggregate bandwidth beyond 2GB.

You can do some crazy things with ethernet port channels, FCIP and FC Port channels. You can keep stacking them up making FCIP tunnels out of port channels that are made up of FCIP tunnels or port channels. It really lets you go crazy making some very confusing stuff that actually works 🙂 There is of course really no need to do this, but it probably would not be out of place to see something that crazy in the CCIE lab as its known for creating complex scenarios that don’t exactly mimic real world but are rather designed to test expert level knowledge.

If you have a good use for Ethernet Port Channels on the MDS post some comments!

This entry was posted in CCIE Storage, FCIP, FSPF and tagged , , . Bookmark the permalink.

7 Responses to An FCIP tunnel made of Ethernet Port Channels or a FC Port Channel made of FCIP tunnels, which is better?

  1. Pingback: Technology Short Take #5 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers

  2. tsanko says:

    Wonderful ..thanks a lot for posting a good informitive blog

  3. Numbers says:

    Would there be an advantage to using a FC Port Channel in the following scenario? MDS1 g1/1 to switch1a to switch1b to MDS2g1/1 and MDS1 g1/2 to switch2a to switch2b to MDS2 g1/2. Trunking the same VSANs over each FCIP link. So would you use a FC Port Channel in this scenario or would you just leave both FCIP links separate and let the MDS load balance traffic between the two FCIP links?? Thanks.

  4. Mason says:

    cisco recently released a lot of cutting edge new techs on nexus, including fabric path,

  5. Yuriy says:

    Good article…Brian, Have you ever practiced with DS-X9708-K9 (10Gb FCOE)?
    If it possible to set FCIP between DS-X9308 and DS-X9708?

    • brian says:

      I have not used the DS-X9708. My understanding of it, is that it supports FCIP so should interop just fine with any other device. Obviously since its 10Gb you can’t directly connect it to a DS-X9308 but you could go through a switch. This is based on very loose documentation, whether it actually supports FCIP I am not 100% sure of, but if it does, I am sure it would interoperate with the 9308.

      Also one point that is unclear to me, is that if the DS-X9708 requires SAN Extension over IP licensing, I have not seen any documentation that says it does or does not.

Leave a Reply