Optimizing the Cisco MDS PAA with Wireshark

The Cisco Port Analyzer Adapter (PAA) is a great tool and very helpful for troubleshooting.  It comes in two types, the PAA and the PAA-2.  The PAA’s have a Gigabit port onboard that can be used in 100MB or 1000MB mode.  You really want to always use the 1000MB mode.  Fibre Channel today is 1, 2, 4 or 8 GB.  So to take a SPAN of that much traffic and try to put it on a wire at 100MB isn’t very practical.  It’s OK at 1000MB because in many cases you can do port captures at less busy times or in a lab to see what is going on.

The PAA has a few different modes and these are all explained in the Cisco documentation.  The documentation is available from Cisco at Port Analyzer Configuration Note.  The default mode is DTM which means Deep Truncate Mode.  This allows for the most frames per second because it truncates the payload.  As you know a FC frame is 2112 plus or minus a bit and a typical ethernet MTU is about 1500.  So by truncating the packet it is put into a format that can go nicely on an ethernet wire and be captured into Wireshark. Wireshark is also called the Cisco Protocol Analyzer.  The disadvantage is that now you have truncated the frame and lost data.  We are not just talking FC payload data, when we say “data” we mean potentially parts of the headers within the FC packet.  This means you lose information that may be essential in troubleshooting.

The standard PAA has no way of accounting for the size of this truncated data, so your bandwidth/traffic information is wrong when its brought into Wireshark.  If you use the PAA2 it inserts a special field called “Original Packet Length” under the Boardwalk header which keeps the original packet length before truncation in tact for reporting purposes. NTOP, also called Cisco Traffic Analyzer, is used for traffic analysis, and is almost useless for good traffic studies without either a PAA-2 or capturing the full amount of traffic.

DTM will work with 100, 1000 or 2000 MB, but obviously as you go up in frame size and speed you may drop packets.  I personally like to run what is called NTM which is No Truncate Mode, as this mode does not truncate.  You can’t run this mode unless your NIC supports Jumbo Frames.  You can usually set the Jumbo Frames under the NIC Properties like so:

You should set this to match your frame size.  I use standard 2112 Fibre Channel size so I went with a 3KB MTU.  So if you use NTM you should typically use a 1000MB Ethernet speed with Jumbo frames.

If you are set to the default DTM mode, you will Deep Truncate.  As you can read in the Cisco docs, this means that only 64-bytes of the FC frame are kept, which is likely to eat into your headers.  Examine the below capture where you can see the Auth_Negotiate packet ends after the Protocol Parameters Length field:

There are actually quite a few more fields, but these are truncated so you do not see all of the FC-SP fields and parameters.  This can hurt your troubleshooting.  In many cases, not all, but most, you really don’t care so much about the actual payload data of the FC frame.  What you want is the headers.  DTM will likely cause you some truncation of headers which is bad.  So any mode that you can use that keeps more data than DTM is a good idea in my opinion.  I just goto NTM because that way I know its going to have everything and I am generally not spanning 4GB links anyways, most of what I do is in a controlled lab.  If you can’t do NTM, then do ETM, if you can’t do ETM do STM and if you have to do DTM.  Make sure if your using FC frames larger than 1496 you use Jumbo Frames on your Ethernet NIC.  Look at the result when we use NTM instead of DTM:

Notice how after the Protocol Parameters Length we have many more fields that can be helpful in troubleshooting.  For example we can see what hash options were offered in the Auth_Negotiate and what Diffe-Hellman groups were supported.

I generally recommend using the latest version of Wireshark and libpcap which you can get from www.wireshark.org.  There is no benefit to getting the program from the Cisco site and it’s likely to be outdated. You will also almost certainly want to enable encapsulation eisl on the SD port you configure to capture traffic. This will insert VSAN information into the Boardwalk header so in Wireshark and NTOP you can get VSAN information.