fcsp re-authentication failures when Port VSAN not allowed on Port-Channel

Update: This has been confirmed as new Cisco bugID CSCti31428, and at least effects 3.3(3) and 3.3(5). What is basically happening is that the DHCHAP SW.ILS traffic is using the port VSAN. Unless you allow the port VSAN on the trunk, it will not work. It is not always guaranteed that the port VLAN is implicitly allowed on a trunk. Proper behavior would be that DHCHAP re-authentication traffic uses VSAN 4094, which is used between MDS switches for control traffic. This behavior is confirmed using the PAA-2 and Wireshark. When the MDS is using a regular ISL, behavior is correct, it uses control VSAN 4094 for the DHCHAP re-authentication traffic. When the MDS is using a PortChannel, it incorrectly places the DHCHAP re-authentication traffic into the port VSAN. If this VSAN is not allowed on the PortChannel trunk then re-authentication fails.

I have found an issue with FCSP for which I have an open TAC case.  I am awaiting to find out if this is a new bug and I believe it is.  Roman Rodichev CCIE #7927 had reported in his CCIE Storage study materials an issue with FCSP re-authentication, so I looked a bit further and eventually opened a TAC case.

The issue happens when you have at least the following:

1. One side of a link is in “auto-active” with a re-authentication time set.

2. The other side of the link is in auto-passive.

3. The interfaces which are part of this link are in a port channel for which the native/port vsan is not being allowed.

As you may know, trunks have a Port VSAN.  The Port VSAN is normally not an issue, that is its fine to not allow the port VSAN explicitly, much the same way where as in LAN switching you may make a trunk but not explicitly allow the native VLAN.

Here is an example of a configuration for one side of the link:

MDS1# show run int po1
version 3.3(5)

interface port-channel 1
fspf cost 100 vsan 20
switchport speed 1000
switchport mode E
no shutdown
channel mode active
switchport trunk allowed vsan add 5
switchport trunk allowed vsan add 20
switchport trunk allowed vsan add 30

MDS1# show run int fc1/9
version 3.3(5)

interface fc1/9
switchport speed 1000
switchport mode E
channel-group 1 force
fcsp auto-active 1
no shutdown

And here is the output of the other side of the link:

MDS2# show run int po1
version 3.3(5)

interface port-channel 1
fspf cost 100 vsan 20
switchport speed 1000
switchport mode E
no shutdown
channel mode active
switchport trunk allowed vsan add 5
switchport trunk allowed vsan add 20
switchport trunk allowed vsan add 30

MDS2# show run int fc1/9
version 3.3(5)

interface fc1/9
switchport speed 1000
switchport mode E
channel-group 1 force
fcsp auto-passive
no shutdown

So you can see the MDS1 side is set to “auto-active” with a re-authentication time of 1 minute, and the other side is auto-passive. When both sides are set to auto-active, I do not see this issue. You will also see that on the Port Channel Trunk I am allowing VSAN’s 5, 20 and 30. The port VSAN I have set for this trunk (not shown) is VSAN 2. What happens is that the initial authentication works just fine:

MDS1# show fcsp interf fc1/9-11

fc1/9:
fcsp authentication mode:SEC_MODE_AUTO_ACTIVE
reauthentication timeout (in minutes):1
Status:Successfully authenticated
Authenticated using local password database

fc1/10:
fcsp authentication mode:SEC_MODE_AUTO_ACTIVE
reauthentication timeout (in minutes):1
Status:Successfully authenticated
Authenticated using local password database

fc1/11:
fcsp authentication mode:SEC_MODE_AUTO_ACTIVE
reauthentication timeout (in minutes):1
Status:Successfully authenticated
Authenticated using local password database

After one minute however, the re-authentication fails:

2010 Aug 6 20:27:03 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/10 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )
2010 Aug 6 20:27:03 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/11 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )
2010 Aug 6 20:27:03 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/9 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )

Simply bouncing the physical interfaces you wish to authenticate will allow successful authentication since it treats it as an initial authentication, but then it will fail again in one minute:

MDS1# show fcsp interf fc1/9-11

fc1/9:
fcsp authentication mode:SEC_MODE_AUTO_ACTIVE
reauthentication timeout (in minutes):1
Status:FC-SP authentication failed

fc1/10:
fcsp authentication mode:SEC_MODE_AUTO_ACTIVE
reauthentication timeout (in minutes):1
Status:FC-SP authentication failed

fc1/11:
fcsp authentication mode:SEC_MODE_AUTO_ACTIVE
reauthentication timeout (in minutes):1
Status:FC-SP authentication failed

Here you can see with fcanalyzer, that the re-auths never make it to the other side:

MDS1(config)# fcanalyzer local brief display-filter fcsp.opcode
Warning: Couldn’t obtain netmask info (eth2: no IPv4 address assigned).
Capturing on eth2
11.900199 ff.ff.fd -> ff.ff.fd 0x6ab 0xffff SW_ILS AUTH_Negotiate
11.992773 ff.ff.fd -> ff.ff.fd 0x698 0xffff SW_ILS DHCHAP_Challenge
12.185244 ff.ff.fd -> ff.ff.fd 0x6b4 0xffff SW_ILS DHCHAP_Reply
12.193308 ff.ff.fd -> ff.ff.fd 0x6b5 0xffff SW_ILS AUTH_Negotiate
12.204058 ff.ff.fd -> ff.ff.fd 0x6b6 0xffff SW_ILS AUTH_Negotiate
12.374826 ff.ff.fd -> ff.ff.fd 0x6a2 0xffff SW_ILS DHCHAP_Challenge
12.381704 ff.ff.fd -> ff.ff.fd 0x6a3 0xffff SW_ILS DHCHAP_Success
12.474281 ff.ff.fd -> ff.ff.fd 0x6a4 0xffff SW_ILS DHCHAP_Challenge
12.556602 ff.ff.fd -> ff.ff.fd 0x6ba 0xffff SW_ILS DHCHAP_Reply
12.571106 ff.ff.fd -> ff.ff.fd 0x6bb 0xffff SW_ILS DHCHAP_Success
12.668658 ff.ff.fd -> ff.ff.fd 0x6a7 0xffff SW_ILS DHCHAP_Success
12.754110 ff.ff.fd -> ff.ff.fd 0x6bd 0xffff SW_ILS DHCHAP_Reply
12.820886 ff.ff.fd -> ff.ff.fd 0x6c0 0xffff SW_ILS DHCHAP_Success
12.958055 ff.ff.fd -> ff.ff.fd 0x6ac 0xffff SW_ILS DHCHAP_Success
13.015277 ff.ff.fd -> ff.ff.fd 0x6c3 0xffff SW_ILS DHCHAP_Success
72.763784 ff.ff.fd -> ff.ff.fd 0x6c7 0xffff SW_ILS AUTH_Negotiate
73.003558 ff.ff.fd -> ff.ff.fd 0x6c8 0xffff SW_ILS AUTH_Negotiate
73.063469 ff.ff.fd -> ff.ff.fd 0x6c9 0xffff SW_ILS AUTH_Negotiate
76.763897 ff.ff.fd -> ff.ff.fd 0x6ca 0xffff SW_ILS AUTH_Negotiate
77.004037 ff.ff.fd -> ff.ff.fd 0x6cb 0xffff SW_ILS AUTH_Negotiate
77.064184 ff.ff.fd -> ff.ff.fd 0x6cc 0xffff SW_ILS AUTH_Negotiate
80.763286 ff.ff.fd -> ff.ff.fd 0x6cd 0xffff SW_ILS AUTH_Negotiate
81.003504 ff.ff.fd -> ff.ff.fd 0x6ce 0xffff SW_ILS AUTH_Negotiate
81.063660 ff.ff.fd -> ff.ff.fd 0x6cf 0xffff SW_ILS AUTH_Negotiate
84.762893 ff.ff.fd -> ff.ff.fd 0x6d0 0xffff SW_ILS AUTH_Negotiate
85.002943 ff.ff.fd -> ff.ff.fd 0x6d1 0xffff SW_ILS AUTH_Negotiate
85.062965 ff.ff.fd -> ff.ff.fd 0x6d2 0xffff SW_ILS AUTH_Negotiate
88.762322 ff.ff.fd -> ff.ff.fd 0x6d3 0xffff SW_ILS AUTH_Negotiate
89.002531 ff.ff.fd -> ff.ff.fd 0x6d4 0xffff SW_ILS AUTH_Negotiate
89.063700 ff.ff.fd -> ff.ff.fd 0x6d5 0xffff SW_ILS AUTH_Negotiate
92.762232 ff.ff.fd -> ff.ff.fd 0x6d6 0xffff SW_ILS AUTH_Negotiate
93.002584 ff.ff.fd -> ff.ff.fd 0x6d7 0xffff SW_ILS AUTH_Negotiate
93.062299 ff.ff.fd -> ff.ff.fd 0x6d8 0xffff SW_ILS AUTH_Negotiate
96.761330 ff.ff.fd -> ff.ff.fd 0x6d9 0xffff SW_ILS AUTH_Negotiate
97.001363 ff.ff.fd -> ff.ff.fd 0x6da 0xffff SW_ILS AUTH_Negotiate
97.061330 ff.ff.fd -> ff.ff.fd 0x6db 0xffff SW_ILS AUTH_Negotiate
100.760808 ff.ff.fd -> ff.ff.fd 0x6dc 0xffff SW_ILS AUTH_Negotiate
101.000880 ff.ff.fd -> ff.ff.fd 0x6dd 0xffff SW_ILS AUTH_Negotiate
101.060861 ff.ff.fd -> ff.ff.fd 0x6de 0xffff SW_ILS AUTH_Negotiate
104.760385 ff.ff.fd -> ff.ff.fd 0x6df 0xffff SW_ILS AUTH_Negotiate
105.000525 ff.ff.fd -> ff.ff.fd 0x6e0 0xffff SW_ILS AUTH_Negotiate
105.060406 ff.ff.fd -> ff.ff.fd 0x6e1 0xffff SW_ILS AUTH_Negotiate
108.759895 ff.ff.fd -> ff.ff.fd 0x6e2 0xffff SW_ILS AUTH_Negotiate
109.000001 ff.ff.fd -> ff.ff.fd 0x6e3 0xffff SW_ILS AUTH_Negotiate
109.059927 ff.ff.fd -> ff.ff.fd 0x6e4 0xffff SW_ILS AUTH_Negotiate
2010 Aug 6 20:52:29 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/10 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )
2010 Aug 6 20:52:29 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/9 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )
2010 Aug 6 20:52:29 MDS1 %FCSP-MGR-2-FCSP_AUTHENT_FAILURE: FC-SP Authentication failure on Port fc1/11 (FC-SP Failure Reason: FCSP_AUTHENT_FAILURE )

and here is MDS2:

MDS2(config)# fcanalyzer local brief display-filter fcsp.opcode
Warning: Couldn’t obtain netmask info (eth2: no IPv4 address assigned).
Capturing on eth2
9.466279 ff.ff.fd -> ff.ff.fd 0x6ab 0xffff SW_ILS AUTH_Negotiate
9.558533 ff.ff.fd -> ff.ff.fd 0x698 0xffff SW_ILS DHCHAP_Challenge
9.751347 ff.ff.fd -> ff.ff.fd 0x6b4 0xffff SW_ILS DHCHAP_Reply
9.759394 ff.ff.fd -> ff.ff.fd 0x6b5 0xffff SW_ILS AUTH_Negotiate
9.770069 ff.ff.fd -> ff.ff.fd 0x6b6 0xffff SW_ILS AUTH_Negotiate
9.940639 ff.ff.fd -> ff.ff.fd 0x6a2 0xffff SW_ILS DHCHAP_Challenge
9.947458 ff.ff.fd -> ff.ff.fd 0x6a3 0xffff SW_ILS DHCHAP_Success
10.040026 ff.ff.fd -> ff.ff.fd 0x6a4 0xffff SW_ILS DHCHAP_Challenge
10.122701 ff.ff.fd -> ff.ff.fd 0x6ba 0xffff SW_ILS DHCHAP_Reply
10.137177 ff.ff.fd -> ff.ff.fd 0x6bb 0xffff SW_ILS DHCHAP_Success
10.234418 ff.ff.fd -> ff.ff.fd 0x6a7 0xffff SW_ILS DHCHAP_Success
10.320151 ff.ff.fd -> ff.ff.fd 0x6bd 0xffff SW_ILS DHCHAP_Reply
10.387025 ff.ff.fd -> ff.ff.fd 0x6c0 0xffff SW_ILS DHCHAP_Success
10.523860 ff.ff.fd -> ff.ff.fd 0x6ac 0xffff SW_ILS DHCHAP_Success
10.581328 ff.ff.fd -> ff.ff.fd 0x6c3 0xffff SW_ILS DHCHAP_Success

You can see from comparing the two fcanalyzer outputs that on MDS1, the last bit of data to go between the switches successfully was at timestamp 13.015277.

I was also able to do a full packet dump using my PAA-2 and provide the pcap to TAC to be analyzed. As I get more information I will post it in this thread.

This entry was posted in CCIE Storage, FCSP and tagged , , , , , . Bookmark the permalink.

Leave a Reply