Using IPS Network Simulator and SAN Extension Tuner

The Cisco MDS 9000 supports a feature known as IPS Network Simulator.  This feature is available on MDS switches equipped with the IPS-4 (DS-X9304) and IPS-8 (DS-X9308) network modules, MSM module (DS-X9304-18K9), as well as the Multiservice Modular Switch 9222i.  What this feature does is allow you to simulate the conditions of a network, controlling attributes such as delay, bandwidth and packet reordering.  This is very helpful when you are trying to test things like replication in a lab scenario before you go into production.  You can think of it as a sort of network shim that sits between two of the Gigabit Ethernet ports.  One port is the input to the network simulator and the other port is the output.  Each of the various parameters of the network can be controlled for each direction.  In order to use the IPS Network Simulator features you must have the SAN_EXTN_OVER_IP or SAN_EXTN_OVER_IP_IPS4 licenses installed.

The simulator works off the Gigabit Ethernet ports, so is not available to test pure fibre channel.  However it is perfect for testing FCIP, which is a common SAN extension technology that can be used over very long distances, typically with asynchronous replication.  The simulator can handle Gigabit traffic at full line rate.  You are also able to test additional FCIP options such as compression and view the effect they would have over the network with the given parameters.

In addition to simulating a network, the MDS also has the ability to simulate I/O.  You can use the SAN Extension Tuner application to generate SCSI reads and writes as well as other useful commands.

Here is a diagram of the topology we will be using

Configuration of Gigabit Ethernet ports

First we will configure the Gigabit Ethernet ports on both MDS1 and MDS2:

MDS1(config)#fcip enable
Enter configuration commands, one per line. End with CNTL/Z.
MDS1(config)# int GigabitEthernet2/2
MDS1(config-if)# ip address 192.168.10.1 255.255.255.0
MDS1(config-if)# no shut
MDS1(config-if)# exit
MDS1(config)# exit

MDS2(config)#fcip enable
Enter configuration commands, one per line. End with CNTL/Z.
MDS2(config)# int GigabitEthernet2/2
MDS2(config-if)# ip address 192.168.10.2 255.255.255.0
MDS2(config-if)# no shut
MDS2(config-if)# exit
MDS2(config)# exit

Configuration of FCIP interfaces

Next, we will do a basic configuration of FCIP by creating a tunnel between MDS1 and MDS2. This tunnel will not come up as we have not yet established connectivity between ports GigabitEthernet2/2 on MDS1 and GigabitEthernet2/2 on MDS2. Once we configure the IPS Network Simulator the ports will be connected. We will use VSAN100 for our testing.

MDS1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDS1(config)# fcip profile 1
MDS1(config-profile)# ip address 192.168.10.1
MDS1(config-profile)# exit
MDS1(config)# exit
MDS1(config)# int fcip1
MDS1(config-if)# use-profile 1
MDS1(config-if)# peer-info ipaddr 192.168.10.2
MDS1(config-if)# switchport trunk allowed vsan 100
MDS1(config-if)# exit
MDS1(config)# exit

MDS2# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDS2(config)# fcip profile 1
MDS2(config-profile)# ip address 192.168.10.2
MDS2(config-profile)# exit
MDS2(config)# exit
MDS2(config)# int fcip1
MDS2(config-if)# use-profile 1
MDS2(config-if)# peer-info ipaddr 192.168.10.1
MDS2(config-if)# switchport trunk allowed vsan 100
MDS2(config-if)# exit
MDS2(config)# exit

Configuration of IPS Network Simulator

The Gigabit Ethernet and FCIP configurations are mostly complete. We will add some additional parameters later, but enough has been configured for basic connectivity. Now we must configure the actual IPS Network Simulator. A few key points about the IPS Network Simulator first:

MDS1(config)# san-ext-tuner enable
MDS1# ips netsim enable interface g2/3 g2/4
MDS1# ips netsim delay-ms 100 ingress g2/3
MDS1# ips netsim delay-ms 100 ingress g2/4
MDS1# ips netsim max-bandwidth-mbps 10 ingress g2/3
MDS1# ips netsim max-bandwidth-mbps 10 ingress g2/4
MDS1# ips netsim qsize 250 ingress g2/3
MDS1# ips netsim qsize 250 ingress g2/4
MDS1# ips netsim drop nth 200 burst 1 ingress g2/3

At this point the FCIP tunnels should be up and operational.

Configuration of SAN Extension Tuner

The SAN Extension Tuner allows us to generate traffic between two virtual endpoints. A few key points about SET configuration:

MDS1(config)# san-ext-tuner enable
MDS1(config)# iscsi enable
MDS1(config)# iscsi enable module 2
MDS1(config)# zone default-zone permit vsan 100
MDS1(config)# int iscsi 2/2
MDS1(config-if)# no shut
MDS1# san-ext-tuner
MDS1(san-ext)# nwwn 10:00:00:00:00:00:00:00
MDS1(san-ext)# nport pwwn 11:11:11:11:11:11:11:11 vsan 100 interface g2/2
MDS1(san-ext-nport)# exit
MDS1(san-ext)# exit

MDS2(config)# san-ext-tuner enable
MDS2(config)# iscsi enable
MDS2(config)# iscsi enable module 2
MDS2(config)# zone default-zone permit vsan 100
MDS2(config)# int iscsi 2/2
MDS2(config-if)# no shut
MDS2# san-ext-tuner
MDS2(san-ext)# nwwn 20:00:00:00:00:00:00:00
MDS2(san-ext)# nport pwwn 22:22:22:22:22:22:22:22 vsan 100 interface g2/2
MDS2(san-ext-nport)# exit
MDS2(san-ext)# exit

MDS1# show san-ext-tuner nports
—————————————————————————-
Interface NODE NAME PORT NAME VSAN
—————————————————————————-
GigabitEthernet2/2 10:00:00:00:00:00:00:00 11:11:11:11:11:11:11:11 100

MDS2# show san-ext-tuner nports
—————————————————————————-
Interface NODE NAME PORT NAME VSAN
—————————————————————————-
GigabitEthernet2/2 20:00:00:00:00:00:00:00 22:22:22:22:22:22:22:22 100

At this point the SET is not really doing anything but its setup and in place for us to start generating some tests. You can test your connectivity by using fcping:

MDS1# fcping pwwn 22:22:22:22:22:22:22:22 vsan 100
28 bytes from 22:22:22:22:22:22:22:22 time = 699973 usec
28 bytes from 22:22:22:22:22:22:22:22 time = 335159 usec
28 bytes from 22:22:22:22:22:22:22:22 time = 670161 usec
28 bytes from 22:22:22:22:22:22:22:22 time = 201197 usec
28 bytes from 22:22:22:22:22:22:22:22 time = 249109 usec

5 frames sent, 5 frames received, 0 timeouts
Round-trip min/avg/max = 201197/431119/699973 usec

Notice the ridiculously high ping time we get. This is because with IPS Network Simulator we have created 200ms of RTT, packet drops, manipulated the qsize and reduced the bandwidth to 1% (10Mbps) of the actual link speed.

Lets generate some traffic:

MDS1# san-ext-tuner
MDS1(san-ext)# nport pwwn 11:11:11:11:11:11:11:11 vsan 100 int g2/2
MDS1(san-ext-nport)# read command-id 1 target 22:22:22:22:22:22:22:22 transfer-size 256000 outstanding-ios 2 continuous

MDS2# san-ext-tuner
MDS2(san-ext)# nport pwwn 22:22:22:22:22:22:22:22 vsan 100 int g2/2
MDS2(san-ext-nport)# read command-id 1 target 11:11:11:11:11:11:11:11 transfer-size 256000 outstanding-ios 2 continuous

We can view statistics from both IPS Network Simulator as well as SET:

MDS1# show san-ext-tuner interface g2/2 nport pwwN 11:11:11:11:11:11:11:11 vsan 100 counters
Statistics for nport
Node name 10:00:00:00:00:00:00:00 Port name 11:11:11:11:11:11:11:11
I/Os per sec : 5
Reads : 100%
Writes : 0%
Egress throughput : 0.75 MBs/sec (Max – 2.50 MBs/sec)
Ingress throughput : 0.69 MBs/sec (Max – 2.50 MBs/sec)
Average response time : Read – 279561 us, Write – 0 us
Minimum response time : Read – 404 us, Write – 0 us
Maximum response time : Read – 3112761 us, Write – 0 us
Errors : 10

MDS1# show ips stats netsim ingress g2/3
Network Simulator Configuration for Ingress on GigabitEthernet2/3
Delay : 100000 microseconds
Rate : 10000 kbps
Max_q : 250000 bytes
Max_qdelay : 150000000 clocks
Drop nth pkt : 200

Network Simulator Statistics for Ingress on GigabitEthernet2/3
Dropped (tot) = 35875
Dropped (netsim) = 3592
Reordered (netsim) = 0
Max Qlen(pkt) = 542
Qlen (pkt) = 83
Max Qlen (byte) = 5602
Qlen (byte) = 0
Mintxdel(poll) = 39600
Mintxdel(ethtx) = 39600
empty = 233
txdel = 603235
late = 111330
Average speed = 1586 Kbps

Now we will enable compression and see the effect is has on our traffic:

MDS1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDS1(config)# int fcip1
MDS1(config-if)# ip-compression mode3
MDS1(config-if)# exit
MDS1(config)# exit

MDS2# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDS2(config)# int fcip1
MDS2(config-if)# ip-compression mode3
MDS2(config-if)# exit
MDS2(config)# exit

MDS1# show san-ext-tuner interface g2/2 nport pwwN 11:11:11:11:11:11:11:11 vsan 100 counters
Statistics for nport
Node name 10:00:00:00:00:00:00:00 Port name 11:11:11:11:11:11:11:11
I/Os per sec : 16
Reads : 100%
Writes : 0%
Egress throughput : 2.00 MBs/sec (Max – 2.50 MBs/sec)
Ingress throughput : 2.00 MBs/sec (Max – 2.50 MBs/sec)
Average response time : Read – 134971 us, Write – 0 us
Minimum response time : Read – 404 us, Write – 0 us
Maximum response time : Read – 3112761 us, Write – 0 us
Errors : 14

Network Simulator Statistics for Ingress on GigabitEthernet2/3
Dropped (tot) = 36375
Dropped (netsim) = 500
Reordered (netsim) = 0
Max Qlen(pkt) = 542
Qlen (pkt) = 0
Max Qlen (byte) = 2236
Qlen (byte) = 0
Mintxdel(poll) = 36000
Mintxdel(ethtx) = 36000
empty = 523
txdel = 70351
late = 28586
Average speed = 914 Kbps

You can see that adding compression has greatly improved the speed on the link. What is shown in the netsim statistics is the actual links speed. What is shown in the SET statistics is the compressed data being moved.

FCIP Interface TCP Parameters

When configuring FCIP parameters, you want to do so based on the network performance. SET can help you find these things. In our example its 10MB bandwidth and 200ms round trip time:

MDS1(config)# fcip profile 1
MDS1(config-profile)# tcp max-bandwidth-mbps 10 min-available-bandwidth-mbps 10 round-trip-time-ms 200
MDS2(config)# fcip profile 1
MDS2(config-profile)# tcp max-bandwidth-mbps 10 min-available-bandwidth-mbps 10 round-trip-time-ms 200