Data Replication Using FCIP; Brocade 7800 SAN switches

Replication Using FCIP & Brocade 7800 SAN switches

I was involved for a project for one of our customers that involved configuring remote replication between two Sun 6180 storage boxes, one located in Cairo and the other located in Port Said (which is 200km away).

We had two brocade 7800 SAN switches. The two sites are connected via a WAN link.

The WAN link (if you are not familiar with it), is usually a long distance link between two routers. These routers are connected via a medium that supports the required distance; for example an E1 link, or a bundle of E1′s to increase bandwidth. The result is two network subnets connected via the routers. The diagram below illustrates how network clients on both sides of the routers can reach each other via regular network routing.

FC over IP = FCIP

SAN replication uses the FC protocol. To allow the FC protocol to travel over the WAN link we will use the FCIP protocol. FCIP will encapsulate the FC packets into IP packets which can then be routed over the long distance network between the two sites.

Sending large amounts of data over a WAN link usually includes sizing. The WAN connection must have enough bandwidth to accommodate the traffic. Proper sizing has to be performed such that the link can complete the data transfer complete in a timely fashion according to the environment requirements.

Connections & Configuration

Connecting the 6180′s to the SAN switch is pretty straight forward, just connect the replication port of the 6180 to one of the FC ports on the SAN switch.

Configuring the FCIP tunnel is where all the work is. The entire FCIP configuration has to be performed on both SAN switches via command line. The configuration should match on both sides, otherwise it will not work.

First we need to configure the network ports on the SAN switch, this will just define the IP addresses of the FC network ports. Don’t get this mixed up with the management port and IP, this is a different port and different configuration.

Once the network ports have been configured we then need to define the network routes. This way the packets know which router to use to reach the remote subnet.

Finally, we configure the tunnel. The tunnel defines the source and target IPs in addition to any other options required for the connection, such as IPSEC, fastwrite, compression, etc… On the brocade 7800 switch, the FCIP tunnel is designated a virtual port number. This port is not a physical port, it is a virtual port used to identify and configure the tunnel. The first FCIP port on the 7800 is port 16, which we will be using.

Connection Diagram

The following diagram illustrates the target configuration;

Command Syntax

portcfg ipif create

portcfg iproute create

portcfg fciptunnel 16 create

Step-by-Step Configuration

To configure the tunnel as shown in the configuration diagram, log on to the Main Switch and run the following commands;

This following command configures the physical ge0 port with;
IP: 192.168.1.100
subnet: 255.255.255.0
MTU: 1500

SAN-MAIN:admin> portcfg ipif ge0 create 192.168.1.100 255.255.255.0 1500

Now configure port ge0 such that;
To reach 192.168.2.0/24 (the remote SAN switch FC network)  use the router 192.168.1.1

SAN-MAIN:admin> portcfg iproute ge0 create 192.168.2.0 255.255.255.0 192.168.1.1

Finally, create the FCIP tunnel 16 with target 192.168.2.100 and source 192.168.1.100

SAN-MAIN:admin> portcfg fciptunnel 16 create 192.168.2.100 192.168.1.100 10000

By running the above three commands, the configuration on the MAIN SAN switch is done. We now need to configure the other side of the tunnel on the DR switch;

SAN-DR:admin> portcfg ipif ge0 create 192.168.2.100 255.255.255.0 1500

SAN-DR:admin> portcfg iproute ge0 create 192.168.1.0 255.255.255.0 192.168.2.1

SAN-DR:admin> portcfg fciptunnel 16 create 192.168.1.100 192.168.2.100 10000

The configuration is now complete, the two 6180′s should be able to communicate as if they were connected to the same SAN switch.

Finally, additional configuration such as fast write, compression and IPSEC security can be configured by modifying the tunnel configuration. Note that this will disrupt the FCIP traffic for a few seconds while the reconfiguration completes. Again, this has to be done on both SAN switches otherwise the tunnel will not come up.

Syntax for the options we used is as follows;

portcfg fciptunnel modify -f 1 -c 1 -i 1 -K <32-byte key="">

To apply this to our example, the 32-byte key is in hexadecimal, and can be anything as long as it is the same on both sides.

portcfg fciptunnel 16 modify -f 1 -c 1 -i 1 -K AABBCCDDEEFF112233445566778899AA

You can also limit the bandwidth used by the tunnel. In this example I am limiting the tunnel to use 10Mbps.

SAN-MAIN:admin> portCfg fciptunnel 16 modify -b 10000 -B 10000

!!!! WARNING !!!!

Modify operation can disrupt the traffic on the fciptunnel specified for a brief period of time. This operation will bring the existing tunnel down (if tunnel is up) before applying new configuration.

Continue with Modification (Y,y,N,n): [ n]    y

Circuit 16.0 modify: Operation Succeeded

Viewing Your Configuration

The following output show the commands and example output of a configured tunnel;

Viewing the ge interface configuration

SAN-DR:admin> portshow ipif ge0

Port: ge0

Interface IPv4 Address    NetMask         Effective MTU  Flags

--------------------------------------------------------------

    0     10.3.202.241    255.255.255.0   1500           U R M

Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running

       N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp

SAN-DR:admin> portshow iproute ge0

Port: ge0

IP Address      Mask            Gateway        Metric   Flags

-------------------------------------------------------------

10.2.3.0           255.255.255.0      10.3.202.150        0    U G S

10.3.202.0         255.255.255.0      *                   0    U C

10.3.202.150       255.255.255.255    *                   0    U H L

Flags: U=Usable G=Gateway H=Host C=Created(Interface) S=Static L=LinkLayer(Arp)

Viewing the FCIP tunnel configuration

SAN-MAIN:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

 Tunnel Circuit  OpStatus Flags   Uptime   TxMBps   RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

 16     -         Up      cf---   11m13s     0.00     0.00    4      -      -

-------------------------------------------------------------------------------

  Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf

        circuit: s=sack

SAN-MAIN:admin> portshow fciptunnel all -c

-------------------------------------------------------------------------------

 Tunnel Circuit  OpStatus Flags   Uptime   TxMBps   RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

 16     -         Up      cf---   11m27s     0.00     0.00    4      -      -

 16     0 ge0     Up      ----s   11m27s     0.00     0.00    4  1000/1000  0

-------------------------------------------------------------------------------

  Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf

        circuit: s=sack

SAN-DR:admin> portshow fciptunnel 16

-------------------------------------------

Tunnel ID: 16

   Tunnel Description:

   Admin Status: Enabled

   Oper Status: Up

   Compression: On (Standard)

   Fastwrite: On

   Tape Acceleration: Off

   TPerf Option: Off

   IPSec: Enabled

   Remote WWN: Not Configured

   Local WWN: 10:00:00:05:33:d1:ac:c2

   Peer WWN: 10:00:00:05:33:a3:73:fa

   Circuit Count: 1

   Flags: 0x00000000

   FICON: Off

SAN-DR:admin> portshow fciptunnel 16 -c

-------------------------------------------

Tunnel ID: 16

   Tunnel Description:

   Admin Status: Enabled

   Oper Status: Up

   Compression: On (Standard)

   Fastwrite: On

   Tape Acceleration: Off

   TPerf Option: Off

   IPSec: Enabled

   Remote WWN: Not Configured

   Local WWN: 10:00:00:05:33:d1:ac:c2

   Peer WWN: 10:00:00:05:33:a3:73:fa

   Circuit Count: 1

   Flags: 0x00000000

   FICON: Off

   -------------------------------------------

   Circuit ID: 16.0

      Circuit Num: 0

      Admin Status: Enabled

      Oper Status: Up

      Remote IP: 10.2.3.241

      Local IP: 10.3.202.241

      Metric: 0

      Min Comm Rt: 10000

      Max Comm Rt: 10000

      SACK: On

      Min Retrans Time: 100

      Max Retransmits: 8

      Keepalive Timeout: 10000

      Path MTU Disc: 0

      VLAN ID: (Not Configured)

      L2CoS: (VLAN Not Configured)

      DSCP:  F:  0 H:  0 M:  0 L:  0

      Flags: 0x00000000

Deleting FCIP tunnel

If you want to change the IP addresses of the ge interfaces you will need to delete the entire configuration and start over.

The following sequence of commands will delete the FCPIP tunnel, the route and IP address assigned to the FC network interface. The tunnel is FC IP is dependent on the route, and the route is dependent on the tunnel. The commands are arranged by dependency, so if you want to delete a components, you must also delete all the components above it.

portcfg fciptunnel delete

portcfg iproute delete

portcfg ipif delete