MegaCli64 Command Usage

MegaCli64 Command Usage Cheat Sheet

Here is a MegaCli64 PDF cheat sheet created by Moritz Mertinkat.

Get Information from All Physical Disks

The -a command line switch specifies the Array ID, in this case All Array IDs.

# ./MegaCli64 -PdList -aAll

Get Virtual Disk Information

The -a command line switch specifies the Array ID, in this case All Array IDs.

# ./MegaCli64 -LDInfo -Lall -aAll

Get the Number of Virtual Drives Configured on Each Adapter

The -a command line switch specifies the Array ID, in this case All Array IDs.

# ./MegaCli64 -LdGetNum -aAll

Get the Number of Array Controllers

# ./MegaCli64 -adpCount

Blink Hard Drive Lights

Start Blinking

The -a command line switch specifies the Array ID, in this case Array ID 0.

# ./MegaCli64 -PdLocate -start -physdrv[:] -a0

Stop Blinking

The -a command line switch specifies the Array ID, in this case Array ID 0.

# ./MegaCli64 -PdLocate -stop -physdrv[] -a0

Set Dedicated Hot Spare

The following physical drive was Unconfigured(good):

Enclosure Device ID: 4
Slot Number: 16
Device Id: 154
Sequence Number: 1
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
Raw Size: 140014MB [0x11177328 Sectors]
Non Coerced Size: 139502MB [0x11077328 Sectors]
Coerced Size: 139392MB [0x11040000 Sectors]
Firmware state: Unconfigured(good)
SAS Address(0): 0x500000e117951c52
SAS Address(1): 0x0
Connected Port Number: 0(path0) 
Inquiry Data: FUJITSU MBE2147RC       D905D304PB30AAGJ

When looking at all of the physical drives by running MegaCli64 -PdList -aAll, Array #: 0, 1, 2, 3, 4 were missing a hotspare.
The following command added the Unconfigured(good) physical drive above as a dedicated hotspare:

The -a command line switch specifies the Array ID, in this case Array ID 0.

# ./MegaCli64 -PDHSP -Set -Dedicated -Array0,1,2,3,4 -PhysDrv [4:16] -a0

RDAC Driver RPM install

RDAC Driver RPM

|Updated September 18, 2013 by JASON WO|Tags: None

Page Actions

This rpm is a highly modified version of the release available from http://pkgs.org/centos-5-rhel-5/puias-computational-x86_64/kmod-rdac-09.03.0C05.0331-0.4.PU_IAS.5.x86_64.rpm.html which was designed for RHEL6. Modifications were made to port it RHEL5 to fit our environment and allow support for both RHEL5/RHEL6.
The original rpm provides kmod rdac kernel modules compiled in a kernel agnostic manner against the stable ABI meaning as long as the ABI doesnt change from one kernel release to the next, then there is no need to recompile/rebuild the rpm.
The rpm dependencies and actions have been modified to add missing files back in and have the rpm automatically perform post install configuration and automatically detect the installation of new kernels and perform reconfiguration as necessary without any manual intervention.
DO NOT install this rpm on a box where you have previous manually compiled/installed the rdac drivers. The rpm must be installed on a box that hasn't yet had rdac drivers installed on it.
1. When do the drivers/rpm need to be recompiled/upgraded?

The rpms will need to be rebuilt any time there is a change in the kernel ABI (kABI) or when you want to upgrade to a new driver version. More information on how the kernel ABI is used can be gained by googling for "Redhat Driver Update Program" and you should find a few pdfs and driverupdateprogram.com as references.

2. How do I check if a new kernel's ABI is supported by the current rdac kernel module rpms?

This is done by validating the hashes of the ABI groups provided in the rpm to that of the kernel (note that this should be handled by rpm automatically but its a good pre kernel upgrade manual test to ensure they are compatible:

# rpm --requires -qp kmod-rdac-09.03.0C05.0504-25.el5.x86_64.rpm | grep kernel
kernel(rhel5_drivers_base_ga) = 61dc730b8ca5e74017f2df5b55ecda8b7df7f9c2
kernel(rhel5_vmlinux_ga) = 78f928da689a93ecf2e044fc0ced6b3eaedf5c19
kernel(rhel5_drivers_scsi_ga) = 1cc16b1f8996d37eaa858690bfbab7f4030c55b6
kernel(rhel5_kernel_ga) = 84d69198cf51b494e38d9d0a54e52607c8a507e2
kernel(rhel5_mm_ga) = d5edc1b3d2a4f2bf8ce28d7f4dbeab27cfeb19bd
kernel(rhel5_lib_ga) = ff25b583d6d314edd98f7c9533c5867194b3d30d
kernel(rhel5_fs_proc_ga) = 30f9166e128d20c7305d5a9bc9ab69451c40a555
kernel(rhel5_block_ga) = 5b4effd1cc3809b4bd243e499ab6be486fd95fd9
kernel(rhel5_fs_ga) = 1c422a6b84a2000991b1b99c61506cfd711a20d1
kernel(rhel5_fs_u4) = 518e7b7963ed0843e4b55b64ba2b25db95cf821a
kernel(rhel5_arch_x86_64_kernel_ga) = 880dbfce5086d666f5bab6ad642c0323fcdabd90
kernel(rhel5_drivers_xen_core_ga) = c05a8027c47f037b99169c7441da64fa0a723869
kernel(rhel5_kernel_module_ga) = a74a9d2bf87d13d6b9412698dc2728248ca92523

# rpm --provides -q kernel-2.6.18-308.13.1.el5 | grep "kernel("
kernel(rhel5_gfs2_ga) = 73e828ddf6a8787f935f6088a18a862409174aaf
kernel(rhel5_fs_lockd_ga) = 67913a0875a3219e6b8c2d914775c91fab5b7c4f
kernel(rhel5_drivers_hwmon_ga) = 21462092aeb2b284e955337136d204c26f79ed92
kernel(rhel5_net_core_ga) = c186a7dc043c903564c2dd9ed49d8847b7043c86
.... 

# for modDep in $(rpm --requires -qp kmod-rdac-09.03.0C05.0504-25.el5.x86_64.rpm | grep kernel); do if ! /bin/grep "${modDep}" <<< "$(rpm --provides -q kernel-2.6.18-308.13.1.el5)" > /dev/null; then echo "ERROR: NOT FOUND ${modDep}"; fi done

3. What steps do I need to perform before installing the rpms on a system which already has the rdac drivers installed manually from source?

  1. Unmount all partitions that are using the rdac driver (and comment them out in /etc/fstab if needed) and unmap all LUNs.
  2. Remove all MPP kernel boot entries from /boot/grub/grub.conf
  3. Remove all MPP init images:
     /bin/rm -f /boot/mpp-*.img
  4. Remove all lines from /etc/modprobe.conf and /etc/modprobe.conf.local that match the lines in /opt/mpp/modprobe.conf.mppappend
  5. Run the uninstall script:
     make -f /opt/mpp/makefile.saved uninstall
  6. Reboot the box
  7. Install the rdac rpms per normal instructions below.
  8. Re-enable any mount points and map any needed LUNs.
  9. Reboot the box

4. What steps do I need to perform to rebuild the rpms?

  1. Get the source using git directly from the repo, or manually browse to the git server and download the tar.gz for the most recent date listed:
     git clone http://10.19.10.164/wdprepos/pub/UnixTools/Linux/rdac-kmod.git
     cd rdac-kmod

     or

     Browse to http://10.19.10.164/gitweb/?p=pub/UnixTools/Linux/rdac-kmod.git;a=summary and click the topmost "tar.gz" to download the latest version and extract it on the build server.
   2. Ensure you have gcc, make, buildsys-macros and rpm-build rpms installed:
      yum --noplugins --disablerepo=* --enablerepo=R* install gcc make buildsys-macros rpm-build
   3. Run the following as a regular user:
      ./rdac-kmod-rpmbuild.sh
   4. Stage the rpms in ~/rpmbuild/RPMS// in the tools repo (do not stage the debug rpm) and reindex the tools repo:
      See instructions for staging rpms at https://w3.tap.ibm.com/w3ki/display/wdpdo/Staging%20RPMs%20to%20ITE%20Tools%20Repository
   5. Use the normal yum install or upgrade command to install (see step 5 below).

5. What steps do I need to perform to install the rpms?

yum --disablerepo=* --enablerepo=WDP-Tools* install kmod-rdac rdac

6. Will the current rdac-kmod.spec file work for new versions of the rdac driver?

There's no way to know before hand whether the existing spec file will work for future versions of the rdac driver without modification as this is 100% dependent on what changes are made to the make file for the new release. You will have to review changes that are made to the copyfiles, copyrpmfiles, moduledep, setupfiles, setupdriver, uninstall, and uninstall_doer sections and ensure the same steps are done in the proper sections of the spec file but only if necessary.

7. I have installed the rdac rpms and will be doing a kernel upgrade, what steps do I need to perform?

If you have done the pre-upgrade check in step 2 and no errors are reported, then just upgrade the kernel as normal and there are no extra steps, the rpm should handle the rest.

SAN swtich Firmware update

Updating SAN switch Firmware -

Swtich A:roo> firmwaredownload

Server Name or IP Address: XXX.XXX.XXX.XXX

User Name: USERNAME

File Name: /home/net/tsaini/firmware/v7.0.2c

Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 3

Verifying if the public key authentication is available.Please wait ...

The public key authentication is not available.

Password:

Server IP: XX.XX.XX.XX, Protocol IPv4

Checking system settings for firmwaredownload...

System settings check passed.

You can run firmwaredownloadstatus to get the status

of this command.

This command will cause a warm/non-disruptive boot but will

require that existing telnet, secure telnet or SSH sessions

be restarted.

It will cause GigE/FC Fast-write traffic to be disrupted.

Do you want to continue (Y/N) [Y]: y

Firmware is being downloaded to the switch. This step may take up to 30 minutes.

Preparing for firmwaredownload...

Start to install packages...

dir                         ##################################################

ldconfig                    ##################################################

glibc                       ##################################################

glibc-linuxthreads          ##################################################

bash                        ##################################################

readline                    ##################################################

terminfo                    ##################################################

termcap                     ##################################################

vixie-cron                  ##################################################

fileutils                   ##################################################

textutils                   ##################################################

setup                       ##################################################

swbd21-setup                ##################################################

which                       ##################################################

findutils                   ##################################################

bzip                        ##################################################

zlib                        ##################################################

chkconfig                   ##################################################

sed                         ##################################################

procps                      ##################################################

psmisc                      ##################################################

modutils                    ##################################################

sin                         ##################################################

rcinit                      ##################################################

misc                        ##################################################

pam                         ##################################################

util-linux                  ##################################################

sh-utils                    ##################################################

popt                        ##################################################

grep                        ##################################################

rpm                         ##################################################

sysvinit                    ##################################################

man                         ##################################################

less                        ##################################################

gzip                        ##################################################

tar                         ##################################################

rsync                       ##################################################

uuid-libs                   ##################################################

e2fsprogs                   ##################################################

telnet                      ##################################################

cpio                        ##################################################

dev                         ##################################################

bootenv                     ##################################################

wdtd                        ##################################################

fwdl                        ##################################################

telnet-server               ##################################################

kernel                      ##################################################

kernel-module-usb           ##################################################

kernel-module-nf            ##################################################

swbd21-drivers              ##################################################

sysklogd                    ##################################################

getty                       ##################################################

net-tools                   ##################################################

uucp                        ##################################################

portmap                     ##################################################

inetd                       ##################################################

iptables                    ##################################################

tcpd                        ##################################################

rsh-server                  ##################################################

rsh                         ##################################################

openssl-libs                ##################################################

openssh                     ##################################################

openssh-server              ##################################################

rusers-server               ##################################################

rdate                       ##################################################

logrotate                   ##################################################

ntp                         ##################################################

pciutils                    ##################################################

strace                      ##################################################

sendmail                    ##################################################

iproute2                    ##################################################

libxml2                     ##################################################

fss                         ##################################################

fabos-setup                 ##################################################

fabos-drivers               ##################################################

fabos-libs                  ##################################################

fabos-diag                  ##################################################

fabos                       ##################################################

fabos-daemons               ##################################################

fabos-zoning                ##################################################

sqlite                      ##################################################

dhcpcd                      ##################################################

fabos-routing               ##################################################

fabos-hmon                  ##################################################

fabos-wwnhs                 ##################################################

fabos-man                   ##################################################

fabos-swbd83                ##################################################

apache                      ##################################################

fastcgi                     ##################################################

fabos-webtools              ##################################################

fabos-webtoolsez            ##################################################

tz                          ##################################################

mtracer-tool                ##################################################

sysstat                     ##################################################

ossimage                    ##################################################

bpimage-swbd85              ##################################################

prom-440epx                 ##################################################

Please avoid powering off the system during prom update.

ipv6                        ##################################################

awk                         ##################################################

ipsec                       ##################################################

Removing unneeded files, please wait ...

Finished removing unneeded files.

All packages have been downloaded successfully.

Firmware has been downloaded to the secondary partition of the switch.

HA Rebooting ...

Data Replication Using FCIP; Brocade 7800 SAN switches

Replication Using FCIP & Brocade 7800 SAN switches

I was involved for a project for one of our customers that involved configuring remote replication between two Sun 6180 storage boxes, one located in Cairo and the other located in Port Said (which is 200km away).

We had two brocade 7800 SAN switches. The two sites are connected via a WAN link.

The WAN link (if you are not familiar with it), is usually a long distance link between two routers. These routers are connected via a medium that supports the required distance; for example an E1 link, or a bundle of E1′s to increase bandwidth. The result is two network subnets connected via the routers. The diagram below illustrates how network clients on both sides of the routers can reach each other via regular network routing.

FC over IP = FCIP

SAN replication uses the FC protocol. To allow the FC protocol to travel over the WAN link we will use the FCIP protocol. FCIP will encapsulate the FC packets into IP packets which can then be routed over the long distance network between the two sites.

Sending large amounts of data over a WAN link usually includes sizing. The WAN connection must have enough bandwidth to accommodate the traffic. Proper sizing has to be performed such that the link can complete the data transfer complete in a timely fashion according to the environment requirements.

Connections & Configuration

Connecting the 6180′s to the SAN switch is pretty straight forward, just connect the replication port of the 6180 to one of the FC ports on the SAN switch.

Configuring the FCIP tunnel is where all the work is. The entire FCIP configuration has to be performed on both SAN switches via command line. The configuration should match on both sides, otherwise it will not work.

First we need to configure the network ports on the SAN switch, this will just define the IP addresses of the FC network ports. Don’t get this mixed up with the management port and IP, this is a different port and different configuration.

Once the network ports have been configured we then need to define the network routes. This way the packets know which router to use to reach the remote subnet.

Finally, we configure the tunnel. The tunnel defines the source and target IPs in addition to any other options required for the connection, such as IPSEC, fastwrite, compression, etc… On the brocade 7800 switch, the FCIP tunnel is designated a virtual port number. This port is not a physical port, it is a virtual port used to identify and configure the tunnel. The first FCIP port on the 7800 is port 16, which we will be using.

Connection Diagram

The following diagram illustrates the target configuration;

Command Syntax

portcfg ipif create

portcfg iproute create

portcfg fciptunnel 16 create

Step-by-Step Configuration

To configure the tunnel as shown in the configuration diagram, log on to the Main Switch and run the following commands;

This following command configures the physical ge0 port with;
IP: 192.168.1.100
subnet: 255.255.255.0
MTU: 1500

SAN-MAIN:admin> portcfg ipif ge0 create 192.168.1.100 255.255.255.0 1500

Now configure port ge0 such that;
To reach 192.168.2.0/24 (the remote SAN switch FC network)  use the router 192.168.1.1

SAN-MAIN:admin> portcfg iproute ge0 create 192.168.2.0 255.255.255.0 192.168.1.1

Finally, create the FCIP tunnel 16 with target 192.168.2.100 and source 192.168.1.100

SAN-MAIN:admin> portcfg fciptunnel 16 create 192.168.2.100 192.168.1.100 10000

By running the above three commands, the configuration on the MAIN SAN switch is done. We now need to configure the other side of the tunnel on the DR switch;

SAN-DR:admin> portcfg ipif ge0 create 192.168.2.100 255.255.255.0 1500

SAN-DR:admin> portcfg iproute ge0 create 192.168.1.0 255.255.255.0 192.168.2.1

SAN-DR:admin> portcfg fciptunnel 16 create 192.168.1.100 192.168.2.100 10000

The configuration is now complete, the two 6180′s should be able to communicate as if they were connected to the same SAN switch.

Finally, additional configuration such as fast write, compression and IPSEC security can be configured by modifying the tunnel configuration. Note that this will disrupt the FCIP traffic for a few seconds while the reconfiguration completes. Again, this has to be done on both SAN switches otherwise the tunnel will not come up.

Syntax for the options we used is as follows;

portcfg fciptunnel modify -f 1 -c 1 -i 1 -K <32-byte key="">

To apply this to our example, the 32-byte key is in hexadecimal, and can be anything as long as it is the same on both sides.

portcfg fciptunnel 16 modify -f 1 -c 1 -i 1 -K AABBCCDDEEFF112233445566778899AA

You can also limit the bandwidth used by the tunnel. In this example I am limiting the tunnel to use 10Mbps.

SAN-MAIN:admin> portCfg fciptunnel 16 modify -b 10000 -B 10000

!!!! WARNING !!!!

Modify operation can disrupt the traffic on the fciptunnel specified for a brief period of time. This operation will bring the existing tunnel down (if tunnel is up) before applying new configuration.

Continue with Modification (Y,y,N,n): [ n]    y

Circuit 16.0 modify: Operation Succeeded

Viewing Your Configuration

The following output show the commands and example output of a configured tunnel;

Viewing the ge interface configuration

SAN-DR:admin> portshow ipif ge0

Port: ge0

Interface IPv4 Address    NetMask         Effective MTU  Flags

--------------------------------------------------------------

    0     10.3.202.241    255.255.255.0   1500           U R M

Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running

       N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp

SAN-DR:admin> portshow iproute ge0

Port: ge0

IP Address      Mask            Gateway        Metric   Flags

-------------------------------------------------------------

10.2.3.0           255.255.255.0      10.3.202.150        0    U G S

10.3.202.0         255.255.255.0      *                   0    U C

10.3.202.150       255.255.255.255    *                   0    U H L

Flags: U=Usable G=Gateway H=Host C=Created(Interface) S=Static L=LinkLayer(Arp)

Viewing the FCIP tunnel configuration

SAN-MAIN:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

 Tunnel Circuit  OpStatus Flags   Uptime   TxMBps   RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

 16     -         Up      cf---   11m13s     0.00     0.00    4      -      -

-------------------------------------------------------------------------------

  Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf

        circuit: s=sack

SAN-MAIN:admin> portshow fciptunnel all -c

-------------------------------------------------------------------------------

 Tunnel Circuit  OpStatus Flags   Uptime   TxMBps   RxMBps ConnCnt CommRt  Met

-------------------------------------------------------------------------------

 16     -         Up      cf---   11m27s     0.00     0.00    4      -      -

 16     0 ge0     Up      ----s   11m27s     0.00     0.00    4  1000/1000  0

-------------------------------------------------------------------------------

  Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf

        circuit: s=sack

SAN-DR:admin> portshow fciptunnel 16

-------------------------------------------

Tunnel ID: 16

   Tunnel Description:

   Admin Status: Enabled

   Oper Status: Up

   Compression: On (Standard)

   Fastwrite: On

   Tape Acceleration: Off

   TPerf Option: Off

   IPSec: Enabled

   Remote WWN: Not Configured

   Local WWN: 10:00:00:05:33:d1:ac:c2

   Peer WWN: 10:00:00:05:33:a3:73:fa

   Circuit Count: 1

   Flags: 0x00000000

   FICON: Off

SAN-DR:admin> portshow fciptunnel 16 -c

-------------------------------------------

Tunnel ID: 16

   Tunnel Description:

   Admin Status: Enabled

   Oper Status: Up

   Compression: On (Standard)

   Fastwrite: On

   Tape Acceleration: Off

   TPerf Option: Off

   IPSec: Enabled

   Remote WWN: Not Configured

   Local WWN: 10:00:00:05:33:d1:ac:c2

   Peer WWN: 10:00:00:05:33:a3:73:fa

   Circuit Count: 1

   Flags: 0x00000000

   FICON: Off

   -------------------------------------------

   Circuit ID: 16.0

      Circuit Num: 0

      Admin Status: Enabled

      Oper Status: Up

      Remote IP: 10.2.3.241

      Local IP: 10.3.202.241

      Metric: 0

      Min Comm Rt: 10000

      Max Comm Rt: 10000

      SACK: On

      Min Retrans Time: 100

      Max Retransmits: 8

      Keepalive Timeout: 10000

      Path MTU Disc: 0

      VLAN ID: (Not Configured)

      L2CoS: (VLAN Not Configured)

      DSCP:  F:  0 H:  0 M:  0 L:  0

      Flags: 0x00000000

Deleting FCIP tunnel

If you want to change the IP addresses of the ge interfaces you will need to delete the entire configuration and start over.

The following sequence of commands will delete the FCPIP tunnel, the route and IP address assigned to the FC network interface. The tunnel is FC IP is dependent on the route, and the route is dependent on the tunnel. The commands are arranged by dependency, so if you want to delete a components, you must also delete all the components above it.

portcfg fciptunnel delete

portcfg iproute delete

portcfg ipif delete