Tag Archives: luns

Check and list luns attached to HBA in RHEL6

This article will show you the mapping from physical HBA card to luns, I use SAN as example below, in general, it’s also applys to any other devices whichever use sysfs, for example direct sas connect.

[root@RHEL6 scsi_host]# lspci | grep Fibre
05:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
05:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
[root@RHEL6 scsi_host]# lspci -v -s 05:00.0
05:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
        Subsystem: Hewlett-Packard Company Device 338f
        Physical Slot: 1
        Flags: bus master, fast devsel, latency 0, IRQ 40
        Memory at f7ff0000 (64-bit, non-prefetchable) [size=4K]
        Memory at f7fe0000 (64-bit, non-prefetchable) [size=16K]
        I/O ports at 5000 [size=256]
        [virtual] Expansion ROM at f1700000 [disabled] [size=256K]
        Capabilities: [58] Power Management version 3
        Capabilities: [60] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [78] MSI-X: Enable- Count=32 Masked-
        Capabilities: [84] Vital Product Data
        Capabilities: [94] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [12c] Power Budgeting <?>
        Kernel driver in use: lpfc
        Kernel modules: lpfc

[root@RHEL6 scsi_host]# ls -l /sys/class/scsi_host
total 0
lrwxrwxrwx. 1 root root 0 Jun 20 18:17 host0 -> ../../devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/scsi_host/host0
lrwxrwxrwx. 1 root root 0 Jun 20 18:17 host1 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.0/host1/scsi_host/host1
lrwxrwxrwx. 1 root root 0 Jun 20 18:17 host2 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.1/host2/scsi_host/host2

[root@RHEL6 scsi_host]# find /sys/class/pci_bus/0000\:05/device/0000\:05\:00.0/host*/rport-*/target*/*/state | awk -F'/' '{print $11}' | sort
1:0:0:0
1:0:0:10
1:0:0:11
1:0:0:12
1:0:0:13
1:0:0:14
1:0:0:15
1:0:0:16
1:0:0:33
1:0:1:0
1:0:1:10
1:0:1:11
1:0:1:12
1:0:1:13
1:0:1:14
1:0:1:15
1:0:1:16
1:0:1:33
1:0:2:0
1:0:2:31
1:0:2:32
1:0:2:33
1:0:2:34
1:0:3:0
1:0:3:1
1:0:3:2
1:0:3:33
1:0:4:0
1:0:4:31
1:0:4:32
1:0:4:33
1:0:4:34
1:0:5:0
1:0:5:1
1:0:5:2
1:0:5:33
1:0:5:34
[root@RHEL6 scsi_host]# find /sys/class/pci_bus/0000\:05/device/0000\:05\:00.1/host*/rport-*/target*/*/state | awk -F'/' '{print $11}' | sort
2:0:0:0
2:0:0:1
2:0:0:2
2:0:1:0
2:0:1:10
2:0:1:11
2:0:1:12
2:0:1:13
2:0:1:14
2:0:1:15
2:0:1:16
2:0:1:33
2:0:2:0
2:0:2:31
2:0:2:32
2:0:2:33
2:0:2:34
2:0:3:0
2:0:3:10
2:0:3:11
2:0:3:12
2:0:3:13
2:0:3:14
2:0:3:15
2:0:3:16
2:0:3:33
2:0:4:0
2:0:4:31
2:0:4:32
2:0:4:33
2:0:4:34
2:0:5:0
2:0:5:1
2:0:5:2
2:0:5:33
[root@RHEL6 scsi_host]# cat /proc/scsi/scsi | grep scsi2
Host: scsi2 Channel: 00 Id: 02 Lun: 00
Host: scsi2 Channel: 00 Id: 02 Lun: 31
Host: scsi2 Channel: 00 Id: 02 Lun: 32
Host: scsi2 Channel: 00 Id: 04 Lun: 00
Host: scsi2 Channel: 00 Id: 04 Lun: 31
Host: scsi2 Channel: 00 Id: 04 Lun: 32
Host: scsi2 Channel: 00 Id: 05 Lun: 00
Host: scsi2 Channel: 00 Id: 05 Lun: 01
Host: scsi2 Channel: 00 Id: 05 Lun: 02
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Host: scsi2 Channel: 00 Id: 00 Lun: 01
Host: scsi2 Channel: 00 Id: 00 Lun: 02
Host: scsi2 Channel: 00 Id: 01 Lun: 00
Host: scsi2 Channel: 00 Id: 01 Lun: 10
Host: scsi2 Channel: 00 Id: 01 Lun: 11
Host: scsi2 Channel: 00 Id: 01 Lun: 12
Host: scsi2 Channel: 00 Id: 01 Lun: 13
Host: scsi2 Channel: 00 Id: 01 Lun: 14
Host: scsi2 Channel: 00 Id: 01 Lun: 15
Host: scsi2 Channel: 00 Id: 01 Lun: 16
Host: scsi2 Channel: 00 Id: 03 Lun: 00
Host: scsi2 Channel: 00 Id: 03 Lun: 10
Host: scsi2 Channel: 00 Id: 03 Lun: 11
Host: scsi2 Channel: 00 Id: 03 Lun: 12
Host: scsi2 Channel: 00 Id: 03 Lun: 13
Host: scsi2 Channel: 00 Id: 03 Lun: 14
Host: scsi2 Channel: 00 Id: 03 Lun: 15
Host: scsi2 Channel: 00 Id: 03 Lun: 16
Host: scsi2 Channel: 00 Id: 05 Lun: 33
Host: scsi2 Channel: 00 Id: 04 Lun: 33
Host: scsi2 Channel: 00 Id: 03 Lun: 33
Host: scsi2 Channel: 00 Id: 01 Lun: 33
Host: scsi2 Channel: 00 Id: 02 Lun: 33
Host: scsi2 Channel: 00 Id: 02 Lun: 34
Host: scsi2 Channel: 00 Id: 04 Lun: 34
[root@RHEL6 scsi_host]# cat /proc/scsi/scsi | grep scsi1
Host: scsi1 Channel: 00 Id: 02 Lun: 00
Host: scsi1 Channel: 00 Id: 02 Lun: 31
Host: scsi1 Channel: 00 Id: 02 Lun: 32
Host: scsi1 Channel: 00 Id: 04 Lun: 00
Host: scsi1 Channel: 00 Id: 04 Lun: 31
Host: scsi1 Channel: 00 Id: 04 Lun: 32
Host: scsi1 Channel: 00 Id: 05 Lun: 00
Host: scsi1 Channel: 00 Id: 05 Lun: 01
Host: scsi1 Channel: 00 Id: 05 Lun: 02
Host: scsi1 Channel: 00 Id: 03 Lun: 00
Host: scsi1 Channel: 00 Id: 03 Lun: 01
Host: scsi1 Channel: 00 Id: 03 Lun: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Host: scsi1 Channel: 00 Id: 00 Lun: 10
Host: scsi1 Channel: 00 Id: 00 Lun: 11
Host: scsi1 Channel: 00 Id: 00 Lun: 12
Host: scsi1 Channel: 00 Id: 00 Lun: 13
Host: scsi1 Channel: 00 Id: 00 Lun: 14
Host: scsi1 Channel: 00 Id: 00 Lun: 15
Host: scsi1 Channel: 00 Id: 00 Lun: 16
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Host: scsi1 Channel: 00 Id: 01 Lun: 10
Host: scsi1 Channel: 00 Id: 01 Lun: 11
Host: scsi1 Channel: 00 Id: 01 Lun: 12
Host: scsi1 Channel: 00 Id: 01 Lun: 13
Host: scsi1 Channel: 00 Id: 01 Lun: 14
Host: scsi1 Channel: 00 Id: 01 Lun: 15
Host: scsi1 Channel: 00 Id: 01 Lun: 16
Host: scsi1 Channel: 00 Id: 05 Lun: 33
Host: scsi1 Channel: 00 Id: 00 Lun: 33
Host: scsi1 Channel: 00 Id: 01 Lun: 33
Host: scsi1 Channel: 00 Id: 03 Lun: 33
Host: scsi1 Channel: 00 Id: 04 Lun: 33
Host: scsi1 Channel: 00 Id: 02 Lun: 33
Host: scsi1 Channel: 00 Id: 05 Lun: 34
Host: scsi1 Channel: 00 Id: 02 Lun: 34
Host: scsi1 Channel: 00 Id: 04 Lun: 34
[root@RHEL6 scsi_host]# find   /sys/class/pci_bus/0000\:05/device/0000\:05\:00.0/host*/rport-*/target*/*/block/*/stat | awk -F'/' '{print $11,$13}'
1:0:0:0 sdb
1:0:0:10 sdc
1:0:0:11 sdd
1:0:0:12 sde
1:0:0:13 sdf
1:0:0:14 sdg
1:0:0:15 sdh
1:0:0:16 sdi
1:0:1:0 sdj
1:0:1:10 sdk
1:0:1:11 sdl
1:0:1:12 sdm
1:0:1:13 sdn
1:0:1:14 sdo
1:0:1:15 sdp
1:0:1:16 sdq
1:0:2:0 sdr
1:0:2:31 sds
1:0:2:32 sdt
1:0:2:33 sdbi
1:0:2:34 sdbk
1:0:3:0 sdu
1:0:3:1 sdv
1:0:3:2 sdw
1:0:4:0 sdx
1:0:4:31 sdy
1:0:4:32 sdz
1:0:4:33 sdbh
1:0:4:34 sdbl
1:0:5:0 sdaa
1:0:5:1 sdab
1:0:5:2 sdac
[root@RHEL6 scsi_host]# find   /sys/class/pci_bus/0000\:05/device/0000\:05\:00.1/host*/rport-*/target*/*/block/*/stat | awk -F'/' '{print $11,$13}'
2:0:0:0 sdad
2:0:0:1 sdae
2:0:0:2 sdaf
2:0:1:0 sdag
2:0:1:10 sdah
2:0:1:11 sdai
2:0:1:12 sdaj
2:0:1:13 sdak
2:0:1:14 sdal
2:0:1:15 sdam
2:0:1:16 sdan
2:0:2:0 sdao
2:0:2:31 sdap
2:0:2:32 sdaq
2:0:2:33 sdbg
2:0:2:34 sdbj
2:0:3:0 sdar
2:0:3:10 sdas
2:0:3:11 sdat
2:0:3:12 sdau
2:0:3:13 sdav
2:0:3:14 sdaw
2:0:3:15 sdax
2:0:3:16 sday
2:0:4:0 sdaz
2:0:4:31 sdba
2:0:4:32 sdbb
2:0:4:33 sdbf
2:0:4:34 sdbm
2:0:5:0 sdbc
2:0:5:1 sdbd
2:0:5:2 sdbe
[root@RHEL6 scsi_host]# udevadm info --query=path --name /dev/sdad
/devices/pci0000:00/0000:00:03.0/0000:05:00.1/host2/rport-2:0-2/target2:0:0/2:0:0:0/block/sdad
[root@RHEL6 scsi_host]# df -h
Filesystem          Size  Used Avail Use% Mounted on
/dev/sda3            50G   20G   27G  43% /
tmpfs                64G 1012K   64G   1% /dev/shm
/dev/sda1           485M   40M  420M   9% /boot
/dev/sda6            92G  9.9G   77G  12% /home
/dev/sda5           7.9G  147M  7.4G   2% /tmp
/dev/mapper/mpathb   99G   17G   82G  18% /oracle
/dev/mapper/mpathc  985G  907G   78G  93% /data01
/dev/mapper/mpathd  985G  911G   74G  93% /data02
/dev/mapper/mpathe  985G  942G   43G  96% /data03
/dev/mapper/mpathf  985G  933G   52G  95% /data04
/dev/mapper/mpathn  985G  920G   65G  94% /data05
/dev/mapper/mpathh  985G  927G   58G  95% /data06
/dev/mapper/mpathi  985G  937G   48G  96% /data07
/dev/mapper/mpathj  985G  895G   90G  91% /data08
/dev/mapper/mpatho  985G  966G   19G  99% /data09
/dev/mapper/mpathg  985G  828G  157G  85% /data10
/dev/mapper/mpathp  985G  545G  441G  56% /data11
/dev/mapper/mpathq  985G   87M  985G   1% /data12

[root@RHEL6 scsi_host]# for port in /sys/class/fc_host/host[0-9]/port_name; { echo -n "$port : "; cat $port; }
/sys/class/fc_host/host1/port_name : 0x1000a0481ce4f1da
/sys/class/fc_host/host2/port_name : 0x1000a0481ce4f1db
[root@RHEL6 scsi_host]#

First, use lspci get HBA card  info installed on the host

# lspci | grep Fibre
15:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
15:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

HBA detail info

# lspci -v -s 15:00.0
15:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
    Subsystem: QLogic Corp. Device 015d
    Physical Slot: 2
    Flags: bus master, fast devsel, latency 0, IRQ 24
    I/O ports at 2200 [size=256]
    Memory at 97b00000 (64-bit, non-prefetchable) [size=16K]
    Expansion ROM at 90000000 [disabled] [size=256K]
    Capabilities: [44] Power Management version 3
    Capabilities: [4c] Express Endpoint, MSI 00
    Capabilities: [88] MSI: Enable- Count=1/32 Maskable- 64bit+
    Capabilities: [98] Vital Product Data
    Capabilities: [a0] MSI-X: Enable+ Count=2 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [138] Power Budgeting <?>
    Kernel driver in use: qla2xxx
    Kernel modules: qla2xxx

It tells you that one HBA card in PCI slot2, two FC ports

Find pci slot and scsi_host mapping

# ls -l /sys/class/scsi_host
total 0
...
lrwxrwxrwx 1 root root 0 Oct  9 12:58 host4 -> ../../devices/pci0000:00/0000:00:1f.5/host4/scsi_host/host4
lrwxrwxrwx 1 root root 0 Oct  9 12:58 host5 -> ../../devices/pci0000:00/0000:00:03.0/0000:15:00.0/host5/scsi_host/host5
lrwxrwxrwx 1 root root 0 Oct  9 12:58 host6 -> ../../devices/pci0000:00/0000:00:03.0/0000:15:00.1/host6/scsi_host/host6

You can easily tell that the first port of pcs slot 2(15:00.0) mapped to host5, the other mapped to host6

Find target luns by HBA port

Once you know the pci info of a HBA card, then you can find its port0 target luns or SAN devices.

Note 15:00.0 is used in this case

#find /sys/class/pci_bus/0000\:15/device/0000\:15\:00.0/host*/rport-*/target*/*/state | awk -F'/' '{print $11}' | sort
...
5:0:0:0
5:0:0:1
5:0:0:10
5:0:0:11
5:0:0:2
5:0:0:3
5:0:0:31
5:0:0:4
5:0:0:5
...

It should be consistent with the devices in /proc/scsi/scsi

#cat /proc/scsi/scsi | grep scsi5
...
Host: scsi5 Channel: 00 Id: 04 Lun: 04
Host: scsi5 Channel: 00 Id: 04 Lun: 05
Host: scsi5 Channel: 00 Id: 04 Lun: 06
Host: scsi5 Channel: 00 Id: 04 Lun: 07
Host: scsi5 Channel: 00 Id: 04 Lun: 08
Host: scsi5 Channel: 00 Id: 04 Lun: 09
Host: scsi5 Channel: 00 Id: 04 Lun: 10
Host: scsi5 Channel: 00 Id: 04 Lun: 11
Host: scsi5 Channel: 00 Id: 04 Lun: 31
...

Note: if use the command for sas direct attached devices, change ‘rport’ to ‘port’, same applies to the example below.

Find block devices

If you are only interested in block devices, like tape drive, disk lun or cd rom, here is a way similar.

# find   /sys/class/pci_bus/0000\:15/device/0000\:15\:00.0/host*/rport-*/target*/*/block/*/stat | awk -F'/' '{print $11,$13}'
5:0:0:0 sdb
5:0:0:1 sdc
5:0:0:10 sdl
5:0:0:11 sdm
5:0:0:2 sdd
5:0:0:3 sde
5:0:0:4 sdf
5:0:0:5 sdg
5:0:0:6 sdh

Reverse search, find the physical port that a lun connected to

/proc/scsi/scsi doesn’t tell you which physical port target luns are connected to In the reverse look, for a given device name, for example /dev/sdd, how do I know which hba port it connected to?

# udevadm info --query=path --name /dev/sdd
/devices/pci0000:00/0000:00:03.0/0000:15:00.0/host5/rport-5:0-0/target5:0:0/5:0:0:2/block/sdd

Is is clear?

Or
Multipath also can tell you some hint

multipath -ll | grep sdd
  `- 5:0:0:2  sdd  8:48    active ready running

Or, look into /dev/disk/by-path/ tree
...
lrwxrwxrwx 1 root root 10 Aug 15 16:49 /dev/disk/by-path/pci-0000:15:00.1-fc-0x22430080e524ebac-lun-4 -> ../../sdcx
lrwxrwxrwx 1 root root 10 Aug 15 16:49 /dev/disk/by-path/pci-0000:15:00.1-fc-0x22430080e524ebac-lun-5 -> ../../sdcy

Get HBA WWNA info:

# for port in /sys/class/fc_host/host[0-9]/port_name; { echo -n "$port : "; cat $port; }
/sys/class/fc_host/host5/port_name : 0x21000024ff3434e4
/sys/class/fc_host/host6/port_name : 0x21000024ff3434e5

Dynamically insert and remove SCSI devices

If a newer kernel and the /proc file system is running, a non-busy device can be removed and installed ‘on the fly’.

To hot remove a SCSI device:

    echo 1 > /sys/class/scsi_device/h:c:t:l/device/delete
    or
    echo 1 > /sys/block/<dev>/device/delete
    where <dev> is like sda or sdb etc..
    old way
    echo "scsi remove-single-device a b c d" > /proc/scsi/scsi

and similar, to hot add a SCSI device, do

    echo "c t l" >  /sys/class/scsi_host/host<h>/scan
    or use wildcard like below
    echo "- - -" > /sys/class/scsi_host/host<h>/scan

    old way
    echo "scsi add-single-device h c t l" > /proc/scsi/scsi

where

          h == hostadapter id (first one being 0)
          c == SCSI channel on hostadapter (first one being 0)
          t == ID
          l == LUN (first one being 0)

How to Rescan new LUN’s added in Linux, HP-UX, Aix, Solaris ?

HP-UX

1. Rescan the devices:

ioscan -fnC <disk|tape>

2. Generate device files:

 insf -e

3. Verify the new devices:

 ioscan -funC <disk|tape>

AIX

1. Rescan the devices ):

 cfgmgr -vl fcsx

Where x is FC adapter number

2. Verify the new devices:

 lsdev -Cc <disk|tape>

Linux

The rescan in Linux is HBA-specific.

For QLogic:

echo scsi-qlascan > /proc/scsi/qla<model#>/<adapter instance>

For Emulex:

 sh force_lpfc_scan.sh lpfc<adapter-instance>

For each identified device, run the following:

echo scsi add-single-device <host> <channel> <ID> <lun> >   /proc/scsi/scsi

Solaris

1. Determine the FC channels:

 cfgadm -al

2. Force rescan :

 cfgadm -o force_update -c configure cx

Where x is the FC channel number

3. Force rescan at HBA port level:

 luxadm -e forcelip /dev/fc/fpx

4. Force rescan on all FC devices:

 cfgadm -al -o show_FCP_dev

5. Install device files:

 devfsadm

6. Display all Qlogic HBA ports

 luxadm -e port

7. Display HBA port information

 luxadm -v display <WWPN>

8. Display HBA port information

 luxadm -e dump_map

Notes If one specific SANclient is missing a drive, please verify that your zoning is correct. Please also make sure the host initiator and VTL™s target ports are showing online via the Fibre Channel switch. (Check HBA link light and check the cable.)

 

Solaris Cleaning up the Operating System device tree after removing LUNs

To clean up the device tree after you remove LUNs

  1. The removed devices show up as drive not available in the output of the format command:
    413. c3t5006016841e02f0Cd252 <drive not available>
            /pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w5006016841e02f0c,fc
  2. After the LUNs are unmapped using Array management or the command line, Solaris also displays the devices as either unusable or failing.
    bash-3.00# cfgadm -al -o show_SCSI_LUN | grep -i unusable
    
    c6::5006016141e02f08,0         disk         connected    configured   unusable
    c6::5006016141e02f08,1         disk         connected    configured   unusable
    c6::5006016141e02f08,2         disk         connected    configured   unusable
    c6::5006016141e02f08,3         disk         connected    configured   unusable
    c6::5006016141e02f08,4         disk         connected    configured   unusable
    c6::5006016141e02f08,5         disk         connected    configured   unusable
    c6::5006016141e02f08,6         disk         connected    configured   unusable
    c6::5006016141e02f08,7         disk         connected    configured   unusable
    c6::5006016141e02f08,8         disk         connected    configured   unusable
    c6::5006016141e02f08,9         disk         connected    configured   unusable
    c6::5006016141e02f08,10        disk         connected    configured   unusable
    c6::5006016141e02f08,11        disk         connected    configured   unusable
    c6::5006016841e02f08,0         disk         connected    configured   unusable
    c6::5006016841e02f08,1         disk         connected    configured   unusable

    bash-3.00# cfgadm -al -o show_SCSI_LUN | grep -i failing
      c2::5006016841e02f03,71    disk  connected configured  failing
      c3::5006016841e02f0c,252   disk  connected configured  failing
  3. If the removed LUNs show up as failing, you need to force a LIP on the HBA. This operation probes the targets again, so that the device shows up as unusable. Unless the device shows up as unusable, it cannot be removed from the device tree.
    luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl
  4. To remove the device from the cfgadm database, run the following commands on the HBA:
    cfgadm -c unconfigure -o unusable_SCSI_LUN c2::5006016841e02f03 cfgadm -c unconfigure -o unusable_SCSI_LUN c3::5006016841e02f0c 
  5. Repeat step 2 to verify that the LUNs have been removed.
  6. Clean up the device tree. The following command removes the /dev/rdsk… links to /devices.
    $devfsadm -Cv