If you remove a disk from the system using rmdev -dl hdiskX
without having previously reduced the volume group to remove the disk from LVM, and thus have not updated properly the on-disk format information (called VGDA), you get a discrepancy between the ODM and the LVM configurations. Here is how to solve the issue (without any warranty though!).
What are the volume group informations:
# lsvg -p rootvg rootvg: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdisk0 active 2157 1019 174..00..00..413..432 0516-304 : Unable to find device id 00ce4b6a01292201 in the Device Configuration Database. 00ce4b6a01292201 missing 2157 1019 174..71..00..342..432 # lspv hdisk0 00ce4b6ade6da849 rootvg active hdisk2 00ce4b6a01b09b83 drakevg active hdisk3 00ce4b6afd175206 drakevg active # lsdev -Cc disk hdisk0 Available Virtual SCSI Disk Drive hdisk2 Available Virtual SCSI Disk Drive hdisk3 Available Virtual SCSI Disk Drive
As we can notice, the disk is still in the LVM configuration but doesn’t show up in the devices. To solve this issue, we need to cheat the ODM in order to be able to use LVM commands to change the LVM configuration, stored on the volume group disks. The idea is to reinsert a disk in the ODM configuration, remove the disk from LVM and then remove it from ODM. Here is how we do it. First, let’s make a copy of the ODM files that we will change:
# cd /etc/objrepos/ # cp CuAt CuAt.before_cheat # cp CuDv CuDv.before_cheat # cp CuPath CuPath.before_cheat
Now, we will extract the hdisk0
‘s definition from ODM and add it back as hdisk1
‘s definition:
# odmget -q "name=hdisk0" CuAt CuAt: name = "hdisk0" attribute = "unique_id" value = "3520200946033223609SYMMETRIX03EMCfcp05VDASD03AIXvscsi" type = "R" generic = "" rep = "n" nls_index = 0 CuAt: name = "hdisk0" attribute = "pvid" value = "00ce4b6ade6da8490000000000000000" type = "R" generic = "D" rep = "s" nls_index = 11 # odmget -q "name=hdisk0" CuDv CuDv: name = "hdisk0" status = 1 chgstatus = 2 ddins = "scsidisk" location = "" parent = "vscsi0" connwhere = "810000000000" PdDvLn = "disk/vscsi/vdisk" # odmget -q "name=hdisk0" CuPath CuPath: name = "hdisk0" parent = "vscsi0" connection = "810000000000" alias = "" path_status = 1 path_id = 0
Basically, we need to insert new entries in the three classes CuAt
, CuDv
and CuPath
with hdisk0
changed to hdisk1
. A few others attributes need to be changed. The most important one is the PVID, located in CuAt
. We will use the value reported as missing by lsvg -p rootvg
. Attribute unique_id
also need to be changed. You can just change a few characters in the existing string, it just need to be unique in the system. The other attributes to change are connwhere
in CuDv
and connection
in CuPath
. Their value represent the LUN ID of the disk. Again, this value is not relevant, it just have to be unique. We can check the current LUN defined by running lscfg
on all the disks defined:
# lscfg -vl hdisk* hdisk0 U9117.570.65E4B6A-V6-C2-T1-L810000000000 Virtual SCSI Disk Drive hdisk2 U9117.570.65E4B6A-V6-C3-T1-L810000000000 Virtual SCSI Disk Drive hdisk3 U9117.570.65E4B6A-V6-C3-T1-L820000000000 Virtual SCSI Disk Drive
LUN 81 is used on controller C2 and LUNs 81 and 82 on C3. Let’s choose 85, which for sure will not collide with other devices. The following commands will generate the text files that will be used to cheat the ODM, according to what was just explained:
# mkdir /tmp/cheat # cd /tmp/cheat # odmget -q "name=hdisk0" CuAt | sed -e 's/hdisk0/hdisk1/g' -e 's/00ce4b6ade6da849/00ce4b6a01292201/' -e 's/609SYMMETRIX/719SYMMETRIX/' > hdisk1.CuAt # odmget -q "name=hdisk0" CuDv | sed -e 's/hdisk0/hdisk1/' -e 's/810000000000/850000000000/' > hdisk1.CuDv # odmget -q "name=hdisk0" CuPath | sed -e 's/hdisk0/hdisk1/' -e 's/810000000000/850000000000/' > hdisk1.CuPAth
Let’s look at the generated files:
# cat hdisk1.CuAt CuAt: name = "hdisk1" attribute = "unique_id" value = "3520200946033223719SYMMETRIX03EMCfcp05VDASD03AIXvscsi" type = "R" generic = "" rep = "n" nls_index = 0 CuAt: name = "hdisk1" attribute = "pvid" value = "00ce4b6a012922010000000000000000" type = "R" generic = "D" rep = "s" nls_index = 11 # cat hdisk1.CuDv CuDv: name = "hdisk1" status = 1 chgstatus = 2 ddins = "scsidisk" location = "" parent = "vscsi0" connwhere = "850000000000" PdDvLn = "disk/vscsi/vdisk" # cat hdisk1.CuPath CuPath: name = "hdisk1" parent = "vscsi0" connection = "850000000000" alias = "" path_status = 1 path_id = 0
So, we are ready to insert the data in the ODM:
# odmadd hdisk1.CuAt # odmadd hdisk1.CuDv # odmadd hdisk1.CuPath # lsvg -p rootvg rootvg: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdisk0 active 2157 1019 174..00..00..413..432 hdisk1 missing 2157 1019 174..71..00..342..432
The disk is now back in ODM! Now, to remove the disk from the VGDA, we use the reducevg
command:
# reducevg rootvg hdisk1 0516-016 ldeletepv: Cannot delete physical volume with allocated partitions. Use either migratepv to move the partitions or reducevg with the -d option to delete the partitions. 0516-884 reducevg: Unable to remove physical volume hdisk1.
We will use the -d
flag to remove the physical partitions associated to each logical volumes and located hdisk1. A few lines have been remove to simplify listing…
# reducevg -d rootvg hdisk1 0516-914 rmlv: Warning, all data belonging to logical volume lv01 on physical volume hdisk1 will be destroyed. rmlv: Do you wish to continue? y(es) n(o)? y 0516-304 putlvodm: Unable to find device id 00ce4b6a012922010000000000000000 in the Device Configuration Database. 0516-896 reducevg: Warning, cannot remove physical volume hdisk1 from Device Configuration Database. # lsvg -l rootvg rootvg: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT hd5 boot 2 2 1 closed/syncd N/A hd6 paging 256 256 1 open/syncd N/A hd8 jfs2log 1 1 1 open/syncd N/A hd4 jfs2 7 7 1 open/syncd / hd2 jfs2 384 384 1 open/syncd /usr hd9var jfs2 64 64 1 open/syncd /var hd3 jfs2 128 128 1 open/syncd /tmp hd1 jfs2 2 2 1 open/syncd /home hd10opt jfs2 32 32 1 open/syncd /opt fslv04 jfs2 256 256 1 open/syncd /usr/sys/inst.images loglv01 jfslog 1 1 1 closed/syncd N/A lv01 jfs 5 5 1 closed/syncd /mkcd/cd_images # lsvg -p rootvg rootvg: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdisk0 active 2157 1019 174..00..00..413..432
The disk has been deleted from the VGDA. What about ODM?
# lsdev -Cc disk hdisk0 Available Virtual SCSI Disk Drive hdisk1 Available Virtual SCSI Disk Drive hdisk2 Available Virtual SCSI Disk Drive hdisk3 Available Virtual SCSI Disk Drive # rmdev -dl hdisk1 Method error (/etc/methods/ucfgdevice): 0514-043 Error getting or assigning a minor number.
We probably forgot to cheat one ODM class… Never mind: let’s remove the cheat we added to ODM and see what appends:
# odmdelete -o CuAt -q "name=hdisk1" 2 objects deleted # lspv hdisk0 00ce4b6ade6da849 rootvg active hdisk2 00ce4b6a01b09b83 drakevg active hdisk1 none None hdisk3 00ce4b6afd175206 drakevg active # rmdev -dl hdisk1 Method error (/etc/methods/ucfgdevice): 0514-043 Error getting or assigning a minor number. # odmdelete -o CuDv -q "name=hdisk1" 1 objects deleted # lspv hdisk0 00ce4b6ade6da849 rootvg active hdisk2 00ce4b6a01b09b83 drakevg active hdisk3 00ce4b6afd175206 drakevg active # lspath Enabled hdisk0 vscsi0 Enabled hdisk2 vscsi0 Enabled hdisk2 vscsi1 Enabled hdisk3 vscsi1 Enabled hdisk3 vscsi0 Unknown hdisk1 vscsi0 # odmdelete -o CuPath -q "name=hdisk1" 1 objects deleted # lspath Enabled hdisk0 vscsi0 Enabled hdisk2 vscsi0 Enabled hdisk2 vscsi1 Enabled hdisk3 vscsi1 Enabled hdisk3 vscsi0
That’s it! Use with care.