How to Upgrade IBM Power server firmware fixes through AIX or Linux without an HMC

Installing server firmware fixes through the operating system is a disruptive process. You will need to restart the system.

Notes:

  1. If your system is managed by an HMC, you must apply server firmware through the HMC. For details, see Managed system updates in Updates.
  2. If you have a System i® model running IBM® i, you must either apply server firmware through an HMC or through an IBM i logical partition. If you have a POWER6® Power Systems™ server that is managed by an HMC, you must use the HMC.
  3. By default, the server firmware is installed on the temporary side only after the existing contents of the temporary side are permanently installed on the permanent side. (This process is performed automatically when you install a server firmware fix.)
  4. If you are unable to start your AIX or Linux operating system or server, refer to Obtaining fixes through AIX or Linux when you are unable to start the system.

Perform Steps 1 through 6 to get server firmware fixes through AIX or Linux when you do not have an HMC.

Step 1. View existing firmware levels for AIX or Linux

The Advanced System Management Interface (ASMI) is the user interface to access the server firmware. You can also use the AIX or Linux operating system to view the firmware levels.
  1. Select from the following options:
    • To use the ASMI (AIX or Linux): On the ASMI Welcome pane, view the existing level of server firmware in the upper-right corner below the copyright statement, for example, EM310_006.
    • To use the AIX command prompt (you must have AIX diagnostics installed on your server), continue with step 2.
    • To use the Linux command prompt, continue with step 4.
  2. At an AIX command prompt, enter the following command:
    lsmcode
    The existing levels of server firmware are displayed. For example, you might see output similar to the following:

    DISPLAY MICROCODE LEVEL                                                   802811
    IBM,8231-E1C
    
    The current permanent system firmware image is AL740_088
    The current temporary system firmware image is AL740_088
    The system is currently booted from the temporary firmware image.
    
    Use Enter to continue.
    Notes:

    • The permanent level is also known as the backup level.
    • The temporary level is also known as the installed level.
    • The system was booted from the temporary side, so at this time, the temporary level is also the activated level.
  3. Continue with Step 2. View or download the firmware fix.
  4. To view existing levels of server firmware for Linux, you must have the following service tools installed on your server:
    • Platform Enablement Library – librtas-xxxxx.rpm
    • Service Aids – ppc64-utils-xxxxx.rpm
    • Hardware Inventory – lsvpd-xxxxx.rpm

    where xxxxx represents a specific version of the RPM file.

    Note: If you do not have the service tools on your server, refer to Obtaining service and productivity tools for Linux.
  5. After the service tools are installed on the server running Linux, enter the following at a Linux command prompt:
    lsmcode

    The existing level of server firmware is displayed. For example, you might see output similar to the following:

    Version of system firmware is: AL740_088 (t)  AL740_088 (p)  AL740_088 (t)

    The following table provides descriptions for each of the server firmware levels displayed in the output.

    Table 1. Server firmware levels
    Server firmware levels displayed
    AL740_088 (t) AL740_088 (p) AL740_088 (t)
    The installed level.Also known as the temporary level. The backup level.Also known as the permanent level. The activated level.The level on which the server is currently running.
  6. Continue with the next step.

Step 2. View or download the firmware fix

Follow this procedure to view or download the firmware fix. You can download the fix directly to your server, or you can download it to a computer with an Internet connection and create a fix CD that you apply on the server. If necessary, contact service and support to order the fix on CD. You can also download the firmware fix to a computer that has a network connection to your server and use FTP to download the firmware fix from the computer to the server.

Note: If you plan to create a CD, you will need a CD burner and software.
  1. From a computer or server with an Internet connection, go to the Fix Central Web site at http://www.ibm.com/support/fixcentral/.
  2. Choose from the following options:
    1. If you have a System p® server, select System p in the Product Group list.
    2. If you have a POWER6 Power Systems server, select Power in the Product Group list.
  3. Select Firmware and HMC in the Product list.
  4. If prompted, select POWER5 and POWER6 class in the Processor type list.
  5. Select your Machine Type-Model and click Continue.
  6. Follow the on-screen prompts to download the fix file.
  7. Select from the following options:

Step 3. View and unpack the RPM file that contains the server firmware

If you created a CD with the RPM file, you will need to view and unpack the RPM file that contains the server firmware.
  1. Select from the following options:
    • If you created a CD with the RPM file, continue with the next step.
    • If you downloaded the RPM file to your server from the Fix Central Web site at http://www.ibm.com/support/fixcentral/ or by using the FTP method, continue with step 6.
  2. Insert the CD that contains the RPM file into the media drive on your server.
  3. To mount the CD, select from the following options (you need root user authority):
    • If you are working on an AIX system, enter the following at an AIX command prompt:
      mount /dev/cd0 /mnt
    • If you are working on a Linux system, enter one of the following commands at a Linux command prompt:
      mount -t iso9660 /dev/cdrom /mnt 

      or

      mount -t iso9660 /dev/dvdrom /mnt
  4. Select from the following options:
    • If the mount was successful, continue with step 6.
    • If the mount was unsuccessful, continue with the next step.
  5. If you received the message,
    mount: 0506-324 Cannot mount /dev/cd0 on /mnt, perform the following steps to mount the CD:

    1. Enter the command:
      /usr/sbin/mount -v 'cdrfs' -f'' -p'' -r'' /dev/cd0 /mnt

      The quotation marks following the f, p, and r are two single quotation marks with no space between them.

      Note: If you prefer, you can use the System Management Interface Tool (SMIT) to mount the CD.
    2. Continue with the next step.
  6. To view the RPM file name, enter the following command at the AIX or Linux command prompt:
    • If the RPM file is on CD, type:
      ls /mnt
    • If the RPM file is on the server, type:
      ls /tmp/fwupdate
    The name of the RPM file is displayed. For example, you might see output similar to the following:

    01EM3xx_yyy_zzz.rpm
  7. To unpack the RPM file, enter one of the following commands at the AIX or Linux command prompt:
    • If you want to unpack from a CD, enter:
      rpm -Uvh --ignoreos /mnt/filename.rpm
    • If you want to unpack from the server’s hard drive, enter:
      rpm -Uvh --ignoreos /tmp/fwupdate/filename.rpm
      where filename is the name of the RPM file that contains the server firmware. For example, 01EM3xx_yyy_zzz.rpm.

      Note: When you unpack the RPM file, the server firmware fix file is saved in the /tmp/fwupdate directory on the server’s hard drive in the following format: 01EM3xx_yyy_zzz.img.
  8. Continue with the next step.

Step 4. Apply server firmware fixes through AIX or Linux to the temporary side of the service processor

Important:

  • Do not interrupt this process after you begin.
  • Do not attempt to log into the ASMI, or use any of the ASMI’s functions, while a firmware installation is in progress.
  1. Ensure you are starting the system from the temporary side of the service processor; the firmware installation will fail if the system has booted from the permanent side. To learn which side you are starting from, and how to change to the other side if necessary, refer to Working with the temporary and permanent side of the service processor.
  2. To use the update_flash command (AIX or Linux) to install the server firmware, continue with step 3.
    Note: If you have AIX installed, you can choose to use the AIX diagnostics to install the fix. However, if you plan to install the fix from CD, you will need to obtain the Microcode Updates Files & Discovery Tool CD to use the AIX diagnostics.
  3. You will need the server firmware fix file name in the next step. To view the name, enter the following at an AIX or Linux command prompt:
    Note: To perform this step, you must have root user authority.
    ls /tmp/fwupdate
    The name of the server firmware fix file is displayed. For example, you might see output similar to the following:

    01EM3xx_yyy_zzz.img
  4. To install the server firmware fix, select from the following options:
    • If you are updating AIX, enter the following at an AIX command prompt:
      cd /tmp/fwupdate
      /usr/lpp/diagnostics/bin/update_flash -f fwlevel
    • # rpm -Uvh --ignoreos 01AL740_100_042.rpm
      01AL740_100_042             ##################################################
      # cd /tmp/fwupdate
      # ls
      01AL740_100_042.img
      # /usr/lpp/diagnostics/bin/update_flash -f 01AL740_100_042
      Error in opening the file 01AL740_100_042
      #  /usr/lpp/diagnostics/bin/update_flash -f 01AL740_100_042.img
      The image is valid and would update the temporary image to AL740_100.
      The new firmware level for the permanent image would be AL740_088.
      
      The current permanent system firmware image is AL740_088.
      The current temporary system firmware image is AL740_088.
      
      ***** WARNING: Continuing will reboot the system! *****
      
      Do you wish to continue?
      Enter 1=Yes or 2=No
      1
      
      SHUTDOWN PROGRAM
      Tue May 14 10:08:53 IST 2013
      0513-044 The sshd Subsystem was requested to stop.
      
      Wait for 'Rebooting...' before stopping.
      Error reporting has stopped.
      Advanced Accounting has stopped...
      Process accounting has stopped.
      nfs_clean: Stopping NFS/NIS Daemons
      0513-004 The Subsystem or Group, nfsd, is currently inoperative.
      0513-044 The biod Subsystem was requested to stop.
      0513-044 The rpc.lockd Subsystem was requested to stop.
      0513-044 The rpc.statd Subsystem was requested to stop.
      0513-004 The Subsystem or Group, gssd, is currently inoperative.
      0513-004 The Subsystem or Group, nfsrgyd, is currently inoperative.
      0513-004 The Subsystem or Group, rpc.mountd, is currently inoperative.
      0513-004 The Subsystem or Group, ypserv, is currently inoperative.
      0513-004 The Subsystem or Group, ypbind, is currently inoperative.
      0513-004 The Subsystem or Group, yppasswdd, is currently inoperative.
      0513-004 The Subsystem or Group, ypupdated, is currently inoperative.
      0513-004 The Subsystem or Group, nis_cachemgr, is currently inoperative.
      0513-004 The Subsystem or Group, rpc.nisd, is currently inoperative.
      0513-004 The Subsystem or Group, rpc.nispasswdd, is currently inoperative.
      0513-044 The qdaemon Subsystem was requested to stop.
      0513-044 The writesrv Subsystem was requested to stop.
      0513-044 The clcomd Subsystem was requested to stop.
      0513-044 The lldpd Subsystem was requested to stop.
      0513-044 The ecpvdpd Subsystem was requested to stop.
      0513-044 The ctrmc Subsystem was requested to stop.
      0513-044 The IBM.ServiceRM Subsystem was requested to stop.
      0513-044 The IBM.MgmtDomainRM Subsystem was requested to stop.
      0513-044 The IBM.DRM Subsystem was requested to stop.
      0513-044 The cas_agent Subsystem was requested to stop.
      All processes currently running will now be killed...
      Unmounting the file systems...
      umount: 0506-349 Cannot unmount /dev/hd10opt: The requested resource is busy.
      umount: 0506-349 Cannot unmount /dev/hd1: The requested resource is busy.

      where fwlevel is the specific file name of the server firmware fix, such as 01EM3xx_yyy_zzz.img

    • If you are updating Linux, enter the following at a Linux command prompt:
      cd /tmp/fwupdate
      /usr/sbin/update_flash -f fwlevel

      where fwlevel is the specific file name of the server firmware fix, such as 01EM3xx_yyy_zzz.img

    During the server firmware installation process, reference codes CA2799FD and CA2799FF are alternately displayed on the control panel. After the installation is complete, the system is automatically powered off and powered on.

    Note: If you receive a message stating:
    This partition does not have the authority to perform the requested function, see Message regarding a server that was previously managed by an HMC.
  5. Continue with the next step.

Step 5. Verify that the fix installed correctly

  1. Select from the following options:
    • To use the AIX or Linux command prompt (the operating system must be running and the diagnostics must be available), continue with the next step.
    • To use the ASMI, view the level of server firmware displayed in the upper-right corner below the copyright statement on the ASMI Welcome pane; for example, EM310_006. If the level of server firmware displayed is not the level that you installed, refer to step 4.
  2. Enter the following at a command prompt:
    lsmcode

    The existing levels of server firmware are displayed. For example, you might see output similar to the following:

    DISPLAY MICROCODE LEVEL                                                   802811
    IBM,8231-E1C
    
    The current permanent system firmware image is AL740_088
    The current temporary system firmware image is AL740_100
    The system is currently booted from the temporary firmware image.
    
    Use Enter to continue.
    
    
    Notes:

    • The permanent level is also known as the backup level.
    • The temporary level is also known as the installed level.
    • The system was booted from the temporary side, so at this time, the temporary level is also the activated level.
  3. Verify that the level of server firmware displayed is the level that you installed.
  4. If the level of server firmware displayed is not the level that you installed, perform the following steps:
    1. Retry the fix procedure. If you created a CD or DVD for this procedure, use a new media.
    2. If the problem persists, contact your next level of support.

Using AIX commands to install a firmware fix permanently

You can install a firmware fix permanently by using either the flash command or the AIX diagnostic service aids.

Note: To perform this task, you must meet the following criteria:

  • You must have root user authority.
  • You must start your server from the temporary side. For details, see Working with the temporary and permanent side of the service processor.

Using the flash command

At an AIX command prompt, type the following:

/usr/lpp/diagnostics/bin/update_flash -c

The update_flash -c command might run for 10 or more minutes.

Using the AIX diagnostic service aids

  1. At the AIX command prompt, type
    diag.
  2. Initialize the terminal type, if requested.
  3. On the function selection screen, select Tasks and Service Aids.
  4. On the task selection screen, scroll to the bottom of the list of options, and select Update and Manage Flash.
  5. Select Commit the Temporary Image, and press Enter. The process might run for 10 or more minutes.

AIX LUNs(LUNz) presented to host?

LUNz is the logical unit number that an application client uses to communicate with, configure and determine information about an SCSI storage array and the logical units attached to it. The LUN_Z value shall be zero.

LUNz has been implemented on CLARiiON arrays to make arrays visible to the host OS and PowerPath when no LUNs are bound on that array.  When using a direct connect configuration, and there is no Navisphere Management station to talk directly to the array over IP, the LUNZ can be used as a pathway for Navisphere CLI to send Bind commands to the array.

LUNz also makes arrays visible to the host OS and PowerPath when the host’s initiators have not yet ‘logged in to the Storage Group created for the host.  Without LUNz, there would be no device on the host for Navisphere Agent to push the initiator record through to the array. This is mandatory for the host to log in to the Storage Group. Once this initiator push is done, the host will be displayed as an available host to add to the Storage Group in Navisphere Manager (Navisphere Express).

LUNz should disappear once a LUN zero is bound, or when Storage Group access has been attained.

To conclude, the LUNz devices will be shown up in following two scenarios:
1. when arraycommpath is set to 1(enabled) and host HBAs are registered and login to Clariion array, but no “Storage Group” is configured for this host.
2. when there is no LUN configured using HLU0(Host LUN0) in the host “Storage Group”.

Figure-1 Storage Group with no LUN assigned using HLU0
To resolve this LUNz issue:
1. Verify LUNz hdisk number by issue:

## lsdev -Cc disk | grep LUNZ

 

hdisk5 Available 08-08-08     EMC CLARiiON FCP LUNZ Disk

hdisk6 Available 08-08-08     EMC CLARiiON FCP LUNZ Disk

hdisk7 Available 08-08-08     EMC CLARiiON FCP LUNZ Disk

hdisk8 Available 08-08-08     EMC CLARiiON FCP LUNZ Disk

 

2. set one of the LUNs assigned to the host using HLU0.

Figure-2 Assign LUN with HLU0
3. Remove each LUNZ device with command:
#rmdev -dl hdisk5
#rmdev -dl hdisk6
#rmdev -dl hdisk7
#rmdev -dl hdisk8

4. Reconfigure devices with command cfgmgr or emc_cfgmgr

Linux httpd service not start

[root@sunx4150 run]# ps -ef | grep http
root      7222  6826  0 10:32 pts/1    00:00:00 grep http
[root@sunx4150 run]# pwd
/etc/httpd/run
[root@sunx4150 run]# ls
acpid.socket  dirmngr           hpssd.pid       mdadm           nscd         radvd          sshd.pid                vmnet-dhcpd-vmnet8.pid    xenstored
agent.pid     dovecot           hpssd.port      mdmpd           ntpd.pid     rpc.statd.pid  sudo                    vmnet-natd-8.mac          xfs.pid
atd.pid       dsviewserver.pid  httpd.pid       messagebus.pid  openldap     saslauthd      syslogd.pid             vmnet-natd-8.pid          xinetd.pid
avahi-daemon  gather            iiim            mysqld          ppp          screen         tog-pegasus             vmnet-netifup-vmnet1.pid
console       gdm.pid           irqbalance.pid  named           ptal-mlcd    sendmail.pid   utmp                    vmnet-netifup-vmnet8.pid
crond.pid     gpm.pid           iscsid.pid      netreport       ptal-printd  setrans        vmnat.4650              vmware
cups          haldaemon.pid     klogd.pid       NetworkManager  pvm3         smbd.pid       vmnet-bridge-0.pid      winbindd
cupsd.pid     hpiod.pid         libvirt         news            quagga       sm-client.pid  vmnet-detect.pid        wpa_supplicant
dbus          hpiod.port        libvirtd.pid    nmbd.pid        radiusd      spamassassin   vmnet-dhcpd-vmnet1.pid  xend
[root@sunx4150 run]# ls -ltr
total 420
drwxr-xr-x 2 pvm     pvm     4096 Jul 13  2006 pvm3
drwxrwx–x 2 quagga  quagga  4096 Jul 13  2006 quagga
drwxr-xr-x 2 root    root    4096 Jul 14  2006 wpa_supplicant
drwxr-xr-x 2 radvd   radvd   4096 Aug 19  2006 radvd
drwxr-x— 2 news    news    4096 Aug 29  2006 news
drwxr-xr-x 2 root    root    4096 Sep  2  2006 winbindd
drwxr-xr-x 2 root    root    4096 Sep 30  2006 saslauthd
drwx—— 4 root    root    4096 Oct  2  2006 sudo
drwx—— 2 root    root    4096 Nov  9  2006 mdmpd
drwx—— 2 root    root    4096 Nov  9  2006 mdadm
drwx—— 2 radiusd radiusd 4096 Nov 30  2006 radiusd
drwxr-xr-x 2 root    root    4096 Dec  1  2006 ppp
drwxrwxr-x 2 root    screen  4096 Dec  4  2006 screen
drwxr-xr-x 2 ldap    ldap    4096 Jan  3  2007 openldap
drwxr-xr-x 2 root    root    4096 Jan  8  2007 NetworkManager
drwxr-x— 2 root    pegasus 4096 Jan 12  2007 gather
drwxr-x–T 2 root    pegasus 4096 Jan 12  2007 tog-pegasus
drwxr-xr-x 2 root    root    4096 Jan 12  2007 setrans
drwxr-xr-x 2 root    root    4096 Jan 16  2007 console
drwxr-xr-x 2 root    root    4096 Jan 16  2007 nscd
drwxrwxr-x 2 root    root    4096 Jan 16  2007 netreport
drwxrwx— 2 named   named   4096 Jan 17  2007 named
drwxr-xr-x 2 root    root    4096 Jun 11  2007 spamassassin
drwxr-xr-x 2 root    root    4096 Oct 11  2007 dirmngr
drwxr-xr-x 2 root    root    4096 Apr 30  2008 xenstored
drwxr-xr-x 2 root    root    4096 Apr 30  2008 xend
drwxr-xr-x 3     100     101 4096 Jul 15  2008 iiim
drwxr-xr-x 3 root    dovecot 4096 Apr  6  2009 dovecot
drwxr-xr-x 3 root    lp      4096 Apr  6  2009 cups
drwxr-xr-x 2 root    root    4096 Apr  6  2009 ptal-printd
drwxr-xr-x 2 root    root    4096 Apr  6  2009 ptal-mlcd
drwxr-xr-x 3 root    root    4096 Nov 24  2010 vmware
-rw——- 1 root    root       5 Mar 12 10:48 iscsid.pid
-rw——- 1 root    root       5 Mar 12 10:49 syslogd.pid
-rw-r–r– 1 rpcuser rpcuser    5 Mar 12 10:49 rpc.statd.pid
-rw——- 1 root    root       5 Mar 12 10:49 klogd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 irqbalance.pid
-rw-rw-rw- 1 root    root       5 Mar 12 10:49 vmnet-bridge-0.pid
-rw-rw-rw- 1 root    root       5 Mar 12 10:49 vmnet-netifup-vmnet8.pid
-rw-rw-rw- 1 root    root       5 Mar 12 10:49 vmnet-netifup-vmnet1.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 vmnet-natd-8.pid
-rw-rw-rw- 1 root    root      18 Mar 12 10:49 vmnet-natd-8.mac
-rw-r—– 1 root    root       5 Mar 12 10:49 vmnet-dhcpd-vmnet8.pid
-rw-r—– 1 root    root       5 Mar 12 10:49 vmnet-dhcpd-vmnet1.pid
-rw-rw-rw- 1 root    root       5 Mar 12 10:49 vmnet-detect.pid
srwxrwxrwx 1 root    root       0 Mar 12 10:49 vmnat.4650
-rw-r–r– 1 root    root       5 Mar 12 10:49 messagebus.pid
drwxr-xr-x 2 root    root    4096 Mar 12 10:49 dbus
-rw-rw-r– 1 root    root       4 Mar 12 10:49 agent.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 hpiod.port
-rw-r–r– 1 root    root       5 Mar 12 10:49 hpiod.pid
srw-rw-rw- 1 root    root       0 Mar 12 10:49 acpid.socket
-rw-r–r– 1 root    root       5 Mar 12 10:49 hpssd.port
-rw-r–r– 1 root    root       5 Mar 12 10:49 hpssd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 xinetd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 sshd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 cupsd.pid
-rw-r–r– 1 root    root       4 Mar 12 10:49 ntpd.pid
drwxr-xr-x 2 mysql   mysql   4096 Mar 12 10:49 mysqld
-rw-r–r– 1 smmsp   smmsp     49 Mar 12 10:49 sm-client.pid
-rw——- 1 root    smmsp     33 Mar 12 10:49 sendmail.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 gpm.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 dsviewserver.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 crond.pid
-rw-r–r– 1 root    root       6 Mar 12 10:49 xfs.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 smbd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 nmbd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 libvirtd.pid
drwxr-xr-x 2 root    root    4096 Mar 12 10:49 libvirt
-rw-r–r– 1 root    root       5 Mar 12 10:49 atd.pid
-rw-r–r– 1 root    root       5 Mar 12 10:49 haldaemon.pid
drwxr-xr-x 2 avahi   avahi   4096 Mar 12 10:49 avahi-daemon
-rw-r–r– 1 root    root       5 Mar 12 10:50 gdm.pid
-rw-r–r– 1 root    root       5 Apr  5 04:02 httpd.pid
-rw-rw-r– 1 root    utmp    6528 May  9 10:23 utmp
[root@sunx4150 run]# rm -rf httpd.pid
[root@sunx4150 run]# service httpd start
Starting httpd:                                            [FAILED]
[root@sunx4150 run]# service httpd status
httpd dead but subsys locked
[root@sunx4150 run]# service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [FAILED]
[root@sunx4150 run]# rm -f  /var/lock/subsys/httpd
You have new mail in /var/spool/mail/root
[root@sunx4150 run]#  service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [FAILED]
[root@sunx4150 run]#  service httpd status
httpd is stopped
[root@sunx4150 run]#  service httpd start
Starting httpd:                                            [FAILED]
[root@sunx4150 run]#  ipcs -s apahe

—— Semaphore Arrays ——–
key        semid      owner      perms      nsems
0x000000a7 0          root      666        1
0x0000033d 1474561    root      644        1
0x0f9b5efc 163842     oracle    640        44
0x00000000 1409027    apache    600        1
0x00000000 229380     apache    600        1
0x00000000 1441797    apache    600        1
0x00000000 1507334    apache    600        1
0x00000000 1540103    apache    600        1
0x00000000 1572872    apache    600        1
0x00000000 1605641    apache    600        1
0x00000000 1638410    apache    600        1
0x00000000 1671179    apache    600        1

[root@sunx4150 run]# ipcs -s | grep apache | perl -e ‘while (<STDIN>) { @a=split(/s+/);print `ipcrm sem $a[1]`}’
resource(s) deleted
resource(s) deleted
resource(s) deleted
resource(s) deleted
resource(s) deleted
resource(s) deleted
resource(s) deleted
resource(s) deleted
resource(s) deleted
[root@sunx4150 run]# cd /var/lock/subsys
[root@sunx4150 subsys]#  rm httpd
rm: cannot lstat `httpd’: No such file or directory
[root@sunx4150 subsys]# service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [FAILED]
[root@sunx4150 subsys]# vi /etc/httpd/conf/httpd.conf
You have new mail in /var/spool/mail/root
[root@sunx4150 subsys]# killall -9 httpd
httpd: no process killed
[root@sunx4150 subsys]# killall -9 php-cgi
php-cgi: no process killed
[root@sunx4150 subsys]# killall -9 perl-cgi
perl-cgi: no process killed
[root@sunx4150 subsys]# netstat -tulpn | grep :80
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      5308/tnslsnr
[root@sunx4150 subsys]# netstat -tulpn | grep :443
[root@sunx4150 subsys]# tail -f /var/log/httpd/error.log
tail: cannot open `/var/log/httpd/error.log’ for reading: No such file or directory
tail: no files remaining
[root@sunx4150 subsys]# vi /etc/httpd/conf/httpd.conf
[root@sunx4150 subsys]# tail -f /var/logs/error_log
tail: cannot open `/var/logs/error_log’ for reading: No such file or directory
tail: no files remaining
[root@sunx4150 subsys]# tail -f /etc/httpd/logs/error_log
[Thu May 09 10:27:40 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:30:14 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:32:47 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:33:08 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:43:41 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:43:57 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:45:28 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 09 10:49:51 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

You have new mail in /var/spool/mail/root
[root@sunx4150 subsys]# cd /etc/httpd/logs/
[root@sunx4150 logs]# ls -ltr
total 2348
-rw-r–r– 1 root root 654815 Nov  2  2011 nss_error_log.2
-rw-r–r– 1 root root 682530 Nov  2  2011 nss_access_log.1
-rw-r–r– 1 root root      0 Nov  3  2011 nss_access_log
-rw-r–r– 1 root root     77 Mar  1 11:15 ssl_request_log.4
-rw-r–r– 1 root root     71 Mar  1 11:15 ssl_access_log.4
-rw-r–r– 1 root root    312 Mar  7 15:56 ssl_request_log.3
-rw-r–r– 1 root root    288 Mar  7 15:56 ssl_access_log.3
-rw-r–r– 1 root root    385 Mar 15 16:57 ssl_request_log.2
-rw-r–r– 1 root root   1306 Mar 15 16:57 ssl_error_log.4
-rw-r–r– 1 root root    355 Mar 15 16:57 ssl_access_log.2
-rw-r–r– 1 root root  85057 Mar 17 04:00 access_log.4
-rw-r–r– 1 root root    237 Mar 17 04:02 ssl_error_log.3
-rw-r–r– 1 root root  87266 Mar 24 04:00 access_log.3
-rw-r–r– 1 root root 197481 Mar 24 04:02 error_log.4
-rw-r–r– 1 root root    237 Mar 24 04:02 ssl_error_log.2
-rw-r–r– 1 root root  91220 Mar 31 04:00 access_log.2
-rw-r–r– 1 root root 197481 Mar 31 04:02 error_log.3
-rw-r–r– 1 root root     77 Apr  4 10:47 ssl_request_log.1
-rw-r–r– 1 root root     71 Apr  4 10:47 ssl_access_log.1
-rw-r–r– 1 root root      0 Apr  5 04:02 ssl_request_log
-rw-r–r– 1 root root    593 Apr  5 04:02 ssl_error_log.1
-rw-r–r– 1 root root      0 Apr  5 04:02 ssl_access_log
-rw-r–r– 1 root root  86524 Apr  7 04:00 access_log.1
-rw-r–r– 1 root root      0 Apr  7 04:02 ssl_error_log
-rw-r–r– 1 root root    340 Apr  7 04:02 nss_error_log.1
-rw-r–r– 1 root root 198038 Apr  7 04:02 error_log.2
-rw-r–r– 1 root root    141 Apr  7 04:02 error_log.1
-rw-r–r– 1 root root      0 Apr  7 04:02 access_log
-rw-r–r– 1 root root    712 May  9 10:49 error_log
-rw-r–r– 1 root root   3740 May  9 10:56 nss_error_log
[root@sunx4150 logs]# cat nss_error_log
[Thu May 09 10:27:40 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:27:40 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:27:40 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:30:14 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:30:14 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:30:14 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:32:47 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:32:47 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:32:47 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:33:08 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:33:08 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:33:08 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:43:41 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:43:41 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:43:41 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:43:57 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:43:57 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:43:57 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:45:28 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:45:28 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:45:28 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:49:51 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:49:51 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:49:51 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:53:09 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:53:09 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:53:09 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:53:26 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:53:26 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:53:26 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[Thu May 09 10:56:35 2013] [error] Certificate not verified: ‘Server-Cert’
[Thu May 09 10:56:35 2013] [error] SSL Library Error: -8181 Certificate has expired
[Thu May 09 10:56:35 2013] [error] Unable to verify certificate ‘Server-Cert’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.
[root@sunx4150 logs]#

Add “NSSEnforceValidCerts off” to nss.conf

[root@sunx4150 httpd]# cd conf.d/
[root@sunx4150 conf.d]# ls
auth_kerb.conf   manual.conf  php.conf        squid.conf
auth_mysql.conf  mrtg.conf    proxy_ajp.conf  ssl.conf
auth_pgsql.conf  nss.conf     python.conf     subversion.conf
authz_ldap.conf  perl.conf    README          welcome.conf
[root@sunx4150 conf.d]# vi nss.conf
[root@sunx4150 conf.d]# /etc/init.d/httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd: Syntax error on line 193 of /etc/httpd/conf.d/nss.conf:
Invalid command ‘SEnforceValidCerts’, perhaps misspelled or defined by a module not included in the server configuration
[FAILED]
[root@sunx4150 conf.d]# vi nss.conf
You have new mail in /var/spool/mail/root
[root@sunx4150 conf.d]# /etc/init.d/httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]
[root@sunx4150 conf.d]#

AIX Memory / RAM performance monitoring

Memory

Memory Leak: Caused by a program that repeatedly allocates memory without freeing it.

When a process exits, its working storage is freed up immediately and its associated memory frames are put back on the free list.
However any files the process may have opened can stay in memory.

AIX tries to use the maximum amount of free memory for file caching.

High levels of file system cache usually means that is the way the application runs and likes it (you have to decide if this is expected by understanding the workload) or AIX can’t find anything else to do with the memory and so thinks it might as well save disk I/O CPU cycles by caching – this is normal and a good idea.

Some notes regarding memory leak:

When a process gets busy, process will use malloc() system call (memory allocation) to get more memory, so its memory usage gets bigger.  Memory requests are satisfied by allocating portions from a large pool of memory called the heap. When the process goes idle, it uses free() system call, but that doesn’t actually free up the memory from the process. It just releases the memory into the “heap area”.

AIX keeps a list of the pages in the heap area about the free memory pages that were used, but not used now. If there are new new malloc() requests, they will be served from heap first. Only if the heap goes to a very small size, only then will be issued new malloc() request to get new memory pages. When heap pages are not used for a long time AIX will page out these to disk.

RSS size is the actual memory occupied by the process in the RAM. (RSS can be active pages or some other pages in the heap). RSS pages will be paged out only if memory is getting short. If there is free mamory, it will not page these out, becaue it maybe useful to have it in the RAM

So, usually it turns out, there is no memory leak at all, just normal memory usage behaviour!!!

————————

memory:
topas -P    This does not tell how much of the application is paged out but how much of the application memory is backed by paging space.
(things in memory (working segment) should be backed by paging space by the actual size in memory of the process.)
svmon -Pt15 | perl -e ‘while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}’        top 15 processes using the most memory
ps aux | head -1 ; ps aux | sort -rn +3 | head -20                                   top memory processes (the above is better)
ps -ef | grep -c LOCAL=NO        shows the number of oracle client connections (each connection takes up memory, so if it is high then…)

paging:
svmon -Pg -t 1 |grep Pid ; svmon -Pg -t 10 |grep “N”                                 top 10 processes using the most paging space
svmon -P -O sortseg=pgsp                                                             shows paging space usage of processes

————————

# ps gv | head -n 1; ps gv | egrep -v “RSS” | sort +6b -7 -n -r
PID    TTY STAT  TIME PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
393428      – A    10:23 2070 54752 54840 32768    69    88  0.0  5.0 /var/opt
364774      – A     0:08  579 28888 28940 32768    32    52  0.0  3.0 [cimserve]
397542      – A     0:18  472  6468  7212    xx   526   744  0.0  1.0 /usr/sbi
344246      – A     0:02   44  7132  7204 32768    50    72  0.0  1.0 /opt/ibm

RSS:    The amount of RAM used for the text and data segments per process. PID 393428 is using 54840k. (RSS:resident set size)
%MEM:    The actual amount of the RSS / Total RAM. Watch for processes that consume 40-70 percent of %MEM.
TRS:    The amount of RAM used for the text segment of a process in kilobytes.
SIZE:    The actual amount of paging space (virtual mem. size) allocated for this process (text and data).

How much big is the process in memory? It is the RSS size.
————————————–

Checking memory usage with nmon:

nmon –> t (top processes) –> 4 (order in process size)

PID       %CPU     Size      Res     Res      Res     Char    RAM      Paging         Command
Used       KB      Set     Text     Data     I/O     Use   io   other repage
16580722     0.0   226280   322004   280640    41364        0    5%      0      0      0 oracle
9371840      0.0   204324   300904   280640    20264        0    5%      0      0      0 oracle
10551416     0.0   198988   305656   280640    25016        0    5%      0      0      0 oracle
8650824      0.0   198756   305428   280640    24788        0    5%      0      0      0 oracle

Size KB: program on disk size
ResSize: Resident Set Size – how big it is in memory (excluding the pages still in the file system (like code) and some parts on paging disks)
ResText: code pages of the Resident Set
ResData: data and stack pages of the Resident Set

————————————–

regarding ORACLE:
ps -ef | grep -c LOCAL=NO

This will show how many client connections we have. Each connections take up some memory, sometimes if there are memory problems too many users are logegd in causing this triouble.
————————————–

shared memory segments:

root@aix2: /root #  ipcs -bm
IPC status from /dev/mem as of Sat Sep 17 10:04:28 CDT 2011
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x010060f0 –rw-rw-rw-     root   system       980
m   1048577 0xffffffff D-rw-rw-rw-     root   system       944
m   4194306 0x78000238 –rw-rw-rw-     root   system  16777216
m   1048579 0x010060f2 –rw-rw-rw-     root   system       976
m        12 0x0c6629c9 –rw-r—–     root   system   1663028
m        13 0x31000002 –rw-rw-rw-     root   system    131164
m 425721870 0x81fc461c –rw-r—–   oracle oinstall 130027520
m        15 0x010060fa –rw-rw-rw-     root   system      1010
m   2097168 0x849c6158 –rw-rw—-   oracle oinstall 18253647872

It will show our memory segments, who owns them and what their size (in bytes). It shows the maximum allocated size, that a memory segment can go to. It does not mean it is allocated, but the exception is Oracle (and DB2).
Oracle line shows the SGA for Oracle. (This memory is allocated for Oracle. It is 18GB in this case)

————————————–

IBM script for checking what is causing paging space activity:
(it will run until po will be 50 then saves processes, svmon and exists)

#!/usr/bin/ksh
/usr/bin/renice -n -20 -p $$
while [ true ]
do
vmstat -I 1 1 | tail -1 | awk ‘{print $9}’ | read po
if [[ $po -gt 50 ]]
then
ps -ef > ps.out &
svmon -G > svmon.G &
exit 0
fi
done

My script for monitoring memory, paging activity:

#!/usr/bin/ksh
/usr/bin/renice -n -20 -p $$

while [ true ]; do
echo `date` “–>” `svmon -G | head -2 | tail -1` “–>” `vmstat -v | grep numperm` >> svmon.out &
echo `date` “–>” `svmon -G | head -3 | tail -1` >> paging.out &
echo `vmstat -Iwt 1 1 | tail -1` >> vmstat.out &
sleep 60
done

AIX SDD (subsystem device driver)

SDD is designed to support the multipath configuration in the ESS.
The software used to balance ESS I/O traffic across all adapters. It provides multiple access to data from the host.
when using sdd cfgmgr is run 3 times (cfgmgr -l fcs0, cfgmgr -l fcs1, cfgmgr (the third one builds the vpaths))

3 policies exist for load balancing:
default: selecting the path with the least number of current I/O operations
-round robin: choosing the path, that was not used for the last operation (alternating if 2 pathes exist)
-failover: all I/O sent ove the most preferred path, until a failure is detected.

SDDSRV:
SDD has a server daemon running in the background: lssrc/stopsrc/startsrc -s sddsrv
If sddsrv is stopped, the feature that automatically recovers failed paths disabled.

vpath:
A logical disk defined in ESS and recognized by AIX. AIX uses vpath instead of hdisk as a unit of physical storage.

root@aix: /dev # lsattr -El vpath0
active_hdisk  hdisk20/00527461/fscsi1          Active hdisk                 False
active_hdisk  hdisk4/00527461/fscsi0           Active hdisk                 False
policy        df                               Scheduling Policy            True    <-path selection policy
pvid          0056db9a77baebb90000000000000000 Physical volume identifier   False
qdepth_enable yes                              Queue Depth Control          True
serial_number 00527461                         LUN serial number            False
unique_id     1D080052746107210580003IBMfcp    Device Unique Identification False

policy:
fo: failover only – all I/O operations sent to the same paths until the path fails
lb: load balancing – the path is chosen by the number of I/O operations currently in process
lbs: load balancing sequential – same as before with optimization for sequential I/O
rr: round ropbin – path is chosen at random from the not used paths
rrs: round robin sequential – same as before with optimization for sequential I/O
df: default – the default policy is load balancing

datapath set device N policy        change the SDD path selection policy dynamically

DPO (Data Path Optimizer):
it is a pseudo device (lsdev | grep dpo), which is the pseudo parent of the vpaths

root@: / # lsattr -El dpo
SDD_maxlun      1200 Maximum LUNS allowed for SDD                  False
persistent_resv yes  Subsystem Supports Persistent Reserve Command False

— ——————————

software requirements for SDD:
-host attachment for SDD (ibm2105.rte, devices.fcp.disk.ibm.rte) – this is the ODM extension
The host attachments for SDD add 2105 (ESS)/2145 (SVC)/1750 (DS6000)/2107 (DS8000) device information to allow AIX to properly configure 2105/2145/1750/2107 hdisks.
The 2105/2145/1750/2107 device information allows AIX to:
– Identify the hdisk(s) as a 2105/2145/1750/2107 hdisk.
– Set default hdisk attributes such as queue_depth and timeout values.
– Indicate to the configure method to configure 2105/2145/1750/2107 hdisk as non-MPIO-capable devices

ibm2105.rte: for 2105 devices
devices.fcp.disk.ibm.rte: for DS8000, DS6000 and SAN Volume Controller)

-devices.sdd.53.rte – this is the driver (sdd)
it provides the multipath configuration environment support

——————————–

addpaths                  dynamically adds more paths to SDD vpath devices (before addpaths, run cfgmgr)
(running cfgmgr alone does not add new paths to SDD vpath devices)
cfgdpo                    configures dpo
cfgvpath                  configures vpaths
cfallvpath                configures dpo+vpaths
dpovgfix <vgname>         fixes a vg that has mixed vpath and hdisk physical volumes
extenfvg4vp               this can be used insteadof extendvg (it will move pvid from hdisk to vpath)

datapath query version    shows sdd version
datapath query essmap     shows vpaths and their hdisks in a list
datapath query portmap    shows vpaths and ports
—————————————
datapath query adapter    information about the adapters
State:
Normal           adapter is in use.
Degraded         one or more paths are not functioning.
Failed           the adapter is no longer being used by SDD.

datapath query device     information about the devices 8datapath query device 0)
State:
Open             path is in use
Close            path is not being used
Failed           due to errors path has been removed from service
Close_Failed     path was detected to be broken and failed to open when the device was opened
Invalid          path is failed to open, but the MPIO device is opened
—————————————
datapath remove device X path Y   removes path# Y from device# X (datapath query device, will show X and Y)
datapath set device N policy      change the SDD path selection policy dynamically
datapath set adapter 1 offline

lsvpcfg                           list vpaths and their hdisks
lsvp -a                           displays vpath, vg, disk informations

lquerypr                          reads and releases the persistent reservation key
lquerypr -h/dev/vpath30           queries the persistent resrevation on the device (0:if it is reserved by current host, 1: if another host)
lquerypr -vh/dev/vpath30          query and display the persistent reservation on a device
lquerypr -rh/dev/vpath30          release the persisten reservation if the device is reserved by the current host
(0: if the command succeeds or not reserved, 2: if the command fails)
lquerypr -ch/dev/vpath30          reset any persistent reserve and clear all reservation key registrations
lquerypr -ph/dev/vpath30          remove the persisten reservation if the device is reserved by another host
—————————————

Removing SDD (after install a new one):
-umount fs on ESS
-varyoffvg
(if HACMP and RG is online on other host: vp2hd <vgname>) <–it converts vpaths to hdisks)
-rmdev -dl dpo -R                                         <–removes all the SDD vpath devices
-stopsrc -s sddsrv                                        <–stops SDD server
-if needed: rmdev -dl hdiskX                              <–removes hdisks
(lsdev -C -t 2105* -F name | xargs -n1 rmdev -dl)

-smitty remove — devices.sdd.52.rte
-smitty install — devices.sdd.53.rte (/mnt/Storage-Treiber/ESS/SDD-1.7)
-cfgmgr
—————————————

Removing SDD Host Attachment:
-lsdev -C -t 2105* -F name | xargs -n1 rmdev -dl          <–removes hdisk devices
-smitty remove — ibm2105.rte (devices.fcp.disk.ibm)
—————————————

Change adapter settings (Un/re-configure paths):

-datapath set adapter 1 offline
-datapath remove adapter 1
-rmdev -Rl fcs0
(if needed: for i in `lsdev -Cc disk | grep -i defined | awk ‘{ print $1 }’`; do rmdev -Rdl $i; done)
-chdev -l fscsi0 -a dyntrk=yes -a fc_err_recov=fast_fail
-chdev -l fcs0 -a init_link=pt2pt
-cfgmgr; addpaths
—————————————

Reconfigure vpaths:
-datapath remove device 2 path 0
-datapath remove device 1 path 0
-datapath remove device 0 path 0
-cfgmgr; addpaths
-rmdev -Rdl vpath0
-cfgmgr;addpaths
—————————————

Can’t give pvid for a vpath:
root@aix: / # chdev -l vpath6 -a pv=yes
Method error (/usr/lib/methods/chgvpath):
0514-047 Cannot access a device.

in errpt:DEVICE LOCKED BY ANOTHER USER
RELEASE DEVICE PERSISTENT RESERVATION

# lquerypr -Vh /dev/vpath6          <–it will show the host key
# lquerypr -Vph /dev/vpath6         <–it will clear the reservation lock
# lquerypr -Vh /dev/vpath6          <–checking again will show it is OK now

How to Encrypt File System in AIX ?

Encrypting Filesystem on AIX 6.1.

EFS offers 2 modes of operation:

Root Admin mode
This is the default mode. Root can reset user and group keystore passwords.

Root Guard mode
Root does not have access to user’s encrypted files and cannot change their passwords.

Note: NFS exports of EFS filesystems are not supported.

1. Prerequisites:
RBAC has to be enabled. Should be by default on AIX 6.1. If not use chdev to enable it.

# lsattr -El sys0 | grep RBAC
enhanced_RBAC   true         Enhanced RBAC Mode        True

CryptoLite needs to be installed, verify using below command

bash-3.2# lslpp -l | grep  CryptoLite
  clic.rte.kernext           4.7.0.1  COMMITTED  CryptoLite for C Kernel
  clic.rte.lib               4.7.0.1  COMMITTED  CryptoLite for C Library
  clic.rte.kernext           4.7.0.1  COMMITTED  CryptoLite for C Kernel

2. EFS Commands:

efsenable – Enables EFS on a given system. This is run only once
efskeymgr – Encryption Key Management tool
efsmgr – File encryption and decryption

3. Setup:
To enable EFS on the system use:

# efsenable -a
Enter password to protect your initial keystore:
Enter the same password again:

If your password for EFS will be identical with your login password the EFS Kernel extention will be loaded automatically into the kernel. Thus
you will be able to access the encrypted files without having to provide a password.
Otherwise `efskeymgr -o ksh` has tto be executed in order to load the key’s.

In order to have the ability to encrypt files, the filesystem that will hold this files needs to be EFS enabled (efs=yes) and Extended Attribute V2 has to be activated.

This can be verified using lsfs -q

# lsfs -q /test
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/fslv12     --         /test               jfs2  262144  rw         yes  no
  (lv size: 262144, fs size: 262144, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no)

# chfs -a efs=yes /test

# lsfs -q /archive
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/fslv12     --         /test               jfs2  262144  rw         yes  no
  (lv size: 262144, fs size: 262144, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v2, Quota: no, DMAPI: no, VIX: yes, EFS: yes, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no)

Now we will have a look at the keys associated  with the current shell.

# efskeymgr -V
List of keys loaded in the current process:
 Key #0:
                           Kind ..................... User key
                           Id   (uid / gid) ......... 0
                           Type ..................... Private key
                           Algorithm ................ RSA_1024
                           Validity ................. Key is valid
                           Fingerprint .............. s6295ea1:be7cae83:82g30ab8:a02379a0
 Key #1:
                           Kind ..................... Group key
                           Id   (uid / gid) ......... 7
                           Type ..................... Private key
                           Algorithm ................ RSA_1024
                           Validity ................. Key is valid
                           Fingerprint .............. 12928ecb:353f4268:e19078be:268c7d56:18928ecb
 Key #2:
                           Kind ..................... Admin key
                           Id   (uid / gid) ......... 0
                           Type ..................... Private key
                           Algorithm ................ RSA_1024
                           Validity ................. Key is valid
                           Fingerprint .............. 940201f9:89h618ac:2e555ac4:60fdb6b5:268c7d56

4. Encrypt file

Now we will create a file, try to encrypt it, have a problem with umask and finally encrypt the file.

# echo "I like black tee with milk." > secret.txt
# ls -U
total 8
-rw-r------    1 root     system           30 May 8  11:18 secret.txt
drwxr-xr-x-    2 root     system          256 Apr 30 14:10 tmp

        Encrypt file
          |
# efsmgr -e secret.txt
./.efs.LZacya: Security authentication is denied.

# umask 077

# efsmgr -e secret.txt
# ls -U
total 16
drwxr-xr-x-    2 root     system          256 30 May 5 12:13 lost+found
-rw-r-----e    1 root     system           30 30 May 8 11:18 secret.txt
          |
          Indicates that this file is encrypted

Display file encryption information:

# efsmgr -l secret.txt
EFS File information:
 Algorithm: AES_128_CBC
List of keys that can open the file:
 Key #1:
  Algorithm       : RSA_1024
  Who             : uid 0
  Key fingerprint : 00f06152:be7cae83:a02379a0:82e30ab8:f6295ea1

Now I set the file permission’s to 644 and try to read the file as another user.

# chmod 644 secret.txt
# ls -la
-rw-r--r--    1 root     system          145 30 May 8 11:19 secret.txt

user1 # file secret.txt
secret.txt: 0653-902 Cannot open the specified file for reading.
user1 # cat secret.txt
cat: 0652-050 Cannot open secret.txt.

As root we will list the inode number of the file, get the block pointer and read directly from the filesystem using fsdb to see if the file is stored  encrypted.

      Display inode no.
      |
# ls -iU
total 32

    5 -rw-r--r--e    1 root     system          145 30 May 8 11:19 secret.txt

# istat 5 /dev/fslv12
Inode 5 on device 10/27 File
Protection: rw-r--r--
Owner: 0(root)          Group: 0(system)
Link count:   1         Length 145 bytes

Last updated:   Tue May 8 11:18:23 GMT+02:00 2012
Last modified:  Tue May 8 11:18:52 GMT+02:00 2012
Last accessed:  Tue May 8 11:18:52 GMT+02:00 2012

Block pointers (hexadecimal):
29
# fsdb /dev/fslv12
Filesystem /dev/fslv12 is mounted.  Modification is not permitted.

File System:                    /dev/fslv12

File System Size:               261728  (512 byte blocks)
Aggregate Block Size:           4096
Allocation Group Size:          8192    (aggregate blocks)

> display 0x29
Block: 41     Real Address 0x29000
00000000:  119CB74E 637C6FE0 C0BF2DCD 36B775BB   |...Nc|o...-.6.u.|
00000010:  569B5A6C 43476ED3 F4BFE938 7C662A3B   |V.ZlCGn....8|f*;|
00000020:  B5D89C51 FA2BE7B6 CEAF2D3E 555EAA06   |...Q.+....->U^..|
00000030:  4FF23413 B11D1170 982690B3 5F1BCA9A   |O.4....p.&.._...|
00000040:  4AD3CEA5 A3CBFAD9 C730EE00 9BD1F409   |J........0......|
00000050:  71203B85 A51320C6 04A97DA4 43002DA7   |q ;... ...}.C.-.|
00000060:  994CC67B A1AC31DF 2C8201AD 3E5B50F7   |.L.{..1.,...>[P.|
00000070:  6BA7B01D EC5CB918 17E13F46 2935FA98   |k........?F)5..|
00000080:  718DF155 D6E69A41 EF592B60 EA5F7B24   |q..U...A.Y+`._{$|
00000090:  32521FE2 7AD8EC61 1A94413D A8338A26   |2R..z..a..A=.3.&|
000000a0:  62E4A319 D6251A66 F19D4739 2FC7E83A   |b....%.f..G9/..:|
000000b0:  DE0F878A 1F95AB89 5C7F3520 C65B7896   |.........5 .[x.|
000000c0:  915A7655 EC269DFF 68E2B08A 871114A9   |.ZvU.&..h.......|
000000d0:  E30B195F 280F7DCD 4F8BE094 4B5603D8   |..._(.}.O...KV..|
000000e0:  962303B0 D957A2A5 24A2A3A5 6260EA5E   |.#...W..$...b`.^|
000000f0:  A4C62B7D FB9B1841 893D253F 72E61065   |..+}...A.=%?r..e|
-hit enter for more-
00000100:  01A150FD AD54677D A856E9B1 320257E1   |..P..Tg}.V..2.W.|
00000110:  5F023AA3 0191E0D6 4B64583B D9F2A4C7   |_.:.....KdX;....|
00000120:  F988937A E0117EB2 26E61976 E4860D7D   |...z..~.&..v...}|
00000130:  0C724A4E 50616226 BDE06FEB 10A19564   |.rJNPab&..o....d|
00000140:  17C90BB7 774338B3 8525ED90 5EADFD8B   |....wC8..%..^...|
00000150:  636FC1AF D46C2E64 6AC37082 3B0168BE   |co...l.dj.p.;.h.|
00000160:  24C0CD2E D8587254 F6DBC1BA 93BE6AD6   |$....XrT......j.|
00000170:  E89EEFF9 08000B07 E3827C10 AE0FD7DB   |..........|.....|
00000180:  162D0E6D EF94D85A 3F09CD85 A19A31FF   |.-.m...Z?.....1.|
00000190:  49E13BFC 5328F670 E0B50878 942CC4BB   |I.;.S(.p...x.,..|
000001a0:  BF1D6C4F 9DA72F3D 8DC90691 328A7053   |..lO../=....2.pS|
000001b0:  99C31EEB 1CD2208A CBF609C1 4DB86819   |...... .....M.h.|
000001c0:  E2746288 5E152ECA 0E2BD9DF D1D1D210   |.tb.^....+......|
000001d0:  7ADDF0EC 522E93E2 CAA0A36F B3CBFB05   |z...R......o....|
000001e0:  4EA56F3C ECBA1A0C AA132269 2024E065   |N.o<......"i $.e|
000001f0:  00BC51B0 88BBCD8A 9C644F66 6A16DBC8   |..Q......dOfj...|

Above we see that the file on the disk is encrypted.

5. Decrypting a file

Decrypt file
          |
# efsmgr -d secret.txt
# ls -U
total 24

-rw-r--r---    1 root     system          145 May 8 12:23 secret.txt

6. Encryption Inheritance

If you enable Encryption Inheritance on a directory all newly created files in that directory will be automatically encrypted.

To enable Encryption inheritance use:

# efsmgr -E /archive

# ls -U / | grep archive
drwxr-xr-xe    3 root     system          256 Jul 17 12:09 archive

# touch next.txt

# ls -U
total 32

-rw-------e    1 root     system            0 May 8 11:10 next.txt
-rw-r--r---    1 root     system          145 May 8 12:25 secret.txt

7. Grant access to another user
Say we are  user1 and want to have a look at who has EFS access to the file.

user1 $ efsmgr -l secret.txt
EFS File information:
 Algorithm: AES_128_CBC
List of keys that can open the file:
 Key #1:
  Algorithm       : RSA_1024
  Who             : uid 0
  Key fingerprint : 00f06152:be7cae83:a02379a0:82e30ab8:f6295ea1

To grant access to a user use:

Add access to the specified file to a user or group(u/g)
          |
# efsmgr -a secret.txt -u user1
                        |
                        Add user to EFS access list

user1 $ cat secret.txt
I like black tee with milk.

Reference Red-books:

AIX 6.1 Diffrence Guide SG24-7559-00 Page 40
AIX V6 Advanced Security Features SG24-7430-00 Page 59

How to Increase paging space logical volume size in AIX ?

To show the current paging space volume, its size along with other information use the ‘lsps’ command:

# lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk1            rootvg         512MB     2   yes   yes    lv     0

The paging space may be increased on the fly using the ‘chps’ command. In order to increase this paging space to 16 GB we need to to find out the PP size, since chps will increase the paging space by the specified number of PP’s

# lslv hd6
LOGICAL VOLUME:     hd6                    VOLUME GROUP:   rootvg
LV IDENTIFIER:      0005e4b80000d7000000013e5a69cf95.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               paging                 WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             1                      SCHED POLICY:   parallel
LPs:                2                      PPs:            2
STALE PPs:          0                      BB POLICY:      non-relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        N/A                    LABEL:          None
MIRROR WRITE CONSISTENCY: off
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO
INFINITE RETRY:     no
#

On this system is the PP size 256M so in order to increase the paging space size to 2Gig we need 62 PP’s

# chps -s 62 hd6
# lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk1            rootvg       16384MB     1   yes   yes    lv     0
#