Dual Virtual I/O Server update/upgrade Procedure

This post explains step by step process to update Dual VIO server from one version to new version.VIOS is nothing but Virtual I/O Server.
Use dual VIOS for redundant purpose, suppose VIOS1 gone through any maintenance activity VIOS2 will take the load within fraction of seconds.So there is no much impact to client partitions(AIX,Linux and IBM i)
Tested and successfully completed this VIOS update procedure in our environment. Follow the below steps to update the VIO Server.

1. Take the configuration backup and mksysb or alternate disk back.
2. Identify the Primary and Secondary VIO server and on both VIOS SEA adapters "ha_mode" should be "auto".
3. Identify the VIOS which is in backup mode.(If VIOS2 in backup mode, update VIOS2 first).
4. Break the rootvg mirror and take the clone for quick rollback. If your VIO server with single disk take the mksysb backup.
5. Remove the all interim fixes and commit the filesets in current version.
6. On VIOS2: For safer side  change SEA mode to standby and proceed with VIOS update.
7. Mount the packages downloaded NFS filesystem
8. Update the VIO Server using updateios command and wait until it completed. Next reboot the VIO Server.
9. Verify the latest VIOs version and check if any missing packages and re-mirror the rootvg.
10. On VIOS2: Change the SEA  adapters mode to "auto" and proceed with other VIOS.
11. On VIOS1: Now move the sea adapters to standby mode.
12. Follow the above step 4,5,7,8 and 9.
13. Now change the SEA  adapters mode to "auto".


1. Take the configuration backup and mksysb or alternate disk back.

The viosbr command creates a backup of only the VIOS configuration.
# viosbr -backup -file /tmp/<file name>

Take the mksysb backup for rootvg. It will create entire rootvg backup
# backupios -file <path to NFS location>/<file name> -mksysb

2. Identify the Primary and Secondary VIO server and on both VIOS SEA adapters "ha_mode" should be "auto".For example ent20 and ent21 are SEA adapters.
   Run the below commands on both VIOS.

# lsdev -Cc adapter|grep -i shared

# entstat -d ent20 |grep -i state
# entstat -d ent21 |grep -i state

3. Identify the VIOS which is in backup mode.(If VIOS2 is on backup mode, update VIOS2 first).


4. Break the rootvg mirror and take the clone for quick rollback. If your VIO server with single disk take the mksysb backup.

To break the rootvg mirror and create a clone for quick rollback
$ bootlist -mode normal –ls
$ unmirrorios hdisk1
#  migratepv hdisk1 hdisk0     (Run as root user)
$ reducevg -rmlv rootvg hdisk1
$ alt_root_vg -target hdisk1

$oem_setup_env (to get root shell)
# bosboot -ad hdisk0 (using root shell)
# bootlist -m normal hdisk0 hdisk1 (using root shell)

5. Remove the all interim fixes and commit the filesets in current version.

To remove all interim fixes, if any installed.
$ lssw
$ updateios -remove <ifix_LABEL>
Commit existing filesets:
$ updateios –commit

6. On VIOS2: For safer side  change SEA mode to standby and proceed with VIOS update.

$ chdev -dev ent20 -attr ha_mode=standby
$ chdev -dev ent21 -attr ha_mode=standby

7. Mount the packages downloaded NFS filesystem.
# mount $nim_server:/nim/$location_of_vios_patches  /mnt
# cd /mnt

8. Update the VIO Server using updateios command and wait until it completed. Next reboot the VIO Server.

Run the following command to perform updating VIOS.
$ updateios -accept -install -dev /mnt -> it will install the VIOS patches

Upon successful update completion take a reboot of server   .

$ shutdown -restart

Once server came up, accept the license and check the VIOS version.

$ license -accept
$ ioslevel

9. Verify the latest VIOs version and check if any missing packages and re-mirror the rootvg.

Check consistency of currently installed filesets
# instfix –i | grep ML
# lppchk –v
$ ioslevel
Once everything confirmed, re-mirror the rootvg.

$ extendvg rootvg hdisk1
$ mirrorios hdisk1
$ bootlist -mode normal -ls (to view)
$ oem_setup_env (to get root shell)
# bosboot -ad hdisk1 (using root shell)
# bootlist -m normal  hdisk0 hdisk1 (using root shell)
# exit (using root shell, to go back to VIOS shell from root shell)

10. On VIOS2: Change the SEA  adapters mode to "auto" and proceed with other VIOS.
$ chdev -dev ent20 -attr ha_mode=auto
$ chdev -dev ent21 -attr ha_mode=auto

Once everything is fine and post-checks completed on VIOS2 then proceed with VIOS1.

11. On VIOS1: Move the sea adapters to standby mode and then proceed with VIOS update.

# lsdev -Cc adapter|grep -i shared

# entstat -d ent20 |grep -i state
# entstat -d ent21 |grep -i state

$ chdev -dev ent20 -attr ha_mode=standby
$ chdev -dev ent21 -attr ha_mode=standby

12. Follow the above step 4,5,7,8 and 9 on VIOS1.

Break the rootvg mirror and take the clone for quick rollback. If your VIO server with single disk take the mksysb backup.
Remove the all interim fixes and commit the filesets in current version.
Mount the packages downloaded NFS filesystem.
Update the VIO Server using updateios command and wait until it completed. Next reboot the VIO Server.
Verify the latest VIOs version and check if any missing packages and re-mirror the rootvg.

13. Now change the SEA  adapters mode to "auto" and verify the SEA status.
$ chdev -dev ent20 -attr ha_mode=auto
$ chdev -dev ent21 -attr ha_mode=auto

# entstat -d ent20 |grep -i state
# entstat -d ent21 |grep -i state

Once everything is fine and post-checks completed on VIOS1. Monitor the VIO servers for sometime and Verify if any errors

VIOS Backout/Rollback Procedure :
Change the bootlist and Reboot the server with older version.

#bootlist –m normal hdisk1 hdisk0
#bootlist –m normal –o

#shutdown –Fr now
Once the server is came up, Re-mirror the rootvg .

$ extendvg rootvg hdisk1
$ mirrorios hdisk1
$ bootlist -mode normal -ls (to view)
$ oem_setup_env (to get root shell)
# bosboot -ad hdisk1 (using root shell)
# bootlist -m normal  hdisk0 hdisk1 (using root shell)
# exit (using root shell, to go back to VIOS shell from root shell

                     Thanks for reading this post, any queries please write a comment

Take the rootvg backup using Alternate disk

Note: To support this blog.Please click the ads displayed in this Blog.That would be helpful for me.Continue with this post.

This method is useful to take the backup to alternate disk and no impact to original rootvg. We can easily roll back the server just by changing the bootlist.
If rootvg got corrupted or any issues after patching the server , we can easily roll back the server.

Required filesets:

bos.alt_disk_install.boot_images
bos.alt_disk_install.rte
bos.msg.en_US.alt_disk_install.rte

Flags 

-B

       Would specify not running bootlist after the mksysb or clone. If set, the -r flag cannot be used. Note: The -B and -X flags are mutually exclusive.

-C

            Clone rootvg.

-S 

       Will "put-to-sleep" the volume group. This is used after a volume group "wake-up". (W).

-W disk

       Used to "wake-up" a volume group for data access between the rootvg and the alternate disk rootvg.

-v new_volume_group_name disk 

       Used to rename the alternate disk volume group. This is especially useful when creating multiple alternate disks, on multiple volume groups,and name identification is necessary.

Examples:

To take the clone of rootvg, not change bootlist.

# alt_disk_install -B -C hdisk1

To "wake_up" an original rootvg, after booting from the new alternate disk:

            alt_disk_install -W hdisk0

Illustrated Example

            # lspv

            hdisk0          000040445043d9f3     old_rootvg

            hdisk1          00076443210a72ea     rootvg

            # alt_disk_install -W hdisk0

            # lspv

            hdisk0          000040445043d9f3     altinst_rootvg

            hdisk1          00076443210a72ea     rootvg

At this point, the "altinst_rootvg" volume group is varied-on and the /alt_inst file systems will be mounted.

To "put-to-sleep" a volume group that had experienced a "wake-up":

            alt_disk_install -S

Illustrated Example

            # lspv

            hdisk0          000040445043d9f3     altinst_rootvg

            hdisk1          00076443210a72ea     rootvg

            # alt_disk_install -S

            # lspv

            hdisk0          000040445043d9f3     altinst_rootvg

            hdisk1          00076443210a72ea     rootvg

The "altinst_rootvg" is no longer varied-on and the /alt_inst file systems are no longer mounted. If it's necessary for the "altinst_rootvg" volume group name to be changed back to "old_rootvg", this can be done with the "-v" flag.

To modify an alt_disk_install volume group name: 

            alt_disk_install -v alt_disk_432 hdisk2

Illustrated Example

            # lspv

            hdisk0          00006091aef8b687     rootvg

            hdisk1          00000103000d1a78     rootvg

            hdisk2          000040445043d9f3     altinst_rootvg

            hdisk3          00076443210a72ea     altinst_rootvg

            hdisk4          0000875f48998649     None

            hdisk5          000005317c58000e     None

            # alt_disk_install -v alt_disk_432   hdisk2

            #lspv

            hdisk0          00006091aef8b687     rootvg

            hdisk1          00000103000d1a78     rootvg

            hdisk2          000040445043d9f3     alt_disk_432

            hdisk3          00076443210a72ea     alt_disk_432

            hdisk4          0000875f48998649     None

            hdisk5          000005317c58000e     None

Process to take the alternate disk backup:

1. If rootvg is mirrored, break the mirror and free up one disk then proceed to take the backup using alternate disk method.

2. Take the alternate disk backup using the command alt_disk_install without changing the bootlist of AIX server.

# alt_disk_install -B -C <hdiskX>

3. Verify the bootlist and boot image created hdiskX.

# bootlist -m normal -o
# ipl_varyon -i

To roll back the server :

1. Verify and Change the bootlist to altinst_rootvg disk(hdiskX).

# bootlist -m normal -o

# bootlist -m normal  <hdiskX> hdisk0

2. Reboot the server using alternate disk(altinst_rootvg).

# shutdown -Fr now

3. Once server came back, verify whether it is booted with alternate disk or not. Make sure old disk with old_rootvg

# lspv

To remove the alternate disk backup we have execute the below command.

# alt_disk_install -X

Break the rootvg mirror and take the clone for quick rollback. If your VIO server with single disk take the mksysb backup:

To break the rootvg mirror and create a clone for quick rollback

$ bootlist -mode normal –ls

$ unmirrorios hdisk1

#  migratepv hdisk1 hdisk0     (Run as root user)

$ reducevg -rmlv rootvg hdisk1

$ alt_root_vg -target hdisk1

$oem_setup_env (to get root shell)

# bosboot -ad hdisk0 (using root shell)

# bootlist -mode normal hdisk0 hdisk1 (using root shell)

Patching using alternate disk:
If rootvg is mirrored, break the mirror and free up one disk then proceed with AIX patching using alternate disk method.

# alt_disk_copy -d <altdiskname> -b update_all -l <TL Location>

# alt_rootvg_op -W -d hdisk8 

# chroot /alt_inst /usr/bin/ksh

# oslevel -s

# exit

# alt_rootvg_op -S altinst_rootvg

AIX LVM : File System Commands List

This post describes that few AIX LVM Volume Group Commands. Below are the list.

Lists all filesystems in the /etc/filesystems entry.
 # lsfs                                                          
                                                     
List all filesystems with detailed info.
 # lsfs -q

To list all filesystems (default).
 # lsfs -a

To specify the output in list format.
 # lsfs -l

To specify the output in column format.
 # lsfs -c  

To lists all jfs filesystems.                                                    
 # lsfs -v jfs                                                

To lists all jfs2 filesystems.
 # lsfs -v jfs2

Creates FS /testfs of type jfs in VG testvg of blocksize 64465.
 # crfs -v jfs -g testvg -a size=64465 -m /testfs

Create FS /testfs of type jfs on device /dev/lv00.
 # crfs -v jfs -d /dev/lv00 -m /testfs

Create a Filesystem /test of type jfs2 on VG testvg with 10G size.
crfs -v jfs2 -g testvg -m /test -A yes -a size=10G -p rw

Create a Filesystem /test of type jfs2 on LV testlv(Not required to specify size here)
crfs -v jfs2 -d testlv -m /test -A yes -p rw

To change size of FS /si to 24576 x 512 bytes blocks (12 MB).
 # chfs -a size=24576 /si

To add 24576 x 512 byte blocks to FS /si.
 # chfs -a size=+24576 /si

To decrease the filesystem size in AIX.
 # chfs -asize=-500M /file/system

To change the mount point from /si to /bi
 # chfs -m /si /bi

To auto mount the filesystem si.
 # chfs -A /si

To Remove account attribute of /si. ( from /etc/filesystems file).
 # chfs -d account /si

To mount the 2nd copy of mirrored filesystem oracle to /backup in read-only mode.
 # chfs -a splitcopy=/backup -a copy=2 /oracle

To delete filesystem /testfs and associated LV.
 # rmfs /testfs

To deletes FS /testfs its mount point and associated LV.
 # rmfs -r /testfs

To defragment the file system /sifs.
 # defragfs /sifs

Display the current defrag status of the file system.
 # defragfs -q /sifs

To fsck the filesystem associated to /dev/lv00 assuming response "yes"
 # fsck  -y n /dev/lv00

To restore superblock from backup superblock
fsck -p /dev/lv00
 

Previous posts we were discussed commands list for physical volumes AIX LVM Physical Volume Commands
AIX Volume Group Commands AIX LVM Volume Group Commands
AIX Logical Volume commands AIX Logical Volume Commands

AIX LVM : Logical Volume Commands List

This post describes that few AIX LVM Logical Volume Commands. Below are the commands list.

Creates a testlv logical volume.
 # mklv -y (lv) -t jfslog (vg) (No of PP's) (pv Name optional)

 # mklv -y testlv -t jfs2 testvg 20 hdiskXX

To display info about LV by PV.
 # lslv -l lv00

To display LV allocation map for hdisk1.
 # lslv -p hdisk1

To change the fslv00 to copy type.
 # chlv -t copy fslv00

To rename LV fslv00 to silv.
 # chlv –n silv fslv00

To change the ora1data_lv to <XXX to 950 PP>.
 # chlv -x 950 ora1data_lv

To change the lv00 to readonly mode.
 # chlv -p r lv00

To remove the logical volume testlv.
 # rmlv testlv

To remove testlv without user intervention.
 # rmlv -f testlv

To make LV testlv  with three copies on hdisk1.
 # mklv  -s n -c 3 testlv hdisk1

To extend the LV testlv with 5 LPs.
 # extendlv  testlv 5

To mirror LV testlv on same PV with 2 copies.
 # mklvcopy -s n testlv 2 hdisk1

To mirror LV testlv on PV hdisk1 and hdisk2 with 3 copies.
 # mklvcopy testlv 3 hdisk1 hdisk2

To remove one copy of  LV testls from hdisk1.
 # rmlvcopy testlv 2 hdisk1

To create a jfslog with name log00 on VG newvg with 2LPs.
 # mklv -t jfslog -y log00 newvg 2

To format jfslog volume log00.
 # logform /dev/log00

Move a logical volume between physical volumes. Both physical volumes must be in the same volume group.
 # migratepv -l (lv) (old pv) (new pv)

Initializes an LV for use as an JFSlog.
 # logform (/dev/lv)

Displays Logical Volume Control Block information.
 # getlvcb -AT (lv)

Previous post describes commands list for physical volumes AIX LVM Physical Volume Commands and AIX Volume Group Commands AIX LVM Volume Group Commands
 

AIX LVM : Volume Group Commands List

This post describes that few AIX LVM Volume Group Commands. Below are the list.

To display all VGs in server.
 # lsvg

To display all active VGs in server.
 # lsvg -o

To display complete information about rootvg.
 # lsvg rootvg

To display information about all LVs in rootvg.
 # lsvg  -l rootvg

To display information about all LVs in all VGs.
 # lsvg -o |lsvg -il

To display information about all PVs in rootvg.
 # lsvg  -p rootvg

Create VG with name testvg on hdisk1 with physical partition size 256MB.
 # mkvg -y testvg -s 256 hdisk1

Create VG with name oravg on hdiskXX with partition size 512MB.
 # mkvg -s 512 -y oravg hdiskXX

Create BIG Volume Group.
 # mkvg -B -y  bigvg -s 512 hdisk1

Create scalable VG with name testvg.
 # mkvg -S -y testvg -s 512 hdisk1

Create testvg on hdisk1 with PP size 4 and no of partions 2 * 1016.
 # mkvg -s 4 -t 2 -y testvg hdisk1

To change VG newvg automatically activated at startup.
 # chvg -a y newvg

To deactivate the automatic activation at startup.
 # chvg -a n newvg

To change maximum no. of PP to 2032 on vg newvg.
 # chvg -t 2 newvg

To disable quorum on VG newvg.
 # chvg -Q n newvg

To reorganises PP allocation of  VG newvg.
 # reorgvg  newvg

To add PV hdisk3 and hdisk4 to VG newvg.
 # extendvg newvg hdisk3 hdisk4

To Export the VG newvg.
 # exportvg newvg

To import the hdisk2 with name  newvg, and assign major number 44.
 # importvg -V 44 -y newvg   hdisk2

To remove PV hdisk3 from VG newvg.
 # redcucevg newvg hdisk3

To deactviate VG newvg.
 # varyoffvg newvg

To activate VG newvg.
 # varyonvg newvg

To sync the mirrored LV in the VG oravg.
 # syncvg -v oravg

To mirror lv’s of sivg with hdisk2(-m for extract mirror,-S for forbackground miror).
 # mirrorvg –S –m sivg hdisk2

To remove the mirrored PV from the set.
 # unmirrorvg sivg hdisk2

 

Previous post describes commands list for physical volumes AIX LVM Physical Volume Commands
 

AIX LVM : Physical Volume Commands List

This post describes that few AIX LVM Physical Volume Commands. Below are the list.

To displays the all physical volumes.
 # lspv

To display  PP usage of hdisk0.
 # lspv -p hdisk0

To list all logical volumes on PV hdisk0.
 # lspv -l hdisk0

To Close the PV (Used while removing PV without varyon).
 # chpv -v r hdisk1

To open the PV in AIX.
 # chpv -v a hdisk1

To clear the master boot record on PV hdisk0.
 # chpv -c hdisk0

To move PPs from PV hdisk1 to hdisk2.
 # migratepv  hdisk1 hdisk2

To migrate LV silv from PV hdisk1 to hdisk2.
 # migratepv -l silv hdisk1 hdisk2

To change or assign the pvid to new disk (Don't execute on existing PV).
 # chdev -l hdiskXX -a pv=clear

 # chdev -l hdiskXX -a pv=yes

Dump device is too small Errpt E87EF1BE

This post gives you the solution if you got the errpt error Dump device is too small in AIX.

Error :

LABEL:          DMPCHK_TOOSMALL
IDENTIFIER:     E87EF1BE

Date/Time:       Sat May 29 15:00:00 EDT 2021
Sequence Number: 2452
Machine Id:      XXXXXXXXX
Node Id:         TEST01
Class:           O
Type:            PEND
WPAR:            Global
Resource Name:   dumpcheck      

Description
The largest dump device is too small.

Probable Causes
Neither dump device is large enough to accommodate a system dump at this time.

Recommended Actions
Increase the size of one or both dump devices.

Detail Data
Largest dump device
lg_dumplv                                                                                                                      
Largest dump device size in kb
     2621440
Current estimated dump size in kb
     3984302
 

Soluction :

1. Verify the how many dump devices are configured in server using below command.

# sysdumpdev -l

2. Verify the largest dump device using below command(dumpcheck -p) .

# /usr/lib/ras/dumpcheck -p
The largest dump device is too small.

Largest dump device
         lg_dumplv
Largest dump device size in kb
         2621440
Current estimated dump size in kb
         3984302

Above output states that we have to increase the dump logical volume 'lg_dumplv' to estimated dump size in 3984302 KB(3890.9 MB).

3. Verify the rootvg whether we have Free PP's there or not. If we have free PP's to meet the estimated dump size then increase the dump logical volume. As per the step 2, current dump size is 2560 MB and
estimated dump size is 3890.9 MB.
To fix this issue we have increase the dump size 1330.9 MB. If your PP size is 128 MB, you have increase the dump logical volumen by 11 LP (1330.9 / 128 = 10.39) . Execute the below commands.

To verify the Free PP's and PP size.
# lsvg rootvg

4. Extend the dump logical volume by 11 LP's.

# extendlv lg_dumplv 11

5. Execute the dumpcheck command again to check if still it showing the any error. Shown nothing means issue got fixed.

#   /usr/lib/ras/dumpcheck -p

 

                                                            Thanks for reading this Post

Recover the root password in AIX

 In this post we will discuss, how to recover the root password in AIX.

1.  Push the NIM resources like lpp_source and SPOT or mksysb and SPOT to the server(Issued server). This oslevel of resources should be equal or above.

2. Take the downtime and boot the server in SMS mode. Once the system is Service Mode and you've booted on the bootable media/ booted using NIM resources,which is of the same version and level as the system, follow the prompts to the Welcome to Base OS menu.

3. Next Choose Start Maintenance Mode for System Recovery (Option 3).

4. The next screen displays prompts for the Maintenance Menu

             Choose Option 1: Acess a Root Volume Group

5. The next screen prints a warning message and asks you to select 0 to continue.
             Select 0 to Continue

6. The next screen asks you to select the proper volume group.
            Select the proper volume group for your machine.

7. The console will display information about rootvg and a menu with two options
                Choose option 1: Access this volume group and start a shell

8. Once you have the command prompt, run the passwd command:

               # passwd
               Changing password for "root"
                root's New password:
                Enter the new password again:
7. With the key in Normal position (if applicable), issue the reboot command: sync;sync;sync;reboot

 

                                                            Thanks for reading this Post