Wednesday 8 December 2021

AIX OS Migration from AIX 7.1 to AIX 7.2 using NIMADM through NIMSH service

In this post describes AIX migration from AIX 7.1 from AIX 7.2 using nimsh. AIX migration tool now supports nimsh service(subsystem). You no longer to use the rsh on your NIM clients. So rsh facility is dead.

AIX uses NIM Service Handler (NIMSH) to eliminate the need for rsh services during NIM client communication. The NIM client daemon (NIMSH) uses reserved ports 3901 and 3902, and it installs as part of the bos.sysmgt.nim.client fileset.

NIMADM is a facility to move the server from one operating system(AIX 7.1) to another operating system(AIX 7.2). The nimadm is an utility having more advantages to compare CD/DVD migration. In CD/DVD migration method will override /tmp, means whatever the data exist inside /tmp will removed be in new OS version. 

NIMADM takes a copy of client server rootvg to spare disk(hdisk1) and simultaneously migrate the server to new OS version. This can be done without downtime and no disruption to client. mksysb backup is optional.

Minimum downtime: Once OS migration completed, downtime required for reboot the client to take into new OS version(AIX7.2).  It will take 30-45min per the system.

No Outage : NIMADM is online migration, during the migration process no impact to applications and database.

Easy to Rollback: If applications/Databases are not running as easier, it very simple to rollback to old level. Just change the bootlist and reboot client server. NIMADM fails in middle whatever the changes are made in altinst_rootvg. It won't impact to original rootvg.

NIM Master take the workload, this will reduce the overhead on client

I have tested this new functionality in test environment and it is working fine. The following AIX versions and levels were installed:

NIM Master                   : aixnim01
OSLEVEL of NIM Master : 7200-05-01-2038
NIM Client                         : lpar02
OSLEVEL of NIM Client   : 7100-05-07-2037
lpp_source                       : AIX72TL03_LPP
SPOT                                : AIX72TL03_SPOT
Spare disk                        : hdisk1

Pre-Checks and Implementation on NIM Master :

1. NIM master running AIX7.2 or latest level of Technology Level/Service Pack.
   # oslevel -s

2. Same level of bos.alt_disk_install.rte fileset must be installed on NIM master rootvg and SPOT
   # lslpp -l|grep -i bos.alt_disk_install.rte

3. NIM resources lpp_source and SPOT level must be same or lower version of NIM Master OS level. We won't migrate higher than NIM Master.
   # lsnim -l AIX72TL03_LPP
   # lsnim -l AIX72TL03_SPOT

4. Get the new disk from SAN with same or higher size of NIM master rootvg. To create nimadmvg.
    # mkvg -y nimadmvg -s 256 hdisk1

5. Update ip address and client name in /etc/hosts
    # vi /etc/hosts

6. Define nim client machine.
    # smit nim_mkmac

7. Inactivate the rsh, if it is active. Refresh inetd subsystem.
    # lssrc -ls inetd
    # vi /etc/inetd.conf (Here comment the shell and save the file)
    # cat /etc/inetd.conf | grep shell
    #shell   stream  tcp6    nowait  root    /usr/sbin/rshd         rshd

   # refresh -s inetd

8. bos.alt_disk_install.rte fileset must be installed on NIM master and also in the SPOT.If not you have to add the fileset to SPOT.  
   # lslpp -l bos.alt_disk_install.rte
   # nim -o showres AIX72TL03_LPP | grep -i bos.alt_disk_install.rte (Check on lpp_source)

   # nim -o showres AIX72TL03_SPOT | grep -i bos.alt_disk_install.rte (Check on SPOT)

   # nim -o cust -a filesets=bos.alt_disk_install.rte -a lpp_source=AIX72TL03_LPP AIX72TL03_SPOT (add fileset to SPOT)

Pre-Checks and Implementation on NIM Client :

9. Verify oslevel, missing filesets and commit applied filesets.
    # oslevel -s
    # lppchk -vm3
    # instfix -i | grep ML
    # instfix -i | grep SP
    # installp -c all   (commit applied filesets)

10. Check hardware, software and firmware levels must be support to required AIX level which you migrating.
     # lsmcode -c

11. Get spare disk with same or higher size of client rootvg. If rootvg mirrored break it and use it for migration.
    # lspv

If rootvg mirrored :

   # bootinfo -b
   # mirrorvg rootvg hdisk1
   # reducevg -df rootvg hdisk1
   # chpv -c hdisk1
   # bootlist -m normal -o
   # bootlist -ad /dev/hdisk0
   # bootlist -m normal -o

12. lv names must be less than 11 characters.
    # lsvg -l rootvg

13. Inactivate the rsh, if it is active. Refresh the inetd subsystem.

   # lssrc -ls inetd   
   # vi /etc/inetd.conf (Here comment the shell and save the file)
   # cat /etc/inetd.conf | grep shell
   #shell   stream  tcp6    nowait  root    /usr/sbin/rshd         rshd

   # refresh -s inetd

14. Start the nimsh subsystem, if it is not active.
     # lssrc -s nimsh
     # startsrc -s inetd 

15. Take the backup of system configuration files like /etc/sendmail.cf , /etc/motd, netstat -rn,vmo -a, ioo -a ...etc

16. Verify the /etc/niminfo file. NIM Master details must updated on this file. If not exist run (#smit nim) and enter NIM master details.

    # cat /etc/niminfo
    # smit nim (to create /etc/niminfo file)

17. Take the mksysb backup if you want to.

   # mksysb -i <destination path>

Run NIMADM Command on NIM Master:

# nimadm -j <nimadmvg> -c <clientname> -s <SPOT> -l <lpp_source> -d <spare disk> -Y

# nimadm -j nimadmvg -c aixnim01 -s AIX72TL03_SPOT -l AIX72TL03_LPP -d hdisk1 -Y 

# nimadm -j nimadmvg -c aixnim01 -s AIX72TL03_SPOT -l AIX72TL03_LPP -d hdisk1 -Y >> /tmp/nimadm.log &  #### ----> [ To run in background ]

   -j VG specified on NIM Master
   -c client machine name defined in NIM Master
   -l Name of lpp_source defined in NIM Master
   -s Name of SPOT defined in NIM Master
   -d spare disk for alternate rootvg(altinst_rootvg)
   -Y Accept license

Once migration starts cache file systems create the in nimadmvg and removes the after successful migration.

The migration process will takes in 12 phases .The following is the output of 12 phases while migration is going on

Post Migration :

1. After NIMADM completed, check the bootlist and set to hdisk1 (altinst_rootvg).

   # bootlist -m normal -o
   # bootlist -m normal hdisk5 hdisk0

2. Inform to application/database team to stop  applications/database.

3. Reboot the server and once it came up check the oslevel.

   # oslevel -s
   # shutdown -Fr (It will take few minutes to Power UP)
   # oslevel -s
   # lppchk -vm3
   # instfix -i | grep ML
   # instfix -i | grep SP

4. Inform to application/database team to start  applications/database


NIMADM 12 Phases :

Initializing the NIM master.

Initializing NIM client aixnim01

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/aixnim01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 1.

Client alt_disk_install command: alt_disk_copy -j -M 7.2 -P1 -d "hdisk1"

Calling mkszfile to create new /image.data file.

Checking disk sizes.

LOGICAL_VOLUME= hd11admin

FS_LV= /dev/hd11admin

Creating cloned rootvg volume group and associated logical volumes.

Creating logical volume alt_hd5.

Creating logical volume alt_hd6.

Creating logical volume alt_hd8.

Creating logical volume alt_hd4.

Creating logical volume alt_hd2.

Creating logical volume alt_hd9var.

Creating logical volume alt_hd3.

Creating logical volume alt_hd1.

Creating logical volume alt_hd10opt.

Creating logical volume alt_hd11admin.

Creating logical volume alt_lg_dumplv.

Creating logical volume alt_livedump.

Creating /alt_inst/ file system.

Creating /alt_inst/admin file system.

Creating /alt_inst/home file system.

Creating /alt_inst/opt file system.

Creating /alt_inst/tmp file system.

Creating /alt_inst/usr file system.

Creating /alt_inst/var file system.

Creating /alt_inst/var/adm/ras/livedump file system.

Generating a list of files

for backup and restore into the alternate file system...

Phase 1 complete.

+-----------------------------------------------------------------------------+

Executing nimadm phase 2.

+-----------------------------------------------------------------------------+

Creating nimadm cache file systems on volume group nimadmvg.

Checking for initial required migration space.

Creating cache file system /aixnim01_alt/alt_inst

Creating cache file system /aixnim01_alt/alt_inst/admin

Creating cache file system /aixnim01_alt/alt_inst/home

Creating cache file system /aixnim01_alt/alt_inst/opt

Creating cache file system /aixnim01_alt/alt_inst/tmp

Creating cache file system /aixnim01_alt/alt_inst/usr

Creating cache file system /aixnim01_alt/alt_inst/var

Creating cache file system /aixnim01_alt/alt_inst/var/adm/ras/livedump

+-----------------------------------------------------------------------------+

Executing nimadm phase 3.

+-----------------------------------------------------------------------------+

Syncing client data to cache ...

+-----------------------------------------------------------------------------+

Executing nimadm phase 4.

+-----------------------------------------------------------------------------+

nimadm: There is no user customization script specified for this phase.

+-----------------------------------------------------------------------------+

Executing nimadm phase 5.

+-----------------------------------------------------------------------------+

Saving system configuration files.

Checking for initial required migration space.

Setting up for base operating system restore.

/aixnim01_alt/alt_inst

Restoring base operating system.

Merging system configuration files.

Can not remove the glbd subsystem

WARNING: rmitab failed! Unable to remove "mkatmpvc" entry from /etc/inittab.

WARNING: rmitab failed! Unable to remove "atmsvcd" entry from /etc/inittab.

No such file or directory

/lpp/devices.common.IBM.atm/deinstl/devices.common.IBM.atm.rte.unconfig[76]: /etc/atm.defs: cannot open

sum: Can't open /usr/lib/hal_hfi.o

sum: Can't open /usr/hfi/hal/lib/hal_hfi.o

sum: Can't open /usr/lib/hal_hfi64.o

sum: Can't open /usr/hfi/hal/lib/hal_hfi64.o

Running migration merge method: ODM_merge Config_Rules.

Running migration merge method: ODM_merge SRCextmeth.

Running migration merge method: ODM_merge SRCsubsys.

Running migration merge method: ODM_merge SWservAt.

Running migration merge method: ODM_merge pse.conf.

Running migration merge method: privcmds_mig.

Running migration merge method: privdevs_mig.

Running migration merge method: ODM_merge vfs.

Running migration merge method: ODM_merge xtiso.conf.

Running migration merge method: ODM_merge PdAtXtd.

Running migration merge method: ODM_merge PdDv.

Running migration merge method: convert_errnotify.

Running migration merge method: passwd_mig.

Running migration merge method: login_mig.

Running migration merge method: user_mrg.

Running migration merge method: secur_mig.

Running migration merge method: RoleMerge.

Running migration merge method: methods_mig.

Running migration merge method: mkusr_mig.

Running migration merge method: methods_mig.

Running migration merge method: mkusr_mig.

Running migration merge method: group_mig.

Running migration merge method: ldapcfg_mig.

Running migration merge method: ldapmap_mig.

Running migration merge method: secvarscfg_mig.

Running migration merge method: convert_errlog.

Running migration merge method: ODM_merge GAI.

Running migration merge method: ODM_merge PdAt.

Running migration merge method: merge_smit_db.

Running migration merge method: ODM_merge fix.

Running migration merge method: merge_swvpds.

Running migration merge method: SysckMerge.

Running migration merge method: nscontrol_mig.

+-----------------------------------------------------------------------------+

Executing nimadm phase 6.

+-----------------------------------------------------------------------------+

Installing and migrating software.

Checking space requirements for installp install.

Expanding /aixnim01_alt/alt_inst/usr local filesystem.

Filesystem size changed to 5767168

Expanding /aixnim01_alt/alt_inst/var local filesystem.

Filesystem size changed to 786432

Installing software with the installp installer.

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

.

.

.

install_all_updates: Initializing system parameters.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Checking for updated install utilities on media.

install_all_updates: Processing media.

install_all_updates: Generating list of updatable installp filesets.

#---------------------------------------------------------------------

# No filesets on the media could be used to update the currently

# installed software.

#

# Either the software is already at the same level as on the media, or

# the media contains only filesets which are not currently installed.

#---------------------------------------------------------------------

install_all_updates: Checking for recommended maintenance level 7200-03.

install_all_updates: Executing /usr/bin/oslevel -rf, Result = 7200-03

install_all_updates: Verification completed.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Result = SUCCESS

Restoring device ODM database.

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

nimadm: There is no user customization script specified for this phase.

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 57345 512 byte blocks.

Writing boot image to client's alternate boot disk hdisk1.

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Expanding /alt_inst/usr client filesystem.

Filesystem size changed to 5242880

Adjusting size for /var

Adjusting size for /var/adm/ras/livedump

Syncing cache data to client ...

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /aixnim01_alt/alt_inst/var/adm/ras/livedump

forced unmount of /aixnim01_alt/alt_inst/var/adm/ras/livedump

forced unmount of /aixnim01_alt/alt_inst/var

forced unmount of /aixnim01_alt/alt_inst/var

forced unmount of /aixnim01_alt/alt_inst/usr

forced unmount of /aixnim01_alt/alt_inst/usr

forced unmount of /aixnim01_alt/alt_inst/tmp

forced unmount of /aixnim01_alt/alt_inst/tmp

forced unmount of /aixnim01_alt/alt_inst/opt

forced unmount of /aixnim01_alt/alt_inst/opt

forced unmount of /aixnim01_alt/alt_inst/home

forced unmount of /aixnim01_alt/alt_inst/home

forced unmount of /aixnim01_alt/alt_inst/admin

forced unmount of /aixnim01_alt/alt_inst/admin

forced unmount of /aixnim01_alt/alt_inst

forced unmount of /aixnim01_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /aixnim01_alt/alt_inst

Removing cache file system /aixnim01_alt/alt_inst/admin

Removing cache file system /aixnim01_alt/alt_inst/home

Removing cache file system /aixnim01_alt/alt_inst/opt

Removing cache file system /aixnim01_alt/alt_inst/tmp

Removing cache file system /aixnim01_alt/alt_inst/usr

Removing cache file system /aixnim01_alt/alt_inst/var

Removing cache file system /aixnim01_alt/alt_inst/var/adm/ras/livedump

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 3.

Client alt_disk_install command: alt_disk_copy -j -M 7.2 -P3 -d "hdisk1"

## Phase 3 ###################

Verifying altinst_rootvg...

Modifying ODM on cloned disk.

forced unmount of /alt_inst/var/adm/ras/livedump

forced unmount of /alt_inst/var/adm/ras/livedump

forced unmount of /alt_inst/var

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk1 blv=hd5

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

Cleaning up alt_disk_migration on client aixnim01.

#


Thanks for reading this Post

Friday 3 December 2021

BLV Issue 0301-168 bosboot: The current boot logical volume, /dev/, does not exist on /dev/hdisk0.

This Post resolves the below BLV issue. If you got this error while creating/re-creating the boot image on rootvg disk, follow the below steps to resolve the issue.

# bosboot -ad /dev/hdisk0

0516-602 lslv: Logical volume name not entered.

Usage: lslv [-L] [-l | -m] [-n DescriptorPV] LVname

       lslv: [-L] [-n DescriptorPV] -p PVname [LVname]

Lists the characteristics of a logical volume.

0301-168 bosboot: The current boot logical volume, /dev/,

        does not exist on /dev/hdisk0.

Solution:

1. Check if hd5 logical volume exists on rootvg disk. If rootvg is mirrored, verify both disks.

# lsvg -p rootvg
# lsvg -l rootvg
# lspv -l hdisk0
# lspv -l hdisk1

2. Logical Volume hd5 must be on the 1st PP of the disk.

# lslv -m hd5

3. To save the base customized information and see verbose output. If it shows no boot image found on disk, next we have to remove and re-create the logical volume hd5.

# savebase -v
No boot images found on disk...

4. Remove and re-create the logical volume hd5.

# rmlv hd5

# mklv -y hd5 -t boot -a e rootvg 1 hdisk0

5. Next remove the file ipldevice under /dev and try to create the boot image on disk. If it shows the same error, then we have to boot device information on hdisk0.

# ls -lrt /dev/ipldevice*
# rm /dev/ipldevice
# ln /dev/rhdisk0 /dev/ipldevice

# bosboot -ad /dev/ipldevice  ( If you got same error )

6. Verify the boot device information on hdisk0. If it displays a 1 the IPL code in the ROS on the machine running the command is capable of booting from the specified disk. Otherwise, the command displays a 0.

# bootinfo -B hdisk0          ( Show output 1 )
# ln /dev/rhd5 /dev/ipl_blv
# cd /dev                             ( Same major /minor numbers)

7. Now we got the same minor and major number for hdisk0.

# ls -ld /dev/ipldevice
crw-------    2 root     system       13,  0 Oct 31 2019  ipldevice

# ls -ld /dev | grep -i "13,  0"
# ls -l /dev | grep "13,  0"
brw-------    1 root     system       13,  0 Dec 03 12:57 hdisk0
crw-------    2 root     system       13,  0 Dec 03 06:59 ipldevice
crw-------    2 root     system       13,  0 Dec 03 06:59 rhdisk0
#

8. Create boot image on ipldevice. Now it will work and next try to create/ re-create boot images on hdisk0.

# bosboot -ad /dev/ipldevice

bosboot: Boot image is 53276 512 byte blocks.

# bosboot -ad /dev/hdisk0
# bosboot -ad /dev/hdisk1



Thanks for reading this post. Please comment if you like it.

Privacy Policy

  Privacy Policy for UNIX TECH INFO At UNIX TECH INFO, accessible from https://unixworld4admin.blogspot.com/, one of our main priorities is ...