I saw Alma had 9.2 available this month so decided to upgrade my 8.7 versions.
My old sledgehammer method of “dnf distro-sync –releasever=nn” did not work for this. It may have if I had spent months resolving conflicting packages but there is an easier way.
This post at https://www.layerstack.com/resources/tutorials/How-to-upgrade-from-AlmaLinux8-to-AlmaLinux9 covered all the key steps. It has a few unnecessary steps and omitted a few issues you will hit that I have added here.
Look at the things to watch out for at the end of this post before deciding if you want to actually do this.
Things not mentioned in that post are you have to cleanup any old gpg-keys stored in rpm, I also had to remove one conflict package (even though pre-upgrade found no issues), and the biggee was finding that the install had inserted a grub command line option to set selinux to permissive even if /etc/selinux/config was set to enforcing. That last biggee may have resolved itself the next time you get a new kernel version but if you are using enforcing you probably do not want to wait that long.
Warning: do not do the initial step to fully dnf update your installed system. I did that on one of my systems and it upgraded it from Alma8.7 to Alma8.8; the “leapp” (as of 21May2023) only supports upgrades from 8.7 and flags 8.8 as unsupported/inhibitor for upgrade to 9.2. They may have fixed that by the time you read this of course.
Another thing of note which I have repeated in the “Things to watch out for” list is that when it reboots to do the upgrade if you have a console session you will see it drops into emergency recovery mode with the dracut prompt, do not as I first did start issuing commands to find out what has gone wrong… just wait, the upgrade wants to be in that place.
And a critical warning the legacy network configuration method of using config files in /etc/sysconfig/network-scripts depreciated in rhel 8 is now fully removed in rhel 9; if you use that method to configure the network your 9,x machine will boot without any networking configured. So you must fully convert your machine to use only NetworkManager before upgrading (and yes, I did ignore that and ended up with a machine with no network configured, but fortunately with a console). HOWEVER on one server I upgraded using network-scripts the network was not configured after the upgrade, on another el9 the legacy scripts did configure the network successfully; so I don’t know what is going on there.
Then the initial steps to upgrade are as below.
# First "vi /etc/firewalld/firewalld.conf" and set "AllowZoneDrifing" off setenforce 0 curl https://repo.almalinux.org/elevate/testing/elevate-testing.repo -o /etc/yum.repos.d/elevate-testing.repo rpm --import https://repo.almalinux.org/elevate/RPM-GPG-KEY-ELevate yum install -y leapp-upgrade leapp-data-almalinux leapp preupgrade # READ THE REPORT AND FIX ANY ISSUES AND REPEAT preupgrade ABOVE UNTIL NO INHIBITORS leapp upgrade grub2-install /dev/sda # << whatever your disk is (see Note1 below) reboot # DO NOT CUT/PASTE THIS LINE, you will want to check for errors first
Note1: On legacy BIOS machines the report produced in the preupgrade and upgrade steps highlights it will not upgrade grub on legacy boot machines (ie: any VM or cloud image) and needs the grub2-install run. My VM boot partition was /dev/sda1 but that was a ext4 filesystem grub2-install refused to write (or may have as it issued lots of warnings rather than errors) to so I used /dev/sda which seems to have worked (now also done for a few /dev/vda so that seems the correct approach).
After rebooting check the version in /etc/os-release is indeed 9.2 (or later is you are reading this long after I typed it in). It is important to note that the upgrade is not completed at the time of that last reboot. If you have a console session you wil see that the "leapp" app is still performing post-install activites for quite a while after it has rebooted and allowed ssh logons.
The upgrade does not upgrade anything from third party repos so you need to do that yourself after the upgrade and do another relabel.
dnf -y update touch /.autorelabel reboot
Things to watch out for
- At 21May2023 if you have upgraded to Alma 8.8 (done if you have done a "dnf update lately") you cannot upgrade... not until they update the "leapp" script anyway as the last version it supports for upgrade is 8.7
- After the upgrade you must run another dnf update as nothing from third party repos is upgraded by "leapp" (probably a wise design decision as they cannot all be tested).
- Not all third party repositories have packages (or even repos) for 9.x yet. DNF behaviour seems to have changes to abort if a repo cannot be reached rather than skipping it and moving onto the next so unless you are faliar with -the disable-repo flag life can be painful here
- Look through the preupgrade report and fix as many issues as you can, do not proceed until you have resolved all inhibitors
- You must convert any legacy network config in /etc/sysconfig/network-scripts to a NetworkManager config as legacy network configuration is not supported at all in rhel 9. You can try "nmcli connection migrate" to attempt to convert your configurations but no guarantees if it is a complicated one.
- you will find that the upgrade has inserted an enforcing=0 into your grub command line so even if /etc/selinux/config is set to enforcing it will be ignored and you will be in permissive mode . That may fix itself on the next kernel update that updates grub but you may want to fix that yourself.
- As noted above you have to manually run another update to update from 3rd party repositories.
- If you have any SHA1 keys stored in RPM the upgrade will be inhibited until you remove them.
- IPTables has been depreciated and while the upgrade still leaves them available for now some packages that used to use iptables no longer have versions with iptables support (for example fail2ban was removed from my servers suring the upgrade as only firewalld versions are now available in the repos).
- If upgrading from a machine (or VM) you have had for a while ypu may see in the preupgrade report issues like "OLD PV header" in which case you must update that before doing the upgrade. That is simply "vgck --updatemetadata YourPVNameHere"
- And absolutely horrible things to ignore, if like me you have a console available when you do the reboot for the upgrade you will see truely horrible messages about the systen dropping into emergency recovery mode with journal errors. You have plenty of time to try and start troubleshooting as it is in recovery mode and you can happily enter commands, don't. After 2-3 minutes you will see "leapp" messages start to appear on the console as it starts upgrading.
- On the journal, if you have a journalled filesystem you will see after the system is stable a lot more journal rotation messages logged which is annoying. If like me you have lots of nice automation rules to manage rotating and cleaning journals they will have to be revisited.
Tips: the SHA1 keys
You can display the gpg-keys in RPM with
rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
You can use the standard RPM commands to display details of each key, for example
rpm -qi gpg-pubkey-4bd6ec30-4c37bb40
You can use the output and clean up old keys, ie: if the install date was a while ago it may not be needed). If you have upgraded from other OS'es you may have a lot of old keys, for example I had gpg-pubkeys for multiple versions of Fedora and CentOS (I migrated from Fedora to CentOS, then to Alma, and all the old gpg-pubkey entries were still there)
And the stndard RPM commands to remove/erase old keys
rpm -e gpg-pubkey-4bd6ec30-4c37bb40
If you have SHA1 signed packages you absolutely must keep you can before the preupgrade and upgrade use "update-crypto-policies --set LEGACY" jst remember after you have finished the upgrade use "update-crypto-policies --set DEFAULT" to set it back to the default. You will have to do that flip-flop every time you want to do anything with packages/repos using SHA1 keys.
