5 Troubleshooting Oracle Linux Upgrades
This chapter provides troubleshooting information and describes known issues that might affect the upgrade process.
Tools for Troubleshooting
Use the following options to generate more output when you are generating the preupgrade report or performing the actual upgrade:
-
--verbose
displays warnings, error messages, and other critical information. -
--debug
adds debug information in addition to the same output as the--verbose
option.
You can use the following resources and tools for obtaining troubleshooting information:
-
/var/log/leapp/leapp-report.txt
-
/var/log/leapp/leapp-upgrade.log
-
/var/log/leapp/dnf-debugdata/
: a directory for debug information. Note that this directory is created only if you use the--debug
option when issuing either the preupgrade or the upgrade command. -
journalctl command
Known Issues
The following are known issues that you might encounter when upgrading an Oracle Linux 8 system to Oracle Linux 9.
Upgrade Issues
-
Optional Resilient Storage group isn't supported with Leapp
Leapp doesn't support the optional Resilient Storage group. The presence in the system of certain packages from that group might prevent a leapp upgrade to complete.
For example, the Resilient Storage group includes the
lvm2-cluster
package. In an Oracle Linux 8 installation, this package can be optionally added as part of either the Infrastructure Server and File profile or the Print Server profile. However, the package is an inhibitor and would cause the Leapp upgrade to fail.BugID 33573562
-
Leapp might report missing packages that are marked for installation
The
/var/log/leapp/leapp-preupgrade.log
or/var/log/leapp/leapp-upgrade.log
files might report a warning similar to the following:Warning: Packages marked by Leapp for install not found in repositories metadata: rpcgen python3-pyxattr libnsl2-devel rpcsvc-proto-devel
These packages are in the
Oracle Linux9 Codeready Builder
repository, which is a developer repository and is disabled by default.If the system requires these packages, then during the preupgrade or the upgrade phase, add the
--enablerepo ol9_codeready_builder
option to the appropriate Leapp command, for example:sudo leapp upgrade --oraclelinux --enablerepo ol9_codeready_builder
Repositories that have been enabled during the Leapp upgrade remain enabled on the Oracle Linux 9 system after the upgrade completes.
Alternatively, after completing the upgrade, you can manually install the packages required for your installation by using the dnf command.
Bug ID 32827043
-
Some
el8
packages might not be upgradedThe same rpm -qa command syntax in the previous item that detects MySQL-related
*.el8
packages might also list more*el8
packages on the system that weren't upgraded. Packages might not be upgraded if they were installed from repositories that aren't supported by Leapp, such as developer repositories. For such packages, do the following:-
Go to https://yum.oracle.com and check the Oracle Linux 9 repositories that would serve the packages you need.
-
After the upgrade is completed, manually install the packages from those Oracle Linux 9 repositories.
-
After all the necessary packages have been installed, remove the residual
el8
packages from the system.
Bug ID 32878386
-
-
(aarch64) Upgrade log might report errors related to the vmd module
After completing an upgrade on aarch64 systems, the Leapp upgrade log might report the following message:
dracut-install: Failed to find module 'vmd'
The VMD module doesn't apply to the Arm architecture and therefore, the error message can be safely ignored.
Bug ID 34172552
-
Errors reported that reference legacy directory /var/run
During the upgrade, messages similar to the following that reference the
/var/run
directory are reported:Installing : leapp-upgrade-el8toel9-0.16.0-6.0.3.el8_6.20220810014405.8fa95c0.noarch 6/6 Running scriptlet: leapp-upgrade-el8toel9-0.16.0-6.0.3.el8_6.20220810014405.8fa95c0.noarch 6/6 [/usr/lib/tmpfiles.d/dnssec-trigger.conf:1] Line references path below legacy directory /var/run/, updating /var/run/dnssec-trigger → /run/dnssec-trigger; please update the tmpfiles.d/ drop-in file accordingly. . . .
These messages can be ignored. The upgrade or package installation completes successfully.
As an alternative workaround, you can update the configuration by following the instructions in the message and change the legacy
/var/run/
* directory path to/run/
*.Bug ID 34491952
-
Unsigned package generates warning when upgrading from Oracle Linux 8 to Oracle Linux 9
This issue applies to Oracle Linux 8 systems that were upgraded from Oracle Linux 7 and are later upgraded to Oracle Linux 9.
When you use Leapp to upgrade an Oracle Linux 7 system to Oracle Linux 8, an unsigned package kernel-workaround-0.1-1.el8.src.rpm is installed as part of the upgrade process. The Leapp upgrade successfully completes, but doesn't notify you of the package.
If you later run Leapp to upgrade this same system to Oracle Linux 9, this time the upgrade process detects the existence of the package and generates a warning in the
/var/log/leapp/leapp-report.txt
as follows:---------------------------------------- Risk Factor: high Title: Packages not signed by Oracle found on the system Summary: The following packages have not been signed by Oracle and may be removed during the upgrade process in case Oracle-signed packages to be removed during the upgrade depend on them: - gpg-pubkey - kernel-workaround Key: f5a5d58476a97bf0a8904d00df5d1321189849ad ----------------------------------------
You can disregard the message. The package doesn't prevent the upgrade from completing.
(Bug ID 35343479)
System Management Issues
-
Ksplice Uptrack software displays error messages
During the upgrade, the Oracle Ksplice Uptrack software might report errors similar to the following:
[ 256.033527] upgrade[390]: Upgrading : uptrack-1.2.80-0.el9.noarch 577/1453 [ 256.037151] upgrade[390]: Running scriptlet: uptrack-1.2.80-0.el9.noarch 577/1453 ... [ 256.045914] upgrade[390]: Traceback (most recent call last): [ 256.049230] upgrade[390]: File "/usr/lib/uptrack/access-key-from-uln", line 9, in <module> [ 256.051376] upgrade[390]: from up2date_client import up2dateAuth, rpcServer [ 256.056490] upgrade[390]: File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 74 [ 256.059251] upgrade[390]: os.chmod(path, 0600) [ 256.060997] upgrade[390]: [ 256.062842] upgrade[390]: SyntaxError: invalid token
The report is a known but harmless issue, which you can ignore. After the upgrade is completed, Ksplice continues to operate normally.
File Systems and Storage Issues
-
Systems with Btrfs in a RAID configuration can't be upgraded
A system that uses the Btrfs file system in a RAID configuration can't be upgraded. In the
/var/log/leapp/leapp-report.txt
that's generated by the preupgrade command, this configuration is flagged as an inhibitor and no remedy is provided. If you upgrade the system and that configuration is detected, the upgrade process halts. -
Upgrade blocked if
winbind
andwins
Samba modules are usedIf
winbind
andwins
Samba modules are used in the/etc/nsswitch.conf
, the upgrade is blocked. As a workaround, remove these modules from the file first, then perform the upgrade. After the upgrade is complete, restore these module entries to the file.For more information about configuring these modules, see Oracle Linux 9: Managing Shared File Systems.
-
Hosts with network mounted file systems can't be upgraded
Leapp doesn't support upgrading systems with mounted file systems on network storage, NFS, or iSCSI. As a workaround, unmount the file systems and comment out their entries from
/etc/fstab
. After the upgrade is completed, you can restore the entries and remount the file systems.
Networking Issues
-
Possible upgrade error if system has several NICs with the same prefix as NIC that's used by kernel
The in-place upgrade process might cause an error if the system to be upgraded has more than one NIC that shares the same prefix as the NIC that's used by the kernel, for example
eth
. After the upgrade, the system's network connectivity is lost.For more information, see About Network Interface Names in Oracle Linux 8: Setting Up Networking.
-
NetworkManager might not start after the upgrade completes
After the upgrade, the system's
NetworkManager
might not start because of the failure of its name resolution service. The failure can be verified by checking the status of the service.systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; > Active: inactive (dead) Docs: man:systemd-resolved.service(8) https://www.freedesktop.org/wiki/Software/systemd/resolved
The
/var/log/messages
file also reports the following error:dbus-daemon[742]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.resolve1.service': Unit dbus-org.freedesktop.resolve1.service not found.
To resolve this issue, choose one of the following workarounds:
-
Configure
NetworkManager
to not usesystemd-resolved.service
.Add the following entries to the
/etc/NetworkManager/conf.d/no-systemd-resolved.conf
file:[main] systemd-resolved=false
-
Enable the
systemd-resolved.service
as follows:systemctl enable systemd-resolved.service
Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → /usr/lib/systemd/system/systemd-resolved.service. Created symlink /etc/systemd/system/multi-user.target.wants/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service.
systemctl start systemd-resolved.service
You can also adopt other methods that are more consistent with the network name resolution model that you're using for the specific setup. For useful information, see About Network Interface Names in Oracle Linux 9: Setting Up Networking.
-
Virtualization and Containers Issues
-
KVM virtual machine snapshots might not be listed after an upgrade
After an upgrade, the
libvirtd
service might report snapshot-related error messages similar to the following:libvirtd[53328]: internal error: Failed to parse snapshot XML from file '/var/lib/libvirt/qemu/snapshot/path-to-previous-snapshot-file'
Furthermore, listing available snapshots from before the upgrade generates an empty list.
sudo virsh snapshot-list previous-snapshot-file
Name Creation Time State -------------------------------
As a workaround, reboot the system. At the end of the boot process, the snapshots are listed and available again.
-
libvirtd service might fail to restart in nested virtualization configurations
In nested virtualization setups, the
libvrtd
service might not restart in the nested KVM host after the upgrade.As a workaround, reboot the nested KVM host.
-
Package installation failures reported when upgrading an Oracle Linux
8 KVM host
Installation of userspace packages fail when upgrading a KVM host. This error can also be detected during the preupgrade phase. The Leapp utility would report the issue with messages similar to the following example:
Risk Factor: high Title: Unable to install RHEL 9 userspace packages Summary: ... Fatal glibc error: CPU does not support x86-64-v2 ...
The error occurs because of a missing
cpu mode
declaration in the virtual machine's XML file. To work around this issue, follow these steps:- Run the following command:
sudo virsh edit vm-name
- Add the following declaration to the virtual machine's XML
file:
<cpu mode='host-model' check='partial'/>
- Rerun the preupgrade process.
Note:
This issue is related to the issueKVM virtual machines panic when started on Oracle Linux 9 hosts
that's documented in Oracle Linux 9: Release Notes for Oracle Linux 9. - Run the following command:
Hardware Related Issues
-
Systems with unrecognized hardware can't be upgraded
Support for certain hardware, such as the
e1000
driver, has been removed from RHCK beginning from RHCK 8. The upgrade can't proceed on platforms that have such hardware installed. Even though UEK might continue to support the hardware, the upgrade procedure is still inhibited if the hardware is detected on the system.
Leapp Overlay Size Issues
-
Upgrading might require increased overlay size
Upgrading Oracle Linux 8 systems with a huge number of packages to Oracle Linux 9 might fail because of insufficient space in the Leapp overlay file systems that are used during the upgrade. You might see the following error message:
Error: Transaction test error: installing package package-name needs 4MB on the / filesystem
As a workaround, increase the
LEAPP_OVL_SIZE
variable. The default size is 4096. The actual size you would need might be larger depending on the specific setup. Use the following command:sudo export LEAPP_OVL_SIZE=new-size
Caution:
The new size that you set for this variable must not exceed 50 percent of the available space on the root partition.
Low Open Files Limit Issues
-
Upgrading might require increased open file limit
Upgrading Oracle Linux 8 systems with a low open file limit setting to Oracle Linux 9 might fail. You might see the following error messages (or similar messages) during the upgrade:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 185, in run File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 199, in _call File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 73, in _multiplex File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 146, in _logfile_logging_handler File "/usr/lib64/python3.6/logging/__init__.py", line 1296, in debug File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 40, in emit File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 45, in _do_emit File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 87, in store File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 73, in get_connection File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 36, in wrapper File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 60, in create_connection File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 27, in _initialize_database sqlite3.OperationalError: unable to open database file
This is caused by an increase in the number of files in the Oracle Linux 9 ca-certificates package. In principle, similar issues can also occur if a package is updated that includes a substantially increased number of packaged files. The error appears when leapp reaches the
open files
limit during the Oracle Linux in-place upgrade to Oracle Linux 9.As a workaround, increase the ulimit
open files
variable on the Oracle Linux system before starting the upgrade. The value known to cause issues is 1024. For example, the following command shows a system still using the defaultopen files
limit:ulimit -a | grep open open files (-n) 1024
Change the open file limit to 10000. Do the following:ulimit -Sn 10000
Verify that the change has occurred. For example, the following command shows an Oracle Linux system with an
open files
limit of 10000:ulimit -a | grep open open files (-n) 10000