5 Troubleshooting Oracle Linux Upgrades

This chapter provides troubleshooting information and describes known issues that might affect the upgrade process.

Tools for Troubleshooting

Use the following options to generate more output when you are generating the preupgrade report or performing the actual upgrade:

  • --verbose displays warnings, error messages, and other critical information.

  • --debug adds debug information in addition to the same output as the --verbose option.

You can use the following resources and tools for obtaining troubleshooting information:

  • /var/log/leapp/leapp-report.txt
  • /var/log/leapp/leapp-upgrade.log

  • /var/log/leapp/dnf-debugdata/: a directory for debug information. Note that this directory is created only if you use the --debug option when issuing either the preupgrade or the upgrade command.

  • journalctl command

Known Issues

The following are known issues that you might encounter when upgrading an Oracle Linux 8 system to Oracle Linux 9.

Upgrade Issues

  • Optional Resilient Storage group isn't supported with Leapp

    Leapp doesn't support the optional Resilient Storage group. The presence in the system of certain packages from that group might prevent a leapp upgrade to complete.

    For example, the Resilient Storage group includes the lvm2-cluster package. In an Oracle Linux 8 installation, this package can be optionally added as part of either the Infrastructure Server and File profile or the Print Server profile. However, the package is an inhibitor and would cause the Leapp upgrade to fail.

    BugID 33573562

  • Leapp might report missing packages that are marked for installation

    The /var/log/leapp/leapp-preupgrade.log or /var/log/leapp/leapp-upgrade.log files might report a warning similar to the following:

    Warning: Packages marked by Leapp for install not found in repositories 
    metadata: rpcgen python3-pyxattr libnsl2-devel rpcsvc-proto-devel

    These packages are in the Oracle Linux9 Codeready Builder repository, which is a developer repository and is disabled by default.

    If the system requires these packages, then during the preupgrade or the upgrade phase, add the --enablerepo ol9_codeready_builder option to the appropriate Leapp command, for example:

    sudo leapp upgrade --oraclelinux --enablerepo ol9_codeready_builder

    Repositories that have been enabled during the Leapp upgrade remain enabled on the Oracle Linux 9 system after the upgrade completes.

    Alternatively, after completing the upgrade, you can manually install the packages required for your installation by using the dnf command.

    Bug ID 32827043

  • Some el8 packages might not be upgraded

    The same rpm -qa command syntax in the previous item that detects MySQL-related *.el8 packages might also list more *el8 packages on the system that weren't upgraded. Packages might not be upgraded if they were installed from repositories that aren't supported by Leapp, such as developer repositories. For such packages, do the following:

    1. Go to https://yum.oracle.com and check the Oracle Linux 9 repositories that would serve the packages you need.

    2. After the upgrade is completed, manually install the packages from those Oracle Linux 9 repositories.

    3. After all the necessary packages have been installed, remove the residual el8 packages from the system.

    Bug ID 32878386

  • (aarch64) Upgrade log might report errors related to the vmd module

    After completing an upgrade on aarch64 systems, the Leapp upgrade log might report the following message:

    dracut-install: Failed to find module 'vmd'

    The VMD module doesn't apply to the Arm architecture and therefore, the error message can be safely ignored.

    Bug ID 34172552

  • Errors reported that reference legacy directory /var/run

    During the upgrade, messages similar to the following that reference the /var/run directory are reported:

      Installing       :
    leapp-upgrade-el8toel9-0.16.0-6.0.3.el8_6.20220810014405.8fa95c0.noarch      
                                                       6/6
      Running scriptlet:
    leapp-upgrade-el8toel9-0.16.0-6.0.3.el8_6.20220810014405.8fa95c0.noarch      
                                                       6/6
    [/usr/lib/tmpfiles.d/dnssec-trigger.conf:1] Line references path below legacy
    directory /var/run/, updating /var/run/dnssec-trigger → /run/dnssec-trigger;
    please update the tmpfiles.d/ drop-in file accordingly.
    .
    .
    .

    These messages can be ignored. The upgrade or package installation completes successfully.

    As an alternative workaround, you can update the configuration by following the instructions in the message and change the legacy /var/run/* directory path to /run/*.

    Bug ID 34491952

  • Unsigned package generates warning when upgrading from Oracle Linux 8 to Oracle Linux 9

    This issue applies to Oracle Linux 8 systems that were upgraded from Oracle Linux 7 and are later upgraded to Oracle Linux 9.

    When you use Leapp to upgrade an Oracle Linux 7 system to Oracle Linux 8, an unsigned package kernel-workaround-0.1-1.el8.src.rpm is installed as part of the upgrade process. The Leapp upgrade successfully completes, but doesn't notify you of the package.

    If you later run Leapp to upgrade this same system to Oracle Linux 9, this time the upgrade process detects the existence of the package and generates a warning in the /var/log/leapp/leapp-report.txt as follows:

    ----------------------------------------
    Risk Factor: high
    Title: Packages not signed by Oracle found on the system
    Summary: The following packages have not been signed by Oracle and may be
    removed during the upgrade process in case Oracle-signed packages to be
    removed during the upgrade depend on them:
    - gpg-pubkey
    - kernel-workaround
    Key: f5a5d58476a97bf0a8904d00df5d1321189849ad
    ----------------------------------------

    You can disregard the message. The package doesn't prevent the upgrade from completing.

    (Bug ID 35343479)

System Management Issues

  • Ksplice Uptrack software displays error messages

    During the upgrade, the Oracle Ksplice Uptrack software might report errors similar to the following:

    [ 256.033527] upgrade[390]: Upgrading : uptrack-1.2.80-0.el9.noarch 577/1453
    [ 256.037151] upgrade[390]: Running scriptlet: uptrack-1.2.80-0.el9.noarch 577/1453
    ...
    [ 256.045914] upgrade[390]: Traceback (most recent call last):
    [ 256.049230] upgrade[390]: File "/usr/lib/uptrack/access-key-from-uln", line 9, in <module>
    [ 256.051376] upgrade[390]: from up2date_client import up2dateAuth, rpcServer
    [ 256.056490] upgrade[390]: File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 74
    [ 256.059251] upgrade[390]: os.chmod(path, 0600)
    [ 256.060997] upgrade[390]: 
    [ 256.062842] upgrade[390]: SyntaxError: invalid token

    The report is a known but harmless issue, which you can ignore. After the upgrade is completed, Ksplice continues to operate normally.

File Systems and Storage Issues

  • Systems with Btrfs in a RAID configuration can't be upgraded

    A system that uses the Btrfs file system in a RAID configuration can't be upgraded. In the /var/log/leapp/leapp-report.txt that's generated by the preupgrade command, this configuration is flagged as an inhibitor and no remedy is provided. If you upgrade the system and that configuration is detected, the upgrade process halts.

  • Upgrade blocked if winbind and wins Samba modules are used

    If winbind and wins Samba modules are used in the /etc/nsswitch.conf, the upgrade is blocked. As a workaround, remove these modules from the file first, then perform the upgrade. After the upgrade is complete, restore these module entries to the file.

    For more information about configuring these modules, see Oracle Linux 9: Managing Shared File Systems.

  • Hosts with network mounted file systems can't be upgraded

    Leapp doesn't support upgrading systems with mounted file systems on network storage, NFS, or iSCSI. As a workaround, unmount the file systems and comment out their entries from /etc/fstab. After the upgrade is completed, you can restore the entries and remount the file systems.

Networking Issues

  • Possible upgrade error if system has several NICs with the same prefix as NIC that's used by kernel

    The in-place upgrade process might cause an error if the system to be upgraded has more than one NIC that shares the same prefix as the NIC that's used by the kernel, for example eth. After the upgrade, the system's network connectivity is lost.

    For more information, see About Network Interface Names in Oracle Linux 8: Setting Up Networking.

  • NetworkManager might not start after the upgrade completes

    After the upgrade, the system's NetworkManager might not start because of the failure of its name resolution service. The failure can be verified by checking the status of the service.

    systemctl status systemd-resolved.service
    ● systemd-resolved.service - Network Name Resolution 
       Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; 
    disabled; > 
       Active: inactive (dead) 
         Docs: man:systemd-resolved.service(8) 
               https://www.freedesktop.org/wiki/Software/systemd/resolved

    The /var/log/messages file also reports the following error:

    dbus-daemon[742]: [system] Activation via systemd failed for unit 
    'dbus-org.freedesktop.resolve1.service': Unit 
    dbus-org.freedesktop.resolve1.service not found.

    To resolve this issue, choose one of the following workarounds:

    • Configure NetworkManager to not use systemd-resolved.service.

      Add the following entries to the /etc/NetworkManager/conf.d/no-systemd-resolved.conf file:

      [main]
      systemd-resolved=false
    • Enable the systemd-resolved.service as follows:

      systemctl enable systemd-resolved.service
      Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → 
      /usr/lib/systemd/system/systemd-resolved.service. 
      Created symlink 
      /etc/systemd/system/multi-user.target.wants/systemd-resolved.service → 
      /usr/lib/systemd/system/systemd-resolved.service.
      systemctl start systemd-resolved.service

    You can also adopt other methods that are more consistent with the network name resolution model that you're using for the specific setup. For useful information, see About Network Interface Names in Oracle Linux 9: Setting Up Networking.

Virtualization and Containers Issues

  • KVM virtual machine snapshots might not be listed after an upgrade

    After an upgrade, the libvirtd service might report snapshot-related error messages similar to the following:

    libvirtd[53328]: internal error: Failed to parse snapshot XML from file 
    '/var/lib/libvirt/qemu/snapshot/path-to-previous-snapshot-file'

    Furthermore, listing available snapshots from before the upgrade generates an empty list.

    sudo virsh snapshot-list previous-snapshot-file
     Name   Creation Time   State 
    -------------------------------

    As a workaround, reboot the system. At the end of the boot process, the snapshots are listed and available again.

  • libvirtd service might fail to restart in nested virtualization configurations

    In nested virtualization setups, the libvrtd service might not restart in the nested KVM host after the upgrade.

    As a workaround, reboot the nested KVM host.

  • Package installation failures reported when upgrading an Oracle Linux 8 KVM host

    Installation of userspace packages fail when upgrading a KVM host. This error can also be detected during the preupgrade phase. The Leapp utility would report the issue with messages similar to the following example:

    Risk Factor: high
    Title: Unable to install RHEL 9 userspace packages
    Summary:
    ...
    Fatal glibc error: CPU does not support x86-64-v2
    ...

    The error occurs because of a missing cpu mode declaration in the virtual machine's XML file. To work around this issue, follow these steps:

    1. Run the following command:
      sudo virsh edit vm-name
    2. Add the following declaration to the virtual machine's XML file:
      <cpu mode='host-model' check='partial'/>
    3. Rerun the preupgrade process.

    Note:

    This issue is related to the issue KVM virtual machines panic when started on Oracle Linux 9 hosts that's documented in Oracle Linux 9: Release Notes for Oracle Linux 9.

Hardware Related Issues

  • Systems with unrecognized hardware can't be upgraded

    Support for certain hardware, such as the e1000 driver, has been removed from RHCK beginning from RHCK 8. The upgrade can't proceed on platforms that have such hardware installed. Even though UEK might continue to support the hardware, the upgrade procedure is still inhibited if the hardware is detected on the system.

Leapp Overlay Size Issues

  • Upgrading might require increased overlay size

    Upgrading Oracle Linux 8 systems with a huge number of packages to Oracle Linux 9 might fail because of insufficient space in the Leapp overlay file systems that are used during the upgrade. You might see the following error message:

    Error: Transaction test error: 
      installing package package-name needs 4MB on the / filesystem 

    As a workaround, increase the LEAPP_OVL_SIZE variable. The default size is 4096. The actual size you would need might be larger depending on the specific setup. Use the following command:

    sudo export LEAPP_OVL_SIZE=new-size

    Caution:

    The new size that you set for this variable must not exceed 50 percent of the available space on the root partition.

Low Open Files Limit Issues

  • Upgrading might require increased open file limit

    Upgrading Oracle Linux 8 systems with a low open file limit setting to Oracle Linux 9 might fail. You might see the following error messages (or similar messages) during the upgrade:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 185, in run
      File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 199, in _call
      File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/call.py", line 73, in _multiplex
      File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/__init__.py", line 146, in _logfile_logging_handler
      File "/usr/lib64/python3.6/logging/__init__.py", line 1296, in debug
      File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
      File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
      File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
      File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
      File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 40, in emit
      File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 45, in _do_emit
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 87, in store
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 73, in get_connection
      File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 36, in wrapper
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 60, in create_connection
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 27, in _initialize_database
    sqlite3.OperationalError: unable to open database file

    This is caused by an increase in the number of files in the Oracle Linux 9 ca-certificates package. In principle, similar issues can also occur if a package is updated that includes a substantially increased number of packaged files. The error appears when leapp reaches the open files limit during the Oracle Linux in-place upgrade to Oracle Linux 9.

    As a workaround, increase the ulimit open files variable on the Oracle Linux system before starting the upgrade. The value known to cause issues is 1024. For example, the following command shows a system still using the default open files limit:

    ulimit -a | grep open
    open files                      (-n) 1024
    Change the open file limit to 10000. Do the following:
    ulimit -Sn 10000

    Verify that the change has occurred. For example, the following command shows an Oracle Linux system with an open files limit of 10000:

    ulimit -a | grep open
    open files                      (-n) 10000