8.11.1 Monitoring and Troubleshooting Exadata Database Server Updates

You can use the log files generated by the update utility to troubleshoot updates.

The update utility orchestrates updating the Exadata database servers. Updating database nodes with the patchmgr tool is less verbose because it prints only minimal information to the screen. If additional information is required, you can view the patchmgr logs and the dbnodeupdate.sh logs that patchmgr copies over from other servers, if available. The log file (dbnodeupdate.log) and the diag file (dbnodeupdate.<runid>.diag) will eventually exist on two locations:

  • On each updated database server, in the /var/log/cellos directory
  • Consolidated on the node running the update utility.

On the node running the update utility, if the -log_dir flag was set to auto, the log files will be stored in the log/<directory based on contents of nodes in list file> directory, relative from the directory where the update utility is started from. For example, if the update utility is located in /u01/dbserver.patch, then the log directory may be /u01/dbserver.patch/dm01db01_dm01db02_e8f1f753.

Important files found in the log directory are:

  • patchmgr.log contains the consolidated screen output from running the remote update commands on the different database servers.
  • <hostname>_dbnodeupdate.<runid>.diag is the diag file for the specific run on a database server.
  • <hostname>_dbnodeupdate.log contains dbnodeupdate.log output appended from /var/log/cellos from the remote database server.

When a prerequisite check, backup, update, or rollback fails, error messages on screen should provide information on which step failed on which node. Consult the log files mentioned above if more information is required. Search the log file for the start of a new run (search for zzz).

Check if the time matches your run. If it matches, note the runid for further reference. Then search for ERROR.

If an update action fails before the actual YUM update, you can retry the update after resolving the error. If the update failed half-way, it is recommended that you roll back, resolve the error, and retry.

In rare cases, patchmgr may be unable to determine the status of an update, whether the update was successful or not. In such cases, it displays a message that the update failed. However, it is possible that the update still completed successfully. To determine the actual status of the update:

  • Check the image status of the (database) node. You can do this by running the imageinfo command. The Image status line displays the status.
  • Check the version of the Exadata software. This can also be determined from the imageinfo command.

If the image status is success, and the Exadata version is the new expected version, then the update was successful and you can ignore the update failed message. Then:

  • Run dbnodeupdate.sh -c manually on the particular node to perform the completion steps of the update.
  • Remove the completed node from the (database) node file.
  • Rerun patchmgr to perform the update on the remaining nodes.

Other things to check if the update fails include:

  • The correct syntax for using patchmgr to update database nodes can be found in the patchmgr online help.
  • SSH equivalence must be configured before using patchmgr.
  • Download the latest dbserver.patch.zip from My Oracle Support document KB444935.
  • Open a service request with Oracle Support Services to analyze why the patchmgr orchestration failed.

After patchmgr completes, you can also check image status and history using the imageinfo and imagehistory commands on each database server. The following example contains typical imageinfo output showing the current image status on an Exadata database server.

# imageinfo

Kernel version: 5.15.0-308.179.6.16.el8uek.x86_64 #2 SMP Thu Sep 18 11:19:34 PDT 2025 x86_64
Uptrack kernel version: 5.15.0-314.193.5.3.el8uek.x86_64 #2 SMP Mon Nov 10 00:36:36 PST 2025 x86_64
Image kernel version: 5.15.0-308.179.6.16.el8uek
Image version: 25.2.5.0.0.251214
Image activated: 2026-01-15 15:00:42 -0800
Image status: success
Exadata software version: 25.2.5.0.0.251214
Node type: COMPUTE
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

The following imagehistory output shows that the database server was updated to Oracle Exadata System Software release 25.2.5 (from release 25.1.6).

# imagehistory

Version                              : 25.1.6.0.0.250622
Exadata Live Update Version          : n/a
Image activation date                : 2025-07-04 00:59:39 -0700
Imaging mode                         : fresh
Imaging status                       : success

Version                              : 25.2.5.0.0.251214
Exadata Live Update Version          : n/a
Image activation date                : 2026-01-15 15:00:42 -0800
Imaging mode                         : patch
Imaging status                       : success

The next example shows imageinfo output for an update to a guest VM using Exadata Live Update. In this example, the update has been applied but there is outstanding work that is yet to be applied.

# imageinfo

Kernel version: 5.15.0-300.163.18.7.el8uek.x86_64 #2 SMP Fri Nov 15 03:14:11 PST 2024 x86_64
Uptrack kernel version: 5.15.0-312.187.5.3.el8uek.x86_64 #2 SMP Sun Sep 21 08:53:14 PDT 2025 x86_64
Image kernel version: 5.15.0-300.163.18.7.el8uek
Image version: 25.1.1.0.0.250121
Image created: 2025-01-21 23:49:50 -0800
Image activated: 2025-10-28 11:42:36 -0700
Image image type: production
Image status: success
Image label: OSS_25.1.1.0.0_LINUX.X64_250121
Exadata software version: 25.1.1.0.0.250121
Exadata Live Update Type: high (CVSS 7-10)
Exadata Live Update Version: 25.1.10.0.0.251012.1 (Live Update applied. Reboot at any time to finalize outstanding items.)
Node type: GUEST
Install type: KVM Guest with ROCE and Secure Fabric
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

The following example shows imagehistory output after an update using Exadata Live Update. In this example, the output shows three image history records:

  1. The first record contains information about the initial (fresh) image.

  2. The second record contains information about the image after a full update using Exadata Live Update. At this stage, the update has been applied but there is outstanding work that is yet to be applied.

  3. The last record show the image state after application of the outstanding work associated with the Exadata Live Update.

# imagehistory

Version                              : 25.2.2.0.0.250919
Exadata Live Update Version          : n/a
Image activation date                : 2025-10-06 18:05:41 -0700
Imaging mode                         : fresh
Imaging status                       : success
 
Version                              : 25.2.3.0.0.251020
Exadata Live Update Version          : 25.2.3.0.0.251020 (full)  (Live Update applied. Reboot at any time to finalize outstanding items.)
Image activation date                : 2025-11-04 20:18:40 -0800
Imaging mode                         : patch
Imaging status                       : success
 
Version                              : 25.2.3.0.0.251020
Exadata Live Update Version          : 25.2.3.0.0.251020 (full)
Image activation date                : 2025-11-04 22:43:30 -0800
Imaging mode                         : patch
Imaging status                       : success