Troubleshooting Oracle Database Appliance

About Oracle Database Appliance Configuration Collection Utility

Oracle Database Appliance configuration collection collects configuration information for all components in the appliance.

The utility generates a configuration collection with a specific timestamp that can be utilized by other utilities. The collection contains configuration details for all Oracle Database Appliance components in the form of JSON files and a unified HTML configuration report. Creating a snapshot of the appliance configuration assists in debugging issues on the appliance and provides a comprehensive record of all object versions and configuration details at a specific date and time. Additionally, it serves as a valuable reference for tracking configuration changes over time.

The configuration collection tool obtains and stores details about the following components:

DCS framework such as DCS controller, DCS agent, DCS admin, etc CPU pools, Oracle Grid Infrastructure, Oracle Database, operating system, Oracle ILOM, BIOS, Oracle ASR, the firmware, Oracle HMP, and so on.
Operating system, storage, Oracle Clusterware, Oracle KVM, and others.

Use the odaconfigcli collectconfig [-loc] [-redact] [-silent] command to run the configuration collection.

# odaconfigcli collectconfig 

Oracle Database Appliance system config collection:

-- Performing validations:

-- Collecting configuration from the system test1
The config collection is being created at:
/opt/oracle/odaconfig/cnr_data/odaconfig_test1_23_01_2025_19_38_47

1. Collecting system info
Collecting system details [e.g. version, hardwaretype, etc] ....
Collecting storage details ....
Collecting RPM drift info ....
Collecting AFD state ....
Collecting network info ....

2. Collecting Baremetal database info
Collecting system info ....
Collecting component info ....
Collecting dbstorage info ....
Collecting database info ....
Collecting dbhome info ....
Collecting backupconfig info ....
Collecting cpupool info ....
Collecting network info ....
Collecting objectstoreswift info ....
Collecting backupreport info ....

3. Processing Odaconfig report. Please note ** it will take few minutes ** to collect all the command outputs
Collecting ODA stack information
Collecting fixed commands information
Generating html ODAConfig report

4. Setting file protection to the files on config collection.

5. Not redacting the collection

Finished config collection and creating jsons. Following is the collection summary:
Config collect bundle created at: /opt/oracle/odaconfig/cnr_data/odaconfig_testoda803c1n1_23_01_2025_19_38_47
HTML ODAConfig report can be found at: /opt/oracle/odaconfig/cnr_data/odaconfig_testoda803c1n1_23_01_2025_19_38_47/odaconfig_testoda803c1n1_23_01_2025_19_38_47.html
Logs for this run can be found at : /opt/oracle/odaconfig/cnr_data/odaconfig_testoda803c1n1_23_01_2025_19_38_47/logs

When you run the utility from the bare metal system, the output includes the list of DB systems and their configurations, as this information is available in the bare metal DCS metadata. The collected configuration data is provided in the form of JSON files and a unified HTML report. The CLI offers a redaction option for configuration collection, with two modes: sanitize and mask. In the sanitize mode, sensitive entities such as host name and database name are replaced with alternative strings, while in the mask mode, they are replaced with *.

Prerequisites to run the utility

You must run the utility from the bare metal system as the root user.
You can also run the utility on multi-user access enabled systems and multi-user access enabled passwordless systems. Since the utility uses odacli-view, no password prompts are required in these environments. If you want to redact the output in a multi-user access enabled environment, ensure that passwordless login for odacli from the odaadmin user is enabled, eliminating the need for password input.
The oakd service must be running, as the utility uses odaadmcli to gather hardware information.
The system must be provisioned, as there is typically minimal DCS configuration to collect on a new system.

Command and parameters

odaconfigcli collectconfig [-loc] [-redact] [-silent]

Parameter Description

Parameter	Description
`--loc`	(Optional) Customizes results location. The default is `/opt/oracle/odaconfig/cnr_data`.
`--redact`	(Optional) Redacts sensitive entities in configuration collection. Permitted values are `sanitize` and `mask`. `sanitize`: Sensitive entities are replaced with a substitution string `mask`: Sensitive entities are replaced with '*'
`--silent`	(Optional) Suppresses all confirmations.

--loc

(Optional) Customizes results location. The default is /opt/oracle/odaconfig/cnr_data.

--redact

(Optional) Redacts sensitive entities in configuration collection. Permitted values are sanitize and mask.

sanitize: Sensitive entities are replaced with a substitution string

mask: Sensitive entities are replaced with '*'

--silent

(Optional) Suppresses all confirmations.

You can specify custom commands for the tool in a JSON file. For example:

# /opt/oracle/odaconfig/customize/commands_output.json:

     "name" : "DCS Extra",
      "commands":
            [
              {
                "command": "rpm -qa | grep -e dcs -e oda-hw-mgmt -e hami -e mysql",
                 "filename": "dcsextra_rpmqadcs.out"
              },
              {
                "command": "/opt/oracle/dcs/bin/odacli list-featuretracking",
                 "filename": "dcsextra_listfeaturetracking.out"
              },
              {
                "command": "/opt/oracle/dcs/bin/odacli list-dataguardstatus",
                 "filename": "dcsextra_listdataguardstatus.out"
              },
              {
                "command": "/opt/oracle/dcs/bin/odacli list-schedules",
                 "filename": "dcsextra_listschedules.out"
              },
              {
..........

............

    "os":
    {
      "name" : "Operating System",
      "commands":
            [
               {
                 "command": "lscpu",
                 "filename": "os_lscpu.out"
               },
               {
                 "command": "free",
                 "filename": "os_free.out"
               },
               {
                 "command": "top -bn1 | head -n 50",
                 "filename": "os_top50.out"
               },

Related Topics

odaconfigcli collectconfig

Parent topic: Troubleshooting Oracle Database Appliance

About Setting Up and Removing SSH Configuration

Understand how you can set up and remove SSH configuration.

In earlier Oracle Database Appliance releases, the DCS agent used a scheduled SSH cleaner job that ran every 30 minutes, to clean up SSH connections configured in the system. Starting with Oracle Database Appliance release 19.24, the DCS agent does not remove SSH configurations.

The DCS agent has a new agentconfig parameter called ODA_MANAGED_SSH that you can use to deactivate the Setup SSH job on Oracle Database Appliance. You must set up SSH before you run any job that requires SSH configuration between nodes. If you run a job that requires SSH configuration, then the job uses the pre-configured SSH setup. If the SSH keys are present but not functioning or SSH configuration does not exist, the job displays an error. You must set up a working SSH equivalence between nodes before you rerun the job.

Parent topic: Troubleshooting Oracle Database Appliance

Viewing Oracle Database Appliance Error Correlation Reports

Understand how to view Error Correlation Report and how to interpret the report to troubleshoot your appliance.

About Error Correlation Reports

If a DCS job fails, an Error Correlation job is created automatically to generate an Error Correlation report. You can access and review the generated Error Correlation report from the BUI to explore possible ways of error resolution.

The Error Correlation Report contains the following:

Log Messages: Errors, exceptions and warnings from various log files.
Failed Task Messages: Error message displayed when the DCS job failed.
Release Notes: Relevant Known Issues from Oracle Database Appliance Release Notes to help resolve the issue.
Documentation: Relevant topics from the Oracle Database Appliance Documentation Library to help resolve the error.

The Error Correlation Report is generated for every failed DCS job and can be accessed from the BUI. On Oracle Database Appliance high-availability deployments, the Error Correlation report contains the error information derived from log files of both the nodes.

Viewing Error Correlation Reports using ODACLI Commands

You can view the Error Correlation report of a failed DCS job by running the odacli describe-job -i failed_dcs_job_id --ecr command. For an example output, see the topic odacli describe-job in this guide.

Viewing Error Correlation Reports from the BUI

To view the Error Correlation Report from the Activities page in the BUI:

Log into the Browser User Interface:
```
https://host-ip-address:7093/mgmt/index.html
```
Click the Activity tab.
In the Activities page, click the Failure or InternalError link in the failed DCS job for which you want to view the Error Correlation report. Note that only failed DCS jobs have associated Error Correlation Reports.
You can also view the Error Correlation Report for the failed DCS job when you click the Actions menu, and select View Error Correlation Report.
The Error Correlation Report contains the following tabs:
- Log Messages: Displays the logs for DCS agent, DCS admin, Oracle HAMI, MySQL, and Oracle FPP. You can expand each section to view the details. Only components that have logs are displayed. If no errors are found, then the message No errors or exceptions found in logs is displayed in the Log Messages section.
- Failed Task Messages: Displays the specific error message displayed when the task failed.
- Release Notes: Displays relevant Known Issues from Oracle Database Appliance Release Notes to help resolve the issue. You can click each of these links to view the Release Notes entry. If no relevant Known issues are found, then the message No matching results were found. is displayed.
- Documentation: Displays relevant topics from the Oracle Database Appliance Documentation Library to help resolve the error. You can click each of these links to view the documentation topic from the Oracle Database Appliance documentation.

To view the Error Correlation Report from the Diagnostics page in the BUI:

In the BUI, click the Diagnostics tab.
In the Diagnostics page, click Collect Diagnostic Data for a failed job.
The Collect Diagnostics page displays the Error Correlation Report and Job Details in separate tabs for the failed DCS job. Click the Report File Name link to download the Error Correlation Report to your local system.
The Job details tab displays the steps in the job and the Error Correlation Report contains the Log Messages, Failed Task Messages, Release Notes, and Documentation tabs.

odacli describe-job

Parent topic: Troubleshooting Oracle Database Appliance

About Enabling Linux Kernel Core Extractor for Troubleshooting

Understand how to manage Linux Kernel Core Extractor to troubleshoot your appliance.

About Linux Kernel Core Extractor

A Linux kernel panic can occur due to various reasons such as faulty hardware, driver crashes, or software bugs. To identify the cause of kernel panic, it is essential to collect and analyze the vmcore of the crashed kernel. The kdump service is used to collect the vmcore after the first kernel crash. This process is slow for systems with high memory and often fails to generate vmcore when the available space is not sufficient. When Linux Kernel Core Extractor is enabled on Oracle Database Appliance bare metal systems, the crash utility in the kdump kernel collects useful information for troubleshooting without generating vmcore.

Linux Kernel Core Extractor Commands

List generated crash reports:

# /usr/sbin/oled lkce list
Followings are the crash*out found in /var/oled/lkce dir:
/var/oled/lkce/crash_20220307-154542.out

Purge existing all but last three crash reports:

# /usr/sbin/oled lkce clean
lkce deletes all but last three /var/oled/lkce/crash*out files. do you want to proceed(yes/no)? [no]:

Purge all crash reports:

# /usr/sbin/oled lkce clean --all
lkce removes all the files in /var/oled/lkce dir. do you want to proceed(yes/no)? [no]:

By default, the crash report contains output for the following crash commands. You can add other crash commands to the /etc/oled/lkce/crash_cmds_file.

#
# This is the input file for crash utility. You can edit this manually
# Add your own list of crash commands one per line.
#
bt
bt -a
bt -FF
dev
kmem -s
foreach bt
log
mod
mount
net
ps -m
ps -S
runq
quit

By default, vmcore generation is disabled. You can enable vmcore generation as follows:

# oled lkce configure --vmcore=yes
Restarting kdump service... done!
lkce: set vmcore to yes

For additional Linux Kernel Core Extractor commands, refer to the Linux Kernel Core Extractor help.

# oled lkce help
Usage: lkce options
options:
    report report-options -- Generate a report from vmcore
    report-options:
        --vmcore=/path/to/vmcore        - path to vmcore
        [--vmlinux=/path/to/vmlinux]        - path to vmlinux
        [--crash_cmds=cmd1,cmd2,cmd3,..]    - crash commands to include
        [--outfile=/path/to/outfile]        - write output to a file
 
    configure [--default]   -- configure lkce with default values
    configure [--show]  -- show lkce configuration -- default
    configure [config-options]
    config-options:
        [--vmlinux_path=/path/to/vmlinux]   - set vmlinux_path
        [--crash_cmds_file=/path/to/file]   - set crash_cmds_file
        [--kdump_report=yes/no]         - set crash report in kdump kernel
        [--vmcore=yes/no]           - set vmcore generation in kdump kernel
        [--max_out_files=<number>]        - set max_out_files
 
    enable  -- enable lkce in kdump kernel
    disable -- disable lkce in kdump kernel
    status  -- status of lkce
 
    clean [--all]   -- clear crash report files
    list        -- list crash report files

Parent topic: Troubleshooting Oracle Database Appliance

Viewing Details About DCS Error Messages

Understand how to view details about DCS errors for troubleshooting them.

About Viewing Information About DCS Errors

To view more details about any errors during DCS operations, use the command dcserr error_code.

# /opt/oracle/dcs/bin/dcserr
dcserr error_code
 
# dcserr 10001
10001, Internal_Error, "Internal error encountered: {0}."
// *Cause: An internal error occurred.
// *Action: Contact Oracle Support Services for assistance.
/
# dcserr 1001
Unknown error code

To view more details about DCS errors in the Browser User Interface (BUI), you can provide the DCS error code in the Search box in the BUI. The Search results display the Cause and Action of DCS error codes.

Parent topic: Troubleshooting Oracle Database Appliance

Collecting Diagnostics Data Using the BUI

Understand how to collect diagnostics data to troubleshoot errors.

About Collecting Diagnostics Data

Use the Diagnostics tab in the Browser User Interface to view diagnostic information about your deployment and the installed components.

In the Diagnostic Collection page, you can view the available diagnostics collections. Click Collect Diagnostic Data to start diagnostics collection. Once the data is collected, click on the collection file path to download the file.

In the Collect Diagnostics page, specify the Job ID for the diagnostics data collection. Optionally, specify a tag and a description for the collection. The details of the Job ID are displayed. Click Collect to start the diagnostics data collection.

You can also collect diagnostics from the Activity page, by selecting Collect Diagnostics from the Actions drop down for a specific job. Click Collect to start the diagnostics data collection.

To delete a diagnostic collection, from the Diagnostic Collection page, select the specific collection, and click Delete.

This diagnostic collection feature does not replace the odaadmcli manage diagcollect command. You can use the odaadmcli manage diagcollect command also to enable diagnostics collections, independently of this new feature from the BUI. The odaadmcli manage diagcollect command and the diagnostics collection from BUI use the tfactl command internally. The diagnostics collection from BUI is aimed to collect other data from DCS metadata that is not collected through tfactl and provide greater context for root cause analysis of related DCS jobs failures.

Parent topic: Troubleshooting Oracle Database Appliance

Resolving Errors When Updating DCS Components During Patching

Understand how to troubleshoot errors when updating DCS components during patching.

.

About DCS Components

When you run the odacli update-dcscomponents command during patching, pre-checks for MySQL installation are automatically verified before update of Oracle HAMI, MySQL, and DCS components. If any of the pre-checks fail, then the command errors out with a reference to the pre-check report log file location /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log. Review the pre-check report and take corrective actions and then rerun the odacli update-dcscomponents command. If there are no pre-check errors, then the patching process proceeds with updating Oracle HAMI, MySQL, and DCS components such as the DCS Agent, DCS CLI, and DCS Controller.

Note:

Run the

odacli
                        update-dcsadmin

command prior to running the

odacli
                        update-dcscomponents

command.

When the odacli update-dcscomponents command completes successfully:

The command output is as follows:

# ./odacli update-dcscomponents -v 19.26.0.0.0            
{
  "jobId" : "3ac3667a-fa22-40b6-a832-504a56aa3fdc",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent
shutdown is successful. MySQL upgrade is done before. Metadata migration is
successful. Agent rpm upgrade is successful. DCS-CLI rpm upgrade is successful.
DCS-Controller rpm upgrade is succ",
  "reports" : null,
  "createTimestamp" : "December 8, 2024 02:37:37 AM CST",
  "description" : "Update-dcscomponents job completed and is not part of Agent
job list",
  "updatedTime" : "December 8, 2024 02:39:10 AM CST"
}

The pre-check report log file at the location /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

dcs-admin version: 
Pre-check Name: dcs-admin version validation
Status: Success
Comments: dcs-admin is already updated :19.26.0.0.0

Config File Exist dcscontroller: 
Pre-check Name: Check DCS config files exists for dcscontroller
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-controller.yml and /opt/oracle/dcs/conf/dcs-controller-logback.xml exist
 
Config File Exist dcsagent: 
Pre-check Name: Check DCS config files exists for dcsagent
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-agent.yml and /opt/oracle/dcs/conf/dcs-agent-logback.xml exist

Validate scheduler cron expressions:
Pre-check Name: Validate scheduler cron expressions
Status: Success
Comments: Scheduler cron expressions for existing job schedules are up to date

When the odacli update-dcscomponents command fails:

On Oracle Database Appliance single-node systems, the command output is as follows:

# ./odacli update-dcscomponents -v 19.26.0.0.0            

DCS-10008:Failed to update DCScomponents: 19.25.0.0.0
Internal error while patching the DCS components :
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer to
/opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log
on node 0 for details.

On Oracle Database Appliance high-availability systems, the command output is as follows:

# ./odacli update-dcscomponents -v 19.26.0.0.0            

Internal error while patching the DCS components :
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer to
/opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log
on node 0 and /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log
on node 1 for details.

The command runs all pre-checks one by one, and errors out at the end if any of the pre-checks is marked as Failed. When a pre-check fails, the error message is displayed on to the console along with the reference to pre-check report log location. The pre-check report log file is at the location /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log.

Pre-check Name: Space check
Status: Failed
Comments: Available space in /opt is 2 GB but minimum required space in /opt is 3 GB 

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

Pre-check Name: dcs-admin version validation
Status: Success
Comments: dcs-admin is already updated :19.20.0.0.0

Config File Exist dcscontroller: 
Pre-check Name: Check DCS config files exists for dcscontroller
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-controller.yml and /opt/oracle/dcs/conf/dcs-controller-logback.xml exist
 
Config File Exist dcsagent: 
Pre-check Name: Check DCS config files exists for dcsagent
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-agent.yml and /opt/oracle/dcs/conf/dcs-agent-logback.xml exist

Validate scheduler cron expressions:
Pre-check Name: Validate scheduler cron expressions
Status: Success
Comments: Scheduler cron expressions for existing job schedules are up to date

When the odacli update-dcscomponents command fails due to space check error: