This topic outlines the purpose and usage of Cluster Health Monitor (CHM) and System Health Monitor (SHM).

crsctl stat res ora.crf -init -d

ahfctl statusahf

System Health Monitor (SHM) is integrated and enabled by default in AHF. AHF now includes the SHM files in its diagnostic collection.

System Health Monitor (SHM) monitors operating system metrics in real time for processes, memory, network, IO and disk to troubleshoot and root cause the system performance issues in real time as well as root cause analysis of past issues. System Health Monitor (SHM) analysis will be available in AHF Insights. For more information, see Explore Diagnostic Insights.

SHM operates as a daemon process, triggered and controlled by AHF, enabled by default, but it is only available on Single-Instance Database and non-GI based systems.

Also, you can use the ahfctl statusahf command to check the status of System Health Monitor.

• To start SHM:
  Run this command only if SHM has been stopped previously for some reason and needs to be switched back on.

  ahf configuration set --property ahf.collectors.enhanced_os_metrics --value on

  Running the command enables ahf-sysmon to be started and the TFA daemon will then start and monitor it.

• To stop SHM:

  ahf configuration set --property ahf.collectors.enhanced_os_metrics --value off

  Running the commands checks if ahf-sysmon is up or not. If it's running, the command will kill the process and stops ahf-sysmon.

• To verify the default value of SHM:

  ahf configuration get --property ahf.collectors.enhanced_os_metrics
  ahf.collectors.enhanced_os_metrics: on

• To verify the SHM process (ahf-sysmon) is active by default:

  ps -fe | grep sysmon
  root     3333453  1  0 22:44 ?   00:00:00 /opt/oracle.ahf/shm/ahf-sysmon/bin/ahf-sysmon

• To check SHM JSON files:

  Locate the JSON files in the SHM data directory /opt/oracle.ahf/data/<hostname>/shm

• To check if SHM runs under the same cgroup of TFAMain or not:

  -bash-4.4$ ps -ef | grep ahf-sysmon
  root        3232       1  0 09:38 ?        00:00:47 /opt/oracle.ahf/shm/ahf-sysmon/bin/ahf-sysmon
  testuser   155833  155678  0 17:04 pts/0    00:00:00 grep --color=auto ahf-sysmon
  -bash-4.4$ cat /proc/3232/cgroup | grep "cpu"
  8:cpu,cpuacct:/oratfagroup
  4:cpuset:/

  -bash-4.4$ ps -ef | grep tfa
  root        1945       1  0 09:37 ?        00:00:02 /bin/sh /etc/init.d/init.tfa run >/dev/null 2>&1 </dev/null
  root        2851       1  1 09:37 ?        00:05:21 /opt/oracle.ahf/jre/bin/java --add-opens java.base/java.lang=ALL-UNNAMED -server -Xms128m -Xmx256m -Djava.awt.headless=true -Ddisable.checkForUpdate=true -XX:+ExitOnOutOfMemoryError oracle.rat.tfa.TFAMain /opt/oracle.ahf/tfa
  testuser   156073  155678  0 17:05 pts/0    00:00:00 grep --color=auto tfa

  -bash-4.4$ cat /proc/2851/cgroup | grep "cpu"
  8:cpu,cpuacct:/oratfagroup
  4:cpuset:/

  -bash-4.4$ cat /proc/3232/cgroup | grep "cpu"
  8:cpu,cpuacct:/oratfagroup
  4:cpuset:/

  cat /proc/[PID_OF_AHF-SYSMON]/cgroup | grep "cpu"

  cat /proc/[PID_OF_TFA]/cgroup | grep "cpu"

• To verify SHM files are collected in AHF collection:
  • As prerequisite, run:

    tfactl set smartprobclassifier=off

  • Then run:

    tfactl diagcollect -last 1h -tag shm_last_1h; 
    unzip -l $REPOSITORY_ROOT/shm_last_1h/$HOSTNAME*.zip

    A directory called SHM should be present in generated zip file.

  • And finally run:

    tfactl diagcollect -last 1h
    Archive:  /opt/oracle.ahf/data/repository/collection_Wed_Apr_10_22_03_04_UTC_2024_node_all/test-node.tfa_Wed_Apr_10_22_03_03_UTC_2024.zip | grep SHM
      Length      Date    Time    Name
    ---------  ---------- -----   ----
          327  04-10-2024 22:03   test-node/SHMDATA/shmdataconverter_3279258.log
         6660  04-10-2024 22:03   test-node/SHMDATA/shmosmeta_1923000.json
        43575  04-10-2024 22:03   test-node/SHMDATA/shmosmetricdescription.json
      9561411  04-10-2024 22:03   test-node/SHMDATA/shmosdata_test-node_2024-04-10-2100.log
       997193  04-10-2024 22:03   test-node/SHMDATA/shmosdata_test-node_2024-04-10-2200.log

Collect Cluster Health Monitor data from any node in the cluster.

Review the metrics collected by CHM and SHM.

Improved ability to detect component failures and self-heal autonomously improves business continuity.

Cluster Health Monitor introduces a new diagnostic feature that identifies critical component events that indicate pending or actual failures and provides recommendations for corrective action. These actions may sometimes be performed autonomously. Such events and actions are then captured and admins are notified through components such as Oracle Trace File Analyzer.

Inline Analysis is a new feature that will automatically execute on all systems where Autonomous Health Framework (AHF) is installed.

Traditionally, the operating system (OS) metrics collected by Cluster Health Monitor (CHM) or System Health Monitor (SHM) are stored directly in the Grid Infrastructure (GI) Base repository. However, this repository is limited in size—250 MB on HAS 19 and 500 MB on MAIN.

On large FASAAS systems, which may run between 30,000 and 40,000 processes, these size limitations mean that CHM OS data can only be retained for a few hours. As a result, the necessary OS data is frequently missing in most Service Requests (SRs), making it challenging to diagnose and resolve issues effectively.

With the CHM Inline Analysis enhancement, instead of preserving large volumes of raw CHM OS data, the system analyzes these OS metrics and stores only the summary analyzed data. The processed analysis files are significantly smaller than the raw data, allowing for several months of data retention within a 100 MB repository quota.

How it works

• Collection and Analysis:
  • OS metrics are collected by CHM/SHM as usual.
  • Every hour, Oracle Trace File Analyzer (TFA) triggers Inline Analysis which processes the hourly raw chmosdata/shmosdata gzip files.
• Storage: Only the analyzed results are kept in the AHF home data repository, dramatically reducing space consumption.
• Execution and Security: Inline Analysis runs as the root user and is configured with specific resource constraints to avoid impact on system performance.

Benefits

• Extended Retention: Analyzed data can be retained for several months rather than just a few hours.
• Reduced Storage Footprint: Only 100 MB of repository space is required, even for large systems.
• Improved Diagnostics: OS data will be available in more SRs, enabling more effective and timely troubleshooting.

System Health Monitor (SHM) is a tool that collects essential operating system metrics for Oracle Support, especially valuable in diagnosing initial failures such as node evictions when Service Requests (SRs) are logged.

With this enhancement, Autonomous Health Framework (AHF) will integrate SHM data into its diagnostic collections for non-root, standalone environments, that is, Oracle Restart or Single Instance. This enables non-root users to benefit from OS-level insight reports generated from SHM data, particularly focusing on OS-specific diagnostics.

For the initial release (25.6), end users are responsible for managing the SHM lifecycle in non-root standalone installations. After SHM data is collected, the host system will be able to generate an insight report that concentrates primarily on the gathered OS metrics.

Manage the SHM background process (daemon) with the following commands:

• sysmonctl start: Start the System Health Monitor (SHM)
• sysmonctl stop: Stop the System Health Monitor (SHM)

Create Insights reports from SHM data to proactively identify and investigate operating system-related issues in a non-root AHF environment.

1. Navigate to the CHM binaries directory.

   cd $AHF_HOME/chm/bin/

2. Run the report generation command.

   $AHF_HOME/python/bin/python3 chm_tfa_driver.zip -o <OUTPUT_DIRECTORY> -s <START_TIME> -e <END_TIME>

   Example Run:

   -bash-4.4$ $AHF_HOME/python/bin/python3 chm_tfa_driver.zip -o /tmp/shm_out/ -s 2025-08-22T12:00:00 -e 2025-08-22T12:30:00
   Successfully ran the chm utils on the SHM data.
   Successfully ran the chm analyzer on the SHM data.
   Successfully ran the chm reportgen on the SHM analysis data.
   executing
   Report is generated at : /tmp/shm_out/test-node_insights_2025_08_22_12_32_44.zip
   Report jsons is generated at : /tmp/shm_out/test-node_insights_jsons_2025_08_22_12_32_44.zip
   Successfully ran the ahf insights driver on the SHM data.

3. Verify SHM Data.

   SHM data files can be checked at: $AHF_HOME/data/<HOSTNAME>/shm

   You can view the AHF Insights report by downloading and unzipping the file, then opening web/index.html in your browser.