12 Monitoring and Maintaining Offline Mediation Controller Cloud Native

Learn how to monitor and maintain your Oracle Communications Offline Mediation Controller cloud native deployment.

Using Prometheus Operator to Monitor Offline Mediation Controller Cloud Native

Offline Mediation Controller cloud native tracks and exposes the following metric data in Prometheus format:

  • Node Manager-level statistics, which include:

    • The total network account records (NARs) processed

    • The current NARs processed

    • The current processing rate

    • The average processing rate

    Node Manager sets expose metrics through a configurable service port. In your override-values.yaml file, each set includes a metrics section where the port for metrics can be specified using the metrics.service.port key. For example:
    sets:
        nm-cc
            name: "nm-cc"
            replicas: 1
            metrics:
                service:
                    port: 31300
  • JVM metrics for all Offline Mediation Controller components, which include:

    • Performance on the Node Manager level

    • JVM parameters

    JVM metrics are exposed through the endpoint http://hostname:JVMport/metrics, where JVMport is the port number where the JVM metrics are exposed. They are exposed on the same port as the Node Manager metrics for each set, which is configured using metrics.service.port in your override-values.yaml file.

  • REST Services Manager metrics, which include:

    • JVM metrics
    • Service status
    • Total uptime
    • Garbage collection events
    • Memory usage statistics
    • Thread count
    • Class loader statistics

    REST Services Manager metrics are exposed through the Helidon framework's http://RSM_hostname:RSM_port/metrics endpoint, where RSM_hostname is the host name of the machine on which REST Services Manager is installed. The endpoint exposes metrics information in both JSON format (according to the MicroProfile Metrics specification) and plain text format suitable for Prometheus.

To monitor Offline Mediation Controller more easily, you can configure an external centralized metrics service, such as Prometheus Operator, to scrape metrics from each endpoint and store them for analysis and monitoring. You can then set up a visualization tool, such as Grafana, to display your metric data in a graphical format.

For the list of compatible Prometheus Operator and Grafana software versions, see "Offline Mediation Controller Cloud Native Deployment Software Compatibility" in Offline Mediation Controller Compatibility Matrix.

Enabling the Automatic Scraping of Metrics

You can configure the Prometheus Operator ServiceMonitor to automatically scrape Offline Mediation Controller metrics and Offline Mediation Controller REST Services Manager metrics. For more information about Prometheus Operator and ServiceMonitors, see the prometheus-operator documentation on the GitHub website (https://github.com/prometheus-operator/prometheus-operator/tree/main/Documentation/getting-started).

To enable the automatic scraping of Offline Mediation Controller metrics:

  1. Install Prometheus Operator on your cloud native environment.

  2. In your override-values.yaml file for oc-cn-ocomc, set the following keys:

    • ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.enabled: Set this key to true.

    • ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.interval: Set this to the interval at which to scrape metrics. The default is 10s.

    • ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.labels.app: Set this to the app label you want applied to the ServiceMonitor resource. This label is used by Prometheus to discover the ServiceMonitor via label selectors.

    • ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.labels.release: Set this to the release label you want applied to the ServiceMonitor resource. This label is used by Prometheus to discover the ServiceMonitor via label selectors.

    • ocomcCore.ocomc.nodeManagerConfigurations.metrics.service.type: Set this to the service type, such as NodePort or ClusterIP. The default is ClusterIP.

    • (For REST Services Manager) ocomcRSM.rsm.serviceMonitor.enabled: Set this key to true.

  3. Run the helm upgrade command to update the Offline Mediation Controller Helm release:

    helm upgrade ReleaseName oc-cn-ocomc --values OverridingValuesFile -n NameSpace 

    where:

    • ReleaseName is the release name, which is used to track the installation instance.

    • OverrideValuesFile is the path to a YAML file that overrides the default configurations in the chart's values.yaml file.

    • NameSpace is the namespace in which to create Offline Mediation Controller Kubernetes objects.

Using the Sample Grafana Dashboards

The Offline Mediation Controller package includes sample Grafana Dashboard templates that you can use for visualizing metrics. To use the sample dashboards, import the following JSON files from the OMC_home/sampleData/dashboards directory into Grafana:

  • OCOMC_JVM_Dashboard.json: This dashboard lets you view JVM-related metrics for Offline Mediation Controller.

  • OCOMC_Node_Manager_Summary.json: This dashboard lets you view NAR processing metrics for the Node Manager.

  • OCOMC_Node_Summary.json: This dashboard lets you view NAR processing metrics for all nodes.

  • OCOMC_Summary_Dashboard.json: This dashboard lets you view NAR-related metrics for all Offline Mediation Controller components.

  • OCOMC_RSM_JVM_Dashboard.json: This dashboard lets you view JVM-related metrics for Offline Mediation Controller RSM.

For information about importing dashboards, see "Manage Dashboards" in the Grafana Dashboards documentation.

Offline Mediation Controller REST Services Manager Metrics

You can use various metrics to monitor the performance and health of Offline Mediation Controller REST Services Manager.

Table 12-1 describes the metrics mapped to their variable names in the /metrics endpoint response.

Table 12-1 Metrics Mapped to Variable Names

Variable Name Metric Description
up The status of the service, indicating whether it is up and running.
base_jvm_uptime_seconds The total uptime of the JVM in seconds since it was started.
base_gc_total The total number of garbage collection events that have occurred using the copy and mark/sweep algorithms.
base_gc_time_seconds The total time spent in garbage collection using the copy and mark/sweep algorithms., measured in seconds.
base_memory_usedHeap_bytes The amount of memory currently used in the heap, measured in bytes.
base_memory_commitedHeap_bytes The amount of memory that has been committed for use in the heap, measured in bytes.
base_memory_maxHeap_bytes The maximum amount of memory that can be allocated for the heap, measured in bytes.
base_thread_count The current number of threads that are actively running in the JVM.
base_classloader_loadedClasses_count The number of classes currently loaded into memory by the class loader.
base_classloader_loadedClasses_total The total number of classes that have been loaded into memory since the JVM started.
base_classloader_unloadedClasses_total The total number of classes that have been unloaded from memory since the JVM started.

Automating Workflows Using RSM Request Automation

The RSM Request Automation feature enables response-driven execution of workflows within the Offline Mediation Controller cloud native environment. It is implemented as a Kubernetes job that dynamically executes workflows based on a structured JSON payload file. This file defines the sequence of API requests to run, response handling, and any conditional logic that determines the execution flow. This feature also allows users to execute any API exposed by the REST Services Manager.

With this feature, you can:

  • Automatically define and execute a sequence of steps based on predefined rules and conditions.

  • Use data from API responses to set environment variables for subsequent requests.

  • Leverage predefined flows for error handling and retries.

  • Support multipart file uploads as part of API requests.

  • Apply conditions based on response data to control execution the process flow.

  • Chain requests based on the output of previous ones.

Setting Up REST Services Manager Request Automation

To set up the REST Services Manager Request Automation feature in your Offline Mediation Controller cloud native deployment:

  1. In your override-values.yaml file for oc-cn-ocomc-rsm, set the rsm.jobs.requestAutomation.enabled key to true.

  2. Configure the following keys under rsm.jobs.requestAutomation.config:

    • apiUrl: Set this to the URL of your Offline Mediation Controller REST Services Manager service, such as http://ocomc-rsm:8080.

    • payloadFile: Specify the absolute path to your workflow payload JSON file.

    • loggingDIr: Define the directory where log files should be written.

    • (Optional) rsmOAuthToken: Set this to the REST Services Manager OAuth 2.0 token.

    The following shows an example configuration:

    jobs:
      requestAutomation:
        enabled: true
        config:
          apiUrl: "http://ocomc-rsm:8080"
          payloadFile: "/app/config/workflow-payload.json"
          loggingDir: "/app/volumes/ocomc-ext/logs
  3. Create a JSON workflow payload file following the workflow file syntax. See "Creating a Workflow Payload File" for details.

  4. Move the JSON workflow payload file to the directory specified in the payloadFile key.

  5. Run a helm install or helm upgrade command to deploy your changes.

Creating a Workflow Payload File

The workflow payload file defines your automation workflow in JSON format.

The following is an example of the workflow file structure:

{
  "workflow": {
    "flow": [
      {
        "id": "UPLOAD_IMPORT_FILE_ACTION",
        "request": "UPLOAD_IMPORT_FILE",
        "onResponse": {
          "environment": {
            "IMPORT_ID": "/importId"
          }
        }
      },
      {
        "id": "START_IMPORT_ACTION",
        "request": "START_IMPORT",
        "onResponse": {
          "statusMapping": {
            "2**": ["CHECK_IMPORT_COMPLETION_ACTION"]
          }
        }
      }
    ]
  },
  "headers": {},
  "environment": {},
  "requests": [
    {
      "id": "UPLOAD_IMPORT_FILE",
      "method": "POST",
      "uri": "/imports/uploadImportFile",
      "headers": {},
      "multipartForm": {
        "configurationFile": "FILE::/path/to/export.xml"
      }
    },
    {
      "id": "START_IMPORT",
      "method": "POST",
      "uri": "/imports/{{ENV:IMPORT_ID}}/tasks",
      "headers": {},
      "payload": {}
    }
  ]
}

The following is a breakdown of the syntax used within the workflow file, covering the key elements and options.

Table 12-2 describes the structure of the workflow section, which defines the sequence of actions to be executed.

Table 12-2 Workflow Section

Element Description
flow

An array defining the sequence of actions to run.

id

A unique identifier for the action within the workflow.

request

The ID of the API request to run (defined in the requests section).

onResponse

How to process the API response and subsequent actions.

Table 12-3 describes the elements used in the OnResponse section to handle API responses and determine subsequent actions.

Table 12-3 OnResponse Section

Element Description

environment

Specifies the environment variables to set based on the response. The value from /importId in the response is assigned to the environment variable IMPORT_ID. These variables can later be accessed in requests using the {{ENV:VARIABLE_NAME}} syntax.

"onResponse": {
  "environment": {
    "IMPORT_ID": "/importId"
  }
}

statusMapping

Maps HTTP status codes to specific actions, such as ABORT. In this case, if a 5xx error is encountered, the transaction is halted and terminated.

"statusMapping": {
  "5**": ["ABORT"]
}

retry

Configures automatic retries for a request.
  • maxRetries: The maximum number of retry attempts (integer).
  • interval: The time interval (in milliseconds) between retries.
"retry": {
  "maxRetries": "5",
  "interval": "10000"
}

maxRetrics

Specifies the maximum number of retry attempts.

dataMapping

Specifies conditions based on values in the API response.

  • path: A specific field in the JSON response, such as /importTaskItem/taskFinished.
  • equals: The value to compare against (string).
  • then: An array of actions to take when the condition is true, such as retry the request if the task is not finished.
"dataMapping": [
  {
    "path": "/importTaskItem/taskFinished",
    "equals": "true",
    "then": []
  },
  {
    "path": "/importTaskItem/taskFinished",
    "equals": "false",
    "then": ["RETRY"]
  }
]

Table 12-4 describes the elements required to define individual API requests that can be referenced within a workflow.

Table 12-4 Requests Section

Element Description

id

A unique identifier for the request.

method

The HTTP method, such as GET, POST, PUT, or DELETE.

uri

The URI or endpoint for the API request. The {{ENV:VARIABLE_NAME}} syntax injects values from environment variables.

headers

Any required HTTP headers.

payload

The data to be sent in the request body (for methods like POST).

multipartForm

Defines files to be uploaded as part of the request. The FILE:: prefix indicates sending a file from a local path.

Managing a Helm Release

After you install a Helm chart, Kubernetes manages all of its objects and deployments. All pods created through oc-cn-ocomc are wrapped in a Kubernetes controller, which creates and manages the pods and performs health checks. For example, if a node fails, a controller can automatically replace a pod by scheduling an identical replacement on a different node.

Administrators can perform these maintenance tasks on a Helm chart release:

Tracking a Release's Status

When you install a Helm chart, it creates a release. A release contains Kubernetes objects, such as ConfigMap, Secret, Deployment, Pod, PersistentVolume, and so on. Not every object is up and running immediately. Some objects have a start delay, but the Helm install command completes immediately.

To track the status of a release and its Kubernetes objects, enter this command:

helm status ReleaseName -n Namespace

where:

  • ReleaseName is the name you assigned to this installation instance.

  • NameSpace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.

Updating a Release

To update any key value after a release has been created, enter this command. This command updates or re-creates the impacted Kubernetes objects, without impacting other objects in the release. It also creates a new revision of the release.

helm upgrade ReleaseName oc-cn-ocomc-helm-chart --values OverridingValueFile --values NewOverridingValueFile -n Namespace 

where:

  • ReleaseName is the name you assigned to this installation instance.

  • OverridingValueFile is the path to the YAML file that overrides the default configurations in the oc-cn-ocomc/values.yaml file.

  • NewOverridingValueFile is the path to the YAML file that has updated values. The values in this file are newer than those defined in values.yaml and OverridingValueFile.

  • Namespace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.

Checking a Release's Revision

Helm keeps track of the revisions you make to a release. To check the revision for a particular release, enter this command:

helm history ReleaseName -n Namespace

where:

  • ReleaseName is the name you assigned to this installation instance.

  • Namespace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.

Rolling Back a Release to a Previous Revision

To roll back a release to any previous revision, enter this command:

helm rollback ReleaseName RevisionNumber -n Namespace

where:

  • ReleaseName is the name you assigned to this installation instance.

  • RevisionNumber is the value from the Helm history command.

  • Namespace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.

Rolling Back an Offline Mediation Controller Cloud Native Upgrade

If you encounter errors after upgrading, you can roll back to a previous version of Offline Mediation Controller.

The following procedure assumes that you have upgraded Offline Mediation Controller from 12.0 Patch Set 5 (Revision 1), to 12.0 Patch Set 6 (Revision 2), and then to 15.1 (Revision 3). To roll back your upgrade from 15.1 to 12.0 Patch Set 6, you would do this:

  1. Check the revision history of the Offline Mediation Controller release:

    helm history ReleaseName -n Namespace

    You should see something similar to this:

    REVISION   UPDATED                    STATUS        CHART APP                 VERSION        DESCRIPTION 
    1          Thu May 30 07:12:46 2030   superseded    oc-cn-ocomc-helm-chart    12.0.0.5.0     Initial install
    2          Thu May 30 08:32:09 2030   superseded    oc-cn-ocomc-helm-chart    12.0.0.6.0     Upgraded successfully
    3          Thu May 30 09:50:00 2030   deployed      oc-cn-ocomc-helm-chart    15.1.0.0.0     Upgraded successfully
  2. Roll back the release to Offline Mediation Controller 12.0 Patch Set 6:

    helm rollback ReleaseName 2 -n BrmNamespace

    If successful, you will see this:

    Rollback was a success! Happy Helming!
  3. Check the revision history of the Offline Mediation Controller release:

    helm history ReleaseName -n BrmNamespace

    If successful, you should see something similar to this:

    REVISION   UPDATED                    STATUS        CHART APP                 VERSION        DESCRIPTION 
    1          Thu May 30 07:12:46 2030   superseded    oc-cn-ocomc-helm-chart    12.0.0.5.0     Initial install
    2          Thu May 30 08:32:09 2030   superseded    oc-cn-ocomc-helm-chart    12.0.0.6.0     Upgraded successfully
    3          Thu May 30 09:50:00 2030   superseded    oc-cn-ocomc-helm-chart    15.1.0.0.0     Upgraded successfully
    4          Thu May 30 11:25:00 2030   deployed      oc-cn-ocomc-helm-chart    12.0.0.6.0     Roll back to 2

Integrating Oracle Unified Directory with Offline Mediation Controller Cloud Native

After verifying the Oracle Unified Directory deployment, follow these steps to integrate it with Offline Mediation Controller cloud native:
  1. Open a terminal session within the running pod:

    kubectl exec -it oud-ds-rs-0 -n oudns -- /bin/bash
  2. Inside the Oracle Unified Directory container, create a temporary directory (tempdir) to hold configuration files.

  3. From the Offline Mediation Controller installation directory, copy the oudConfig and populateDir.ldif files to tempdir.

  4. Inside tempdir, create a file named populateDirTemp.ldif. This file updates the user information in Oracle Unified Directory to match the Offline Mediation Controller requirements. Add the following content:

    dn: uid=Admin,ou=People,dc=ocomcexample.com
    changetype: modify
    replace: userpassword
    userpassword: adminpassword
  5. Run the tempdir/oudConfig script with the Oracle Unified Directory container:

    sh oudConfig -i oud_instance_path -h oud_host -p oud_admin_port -d "oud_binddn" -w oud_admin_password -b "dc=ocomcexample.com" -l oud_ldapport

    Note:

    If the command fails with a "host not found" error, replace the host name with localhost in both the above and below commands.

  6. Locate the values.yaml file in your Offline Mediation Controller installation directory, such as /scratch/username/ocomc. Under the ocomcCore/lcmc.configEnv section, add the following fields, adjusting values if necessary to match your setup:

    adminsvrAuthMode: true
    
    adminsvrAuthuser: "true"
    
    adminsvrLdapurl: "ldap://oud-ds-rs-lbr-ldap.oudns.svc.cluster.local:1389"
    
    oudRootUserDn: cn=Directory Manager
    
    oudPath: /u01/oracle/user_projects/oud-ds-rs-0/OUD
    
    oudLdapPort: 1389
    
    oudBaseDn: dc=ocomcexample.com
    
    adminConnectPort: 4444
    
    hostName: oud-ds-rs-0.oud-ds-rs.oudns.svc.cluster.local

    Note:

    Upgrade the Offline Mediation Controller Helm installation after configuring the values.yaml file.

  7. Log in to Offline Mediation Controller through Administration Client.