12 Monitoring and Maintaining Offline Mediation Controller Cloud Native
Learn how to monitor and maintain your Oracle Communications Offline Mediation Controller cloud native deployment.
Using Prometheus Operator to Monitor Offline Mediation Controller Cloud Native
Offline Mediation Controller cloud native tracks and exposes the following metric data in Prometheus format:
-
Node Manager-level statistics, which include:
-
The total network account records (NARs) processed
-
The current NARs processed
-
The current processing rate
-
The average processing rate
Node Manager sets expose metrics through a configurable service port. In your override-values.yaml file, each set includes a metrics section where the port for metrics can be specified using the metrics.service.port key. For example:sets: nm-cc name: "nm-cc" replicas: 1 metrics: service: port: 31300
-
-
JVM metrics for all Offline Mediation Controller components, which include:
-
Performance on the Node Manager level
-
JVM parameters
JVM metrics are exposed through the endpoint http://hostname:JVMport/metrics, where JVMport is the port number where the JVM metrics are exposed. They are exposed on the same port as the Node Manager metrics for each set, which is configured using metrics.service.port in your override-values.yaml file.
-
-
REST Services Manager metrics, which include:
- JVM metrics
- Service status
- Total uptime
- Garbage collection events
- Memory usage statistics
- Thread count
- Class loader statistics
REST Services Manager metrics are exposed through the Helidon framework's http://RSM_hostname:RSM_port/metrics endpoint, where RSM_hostname is the host name of the machine on which REST Services Manager is installed. The endpoint exposes metrics information in both JSON format (according to the MicroProfile Metrics specification) and plain text format suitable for Prometheus.
To monitor Offline Mediation Controller more easily, you can configure an external centralized metrics service, such as Prometheus Operator, to scrape metrics from each endpoint and store them for analysis and monitoring. You can then set up a visualization tool, such as Grafana, to display your metric data in a graphical format.
For the list of compatible Prometheus Operator and Grafana software versions, see "Offline Mediation Controller Cloud Native Deployment Software Compatibility" in Offline Mediation Controller Compatibility Matrix.
Enabling the Automatic Scraping of Metrics
You can configure the Prometheus Operator ServiceMonitor to automatically scrape Offline Mediation Controller metrics and Offline Mediation Controller REST Services Manager metrics. For more information about Prometheus Operator and ServiceMonitors, see the prometheus-operator documentation on the GitHub website (https://github.com/prometheus-operator/prometheus-operator/tree/main/Documentation/getting-started).
To enable the automatic scraping of Offline Mediation Controller metrics:
-
Install Prometheus Operator on your cloud native environment.
-
In your override-values.yaml file for oc-cn-ocomc, set the following keys:
-
ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.enabled: Set this key to true.
-
ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.interval: Set this to the interval at which to scrape metrics. The default is 10s.
-
ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.labels.app: Set this to the app label you want applied to the ServiceMonitor resource. This label is used by Prometheus to discover the ServiceMonitor via label selectors.
-
ocomcCore.ocomc.nodeManagerConfigurations.serviceMonitor.labels.release: Set this to the release label you want applied to the ServiceMonitor resource. This label is used by Prometheus to discover the ServiceMonitor via label selectors.
-
ocomcCore.ocomc.nodeManagerConfigurations.metrics.service.type: Set this to the service type, such as NodePort or ClusterIP. The default is ClusterIP.
-
(For REST Services Manager) ocomcRSM.rsm.serviceMonitor.enabled: Set this key to true.
-
-
Run the helm upgrade command to update the Offline Mediation Controller Helm release:
helm upgrade ReleaseName oc-cn-ocomc --values OverridingValuesFile -n NameSpace
where:
-
ReleaseName is the release name, which is used to track the installation instance.
-
OverrideValuesFile is the path to a YAML file that overrides the default configurations in the chart's values.yaml file.
-
NameSpace is the namespace in which to create Offline Mediation Controller Kubernetes objects.
-
Using the Sample Grafana Dashboards
The Offline Mediation Controller package includes sample Grafana Dashboard templates that you can use for visualizing metrics. To use the sample dashboards, import the following JSON files from the OMC_home/sampleData/dashboards directory into Grafana:
-
OCOMC_JVM_Dashboard.json: This dashboard lets you view JVM-related metrics for Offline Mediation Controller.
-
OCOMC_Node_Manager_Summary.json: This dashboard lets you view NAR processing metrics for the Node Manager.
-
OCOMC_Node_Summary.json: This dashboard lets you view NAR processing metrics for all nodes.
-
OCOMC_Summary_Dashboard.json: This dashboard lets you view NAR-related metrics for all Offline Mediation Controller components.
-
OCOMC_RSM_JVM_Dashboard.json: This dashboard lets you view JVM-related metrics for Offline Mediation Controller RSM.
For information about importing dashboards, see "Manage Dashboards" in the Grafana Dashboards documentation.
Offline Mediation Controller REST Services Manager Metrics
You can use various metrics to monitor the performance and health of Offline Mediation Controller REST Services Manager.
Table 12-1 describes the metrics mapped to their variable names in the /metrics endpoint response.
Table 12-1 Metrics Mapped to Variable Names
Variable Name | Metric Description |
---|---|
up | The status of the service, indicating whether it is up and running. |
base_jvm_uptime_seconds | The total uptime of the JVM in seconds since it was started. |
base_gc_total | The total number of garbage collection events that have occurred using the copy and mark/sweep algorithms. |
base_gc_time_seconds | The total time spent in garbage collection using the copy and mark/sweep algorithms., measured in seconds. |
base_memory_usedHeap_bytes | The amount of memory currently used in the heap, measured in bytes. |
base_memory_commitedHeap_bytes | The amount of memory that has been committed for use in the heap, measured in bytes. |
base_memory_maxHeap_bytes | The maximum amount of memory that can be allocated for the heap, measured in bytes. |
base_thread_count | The current number of threads that are actively running in the JVM. |
base_classloader_loadedClasses_count | The number of classes currently loaded into memory by the class loader. |
base_classloader_loadedClasses_total | The total number of classes that have been loaded into memory since the JVM started. |
base_classloader_unloadedClasses_total | The total number of classes that have been unloaded from memory since the JVM started. |
Automating Workflows Using RSM Request Automation
The RSM Request Automation feature enables response-driven execution of workflows within the Offline Mediation Controller cloud native environment. It is implemented as a Kubernetes job that dynamically executes workflows based on a structured JSON payload file. This file defines the sequence of API requests to run, response handling, and any conditional logic that determines the execution flow. This feature also allows users to execute any API exposed by the REST Services Manager.
With this feature, you can:
-
Automatically define and execute a sequence of steps based on predefined rules and conditions.
-
Use data from API responses to set environment variables for subsequent requests.
-
Leverage predefined flows for error handling and retries.
-
Support multipart file uploads as part of API requests.
-
Apply conditions based on response data to control execution the process flow.
-
Chain requests based on the output of previous ones.
Setting Up REST Services Manager Request Automation
To set up the REST Services Manager Request Automation feature in your Offline Mediation Controller cloud native deployment:
-
In your override-values.yaml file for oc-cn-ocomc-rsm, set the rsm.jobs.requestAutomation.enabled key to true.
-
Configure the following keys under rsm.jobs.requestAutomation.config:
-
apiUrl: Set this to the URL of your Offline Mediation Controller REST Services Manager service, such as http://ocomc-rsm:8080.
-
payloadFile: Specify the absolute path to your workflow payload JSON file.
-
loggingDIr: Define the directory where log files should be written.
-
(Optional) rsmOAuthToken: Set this to the REST Services Manager OAuth 2.0 token.
The following shows an example configuration:
jobs: requestAutomation: enabled: true config: apiUrl: "http://ocomc-rsm:8080" payloadFile: "/app/config/workflow-payload.json" loggingDir: "/app/volumes/ocomc-ext/logs
-
-
Create a JSON workflow payload file following the workflow file syntax. See "Creating a Workflow Payload File" for details.
-
Move the JSON workflow payload file to the directory specified in the payloadFile key.
-
Run a helm install or helm upgrade command to deploy your changes.
Creating a Workflow Payload File
The workflow payload file defines your automation workflow in JSON format.
The following is an example of the workflow file structure:
{ "workflow": { "flow": [ { "id": "UPLOAD_IMPORT_FILE_ACTION", "request": "UPLOAD_IMPORT_FILE", "onResponse": { "environment": { "IMPORT_ID": "/importId" } } }, { "id": "START_IMPORT_ACTION", "request": "START_IMPORT", "onResponse": { "statusMapping": { "2**": ["CHECK_IMPORT_COMPLETION_ACTION"] } } } ] }, "headers": {}, "environment": {}, "requests": [ { "id": "UPLOAD_IMPORT_FILE", "method": "POST", "uri": "/imports/uploadImportFile", "headers": {}, "multipartForm": { "configurationFile": "FILE::/path/to/export.xml" } }, { "id": "START_IMPORT", "method": "POST", "uri": "/imports/{{ENV:IMPORT_ID}}/tasks", "headers": {}, "payload": {} } ] }
The following is a breakdown of the syntax used within the workflow file, covering the key elements and options.
Table 12-2 describes the structure of the workflow section, which defines the sequence of actions to be executed.
Table 12-2 Workflow Section
Element | Description |
---|---|
flow |
An array defining the sequence of actions to run. |
id |
A unique identifier for the action within the workflow. |
request |
The ID of the API request to run (defined in the requests section). |
onResponse |
How to process the API response and subsequent actions. |
Table 12-3 describes the elements used in the OnResponse section to handle API responses and determine subsequent actions.
Table 12-3 OnResponse Section
Element | Description |
---|---|
environment |
Specifies the environment variables to set based on the response. The value from /importId in the response is assigned to the environment variable IMPORT_ID. These variables can later be accessed in requests using the {{ENV:VARIABLE_NAME}} syntax. "onResponse": { "environment": { "IMPORT_ID": "/importId" } } |
statusMapping |
Maps HTTP status codes to specific actions, such as ABORT. In this case, if a 5xx error is encountered, the transaction is halted and terminated. "statusMapping": { "5**": ["ABORT"] } |
retry |
Configures automatic retries for a
request.
"retry": { "maxRetries": "5", "interval": "10000" } |
maxRetrics |
Specifies the maximum number of retry attempts. |
dataMapping |
Specifies conditions based on values in the API response.
"dataMapping": [ { "path": "/importTaskItem/taskFinished", "equals": "true", "then": [] }, { "path": "/importTaskItem/taskFinished", "equals": "false", "then": ["RETRY"] } ] |
Table 12-4 describes the elements required to define individual API requests that can be referenced within a workflow.
Table 12-4 Requests Section
Element | Description |
---|---|
id |
A unique identifier for the request. |
method |
The HTTP method, such as GET, POST, PUT, or DELETE. |
uri |
The URI or endpoint for the API request. The {{ENV:VARIABLE_NAME}} syntax injects values from environment variables. |
headers |
Any required HTTP headers. |
payload |
The data to be sent in the request body (for methods like POST). |
multipartForm |
Defines files to be uploaded as part of the request. The FILE:: prefix indicates sending a file from a local path. |
Managing a Helm Release
After you install a Helm chart, Kubernetes manages all of its objects and deployments. All pods created through oc-cn-ocomc are wrapped in a Kubernetes controller, which creates and manages the pods and performs health checks. For example, if a node fails, a controller can automatically replace a pod by scheduling an identical replacement on a different node.
Administrators can perform these maintenance tasks on a Helm chart release:
Tracking a Release's Status
When you install a Helm chart, it creates a release. A release contains Kubernetes objects, such as ConfigMap, Secret, Deployment, Pod, PersistentVolume, and so on. Not every object is up and running immediately. Some objects have a start delay, but the Helm install command completes immediately.
To track the status of a release and its Kubernetes objects, enter this command:
helm status ReleaseName -n Namespace
where:
-
ReleaseName is the name you assigned to this installation instance.
-
NameSpace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.
Updating a Release
To update any key value after a release has been created, enter this command. This command updates or re-creates the impacted Kubernetes objects, without impacting other objects in the release. It also creates a new revision of the release.
helm upgrade ReleaseName oc-cn-ocomc-helm-chart --values OverridingValueFile --values NewOverridingValueFile -n Namespace
where:
-
ReleaseName is the name you assigned to this installation instance.
-
OverridingValueFile is the path to the YAML file that overrides the default configurations in the oc-cn-ocomc/values.yaml file.
-
NewOverridingValueFile is the path to the YAML file that has updated values. The values in this file are newer than those defined in values.yaml and OverridingValueFile.
-
Namespace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.
Checking a Release's Revision
Helm keeps track of the revisions you make to a release. To check the revision for a particular release, enter this command:
helm history ReleaseName -n Namespace
where:
-
ReleaseName is the name you assigned to this installation instance.
-
Namespace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.
Rolling Back a Release to a Previous Revision
To roll back a release to any previous revision, enter this command:
helm rollback ReleaseName RevisionNumber -n Namespace
where:
-
ReleaseName is the name you assigned to this installation instance.
-
RevisionNumber is the value from the Helm history command.
-
Namespace is the namespace in which the Offline Mediation Controller Kubernetes objects reside.
Rolling Back an Offline Mediation Controller Cloud Native Upgrade
If you encounter errors after upgrading, you can roll back to a previous version of Offline Mediation Controller.
The following procedure assumes that you have upgraded Offline Mediation Controller from 12.0 Patch Set 5 (Revision 1), to 12.0 Patch Set 6 (Revision 2), and then to 15.1 (Revision 3). To roll back your upgrade from 15.1 to 12.0 Patch Set 6, you would do this:
-
Check the revision history of the Offline Mediation Controller release:
helm history ReleaseName -n Namespace
You should see something similar to this:
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Thu May 30 07:12:46 2030 superseded oc-cn-ocomc-helm-chart 12.0.0.5.0 Initial install 2 Thu May 30 08:32:09 2030 superseded oc-cn-ocomc-helm-chart 12.0.0.6.0 Upgraded successfully 3 Thu May 30 09:50:00 2030 deployed oc-cn-ocomc-helm-chart 15.1.0.0.0 Upgraded successfully
-
Roll back the release to Offline Mediation Controller 12.0 Patch Set 6:
helm rollback ReleaseName 2 -n BrmNamespace
If successful, you will see this:
Rollback was a success! Happy Helming!
-
Check the revision history of the Offline Mediation Controller release:
helm history ReleaseName -n BrmNamespace
If successful, you should see something similar to this:
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Thu May 30 07:12:46 2030 superseded oc-cn-ocomc-helm-chart 12.0.0.5.0 Initial install 2 Thu May 30 08:32:09 2030 superseded oc-cn-ocomc-helm-chart 12.0.0.6.0 Upgraded successfully 3 Thu May 30 09:50:00 2030 superseded oc-cn-ocomc-helm-chart 15.1.0.0.0 Upgraded successfully 4 Thu May 30 11:25:00 2030 deployed oc-cn-ocomc-helm-chart 12.0.0.6.0 Roll back to 2
Integrating Oracle Unified Directory with Offline Mediation Controller Cloud Native
-
Open a terminal session within the running pod:
kubectl exec -it oud-ds-rs-0 -n oudns -- /bin/bash
-
Inside the Oracle Unified Directory container, create a temporary directory (tempdir) to hold configuration files.
-
From the Offline Mediation Controller installation directory, copy the oudConfig and populateDir.ldif files to tempdir.
-
Inside tempdir, create a file named populateDirTemp.ldif. This file updates the user information in Oracle Unified Directory to match the Offline Mediation Controller requirements. Add the following content:
dn: uid=Admin,ou=People,dc=ocomcexample.com changetype: modify replace: userpassword userpassword: adminpassword
-
Run the tempdir/oudConfig script with the Oracle Unified Directory container:
sh oudConfig -i oud_instance_path -h oud_host -p oud_admin_port -d "oud_binddn" -w oud_admin_password -b "dc=ocomcexample.com" -l oud_ldapport
Note:
If the command fails with a "host not found" error, replace the host name with localhost in both the above and below commands.
-
Locate the values.yaml file in your Offline Mediation Controller installation directory, such as /scratch/username/ocomc. Under the ocomcCore/lcmc.configEnv section, add the following fields, adjusting values if necessary to match your setup:
adminsvrAuthMode: true adminsvrAuthuser: "true" adminsvrLdapurl: "ldap://oud-ds-rs-lbr-ldap.oudns.svc.cluster.local:1389" oudRootUserDn: cn=Directory Manager oudPath: /u01/oracle/user_projects/oud-ds-rs-0/OUD oudLdapPort: 1389 oudBaseDn: dc=ocomcexample.com adminConnectPort: 4444 hostName: oud-ds-rs-0.oud-ds-rs.oudns.svc.cluster.local
Note:
Upgrade the Offline Mediation Controller Helm installation after configuring the values.yaml file.
-
Log in to Offline Mediation Controller through Administration Client.