Prometheus Metrics Processor

The Prometheus Metrics Processor microservice filters the metrics collected by the Prometheus server and inserts them into the Metric database.

This microservice is part of the Metric microservice pipeline. See Understanding Microservice Pipelines in Unified Assurance Concepts for conceptual information.

Prometheus Metrics Processor Prerequisites

1. A microservice cluster is set up. See Microservice Cluster Setup.

2. Deploy the Prometheus microservice (kube-prometheus-stack) with an override configuration:

   1. If Prometheus is already deployed, undeploy it:

      a1helm uninstall prometheus -n a1-monitoring
      

   2. Create a configuration override yaml file with the following content:

      prometheus:
        prometheusSpec:
          remoteWrite:
          - name: prometheus-metrics-processor
            url: "http://prometheus-metrics-processor.a1-monitoring.svc.cluster.local:8185/write"
            metadataConfig:
              send: false
      

   3. Redeploy Prometheus, using the override file:

   a1helm install prometheus assure1/kube-prometheus-stack -n a1-monitoring --set global.imageRegistry=$WEBFQDN -f <override-file-name>
   

   where <override-file-name> is the name of the override file that you created.

3. Create a filters.json file with metric filters in the core/default/collection/metrics/prometheus SVN directory. You can name the file anything, but for consistency, Oracle recommends filters.json.

   See Creating Prometheus Metrics Filters for information about the expected format for this file and the filters within it.

   If you do not set any filters, Prometheus Metrics Processor sends all incoming metrics to Telegraf for insertion into the Metric database.

Deploying Prometheus Metrics Processor

To deploy the microservice, run the following commands:

su - assure1
export NAMESPACE=a1-monitoring
export WEBFQDN=<WebFQDN> 
a1helm install <microservice-release-name> assure1/prometheus-metrics-processor -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

In the commands:

• Note that this microservice must be deployed to the a1-monitoring namespace rather than a zone-specific namespace.

• <WebFQDN> is the web fully-qualified domain name of the primary presentation server

• <microservice-release-name> is the name to use for the microservice instance. Oracle recommends using the microservice name (prometheus-metrics-processor) unless you are deploying multiple instances of the microservice to the same cluster.

You can also use the Unified Assurance UI to deploy microservices. See Deploying a Microservice by Using the UI for more information.

Changing Prometheus Metrics Processor Configuration Parameters

When running the install command, you can optionally change default configuration parameter values by including them in the command with additional --set arguments. You can add as many additional --set arguments as you need.

For example, set a parameter described in Default Prometheus Metrics Processor Configuration by adding --set configData.<parameter_name>=<parameter_value>. For example, --set configData.LOG_LEVEL=DEBUG.

You can also use the Unified Assurance UI to deploy microservices. See Deploying a Microservice by Using the UI for more information.

Default Prometheus Metrics Processor Configuration

Debugging Prometheus Metrics Processor

The Prometheus Metrics Processor consists of two components: prometheus-metrics-processor and telegraf. To see logs for each of the components, run the following commands, replacing placeholders as appropriate.

For prometheus-metrics-processor logs:

a1k logs <Pod Name> -n <Namespace> prometheus-metrics-processor

For telegraf logs:

a1k logs <Pod Name> -n <Namespace> telegraf

The <Pod Name> and <Namespace> values are available in the get pods command:

a1k get pods --all-namespaces

Creating Prometheus Metrics Filters

The Prometheus Metrics Processor filters the metrics sent to the database by using the filters.json file in the core/default/collection/metric/prometheus directory. You can filter metrics by exact names and labels, or by applying regular expressions to names and labels.

The file must use the following JSON schema format:

{
  "Filters": {
    "metrics": {
      "filterType": "<action>",
      "values": [
        "<metric_name>"
      ]
    },
    "labels": {
      "filterType": "<action>",
      "values": {
        "<label_key>": "<label_value>"
      }
    },
    "regex": [
      {
        "field": "<type>",
        "pattern": "<regex_pattern>",
        "filterType": "<action>"
      }
    ]
  }
}

The placeholders in the example are described below.

When creating filters, keep these points in mind:

• The filters are not automatically validated; you are responsible for using the correct format.

• You do not have to include all the filters. For example, you could include only a metrics filter or only a labels filter.

• If you don't specify any filters, all incoming metrics are sent to Telegraf.

• The filters work independently of each other.

Using Metrics

Metrics filters are applied to metric names. The following table describes the metrics filter properties. All of the properties in the table are required when using metrics filters.

For example, the following filter discards Pulsar storage latency metrics:

{
    "metrics": {
        "filterType": "discard",
        "values": [
          "pulsar_storage_write_latency_count",
          "pulsar_storage_ledger_write_latency_count"
        ]
    }
}

Using Labels Filters

Labels filters are applied to metric labels. The following table describes the labels filter properties. All of the properties in the table are required when using labels filters.

For example, the following filter keeps all metrics from the FCOM processor or in the a1-messaging namespace:

{
    "labels": {
        "filterType": "keep",
        "values": {
            "namespace": "a1-messaging",
            "service": "fcom-processor"
        }
    }
}

Using Regex Filters

Regex filters are applied to metric names or labels, depending on the value of the field property. The following table describes the regex filter properties. All of the properties in the table are required when using regex filters.

For example, the following filter discards metrics with names starting with kube or from the Pulsar topics for the Discovery Service from all zones:

{
    "regex": [
      {
        "field": "Name",
        "pattern": "kube.*",
        "filterType": "discard"
      },
      {
        "field": "topic",
        "pattern": "discovery-service-zone.*",
        "filterType": "discard"
      }
    ]
}