Graph Sink

The Graph Sink microservice consumes topology data that has been normalized and inserts it into the Neo4j Graph database.

This microservice is part of the Topology microservice pipeline. See Understanding Microservice Pipelines in Unified Assurance Concepts for conceptual information.

Autoscaling is enabled by default for this microservice. You can optionally disable autoscaling when you deploy the microservice. See Configuring Autoscaling.

This microservice provides additional Prometheus monitoring metrics. See Graph Sink Self-Monitoring Metrics.

Graph Sink Prerequisites

Before deploying the microservice, confirm that the following prerequisites are met:

A microservice cluster is set up. See Microservice Cluster Setup.
The Apache Pulsar microservice is deployed. See Pulsar.

Deploying Graph Sink

To deploy the microservice, run the following commands:

su - assure1
export NAMESPACE=<namespace>
export WEBFQDN=<WebFQDN> 
a1helm install <microservice-release-name> assure1/graph-sink -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

In the commands:

<namespace> is the namespace where you are deploying the microservice. The default namespace is a1-zone1-pri, but you can change the zone number and, when deploying to a redundant cluster, change pri to sec.
<WebFQDN> is the fully-qualified domain name of the primary presentation server for the cluster.
<microservice-release-name> is the name to use for the microservice instance. Oracle recommends using the microservice name (graph-sink) unless you are deploying multiple instances of the microservice to the same cluster.

You can also use the Unified Assurance UI to deploy microservices. See Deploying a Microservice by Using the UI for more information.

Changing Graph Sink Configuration Parameters

When running the install command, you can optionally change default configuration parameter values by including them in the command with additional --set arguments. You can add as many additional --set arguments as you need.

For example:

Set a parameter described in Default Graph Sink Configuration by adding --set configData.<parameter_name>=<parameter_value>. For example, --set configData.LOG_LEVEL=DEBUG.
Disable autoscaling for the microservice by adding --set autoscaling.enabled=false.
Set an autoscaling parameter described in Graph Sink Autoscaling Configuration by adding --set autoscaling.<parameter_name>=<parameter_value>. For example, --set autoscaling.thresholds.backlogSize=2000.

Default Graph Sink Configuration

The following table describes the default configuration parameters found in the Helm chart under configData for the microservice.

Name	Default Value	Possible Values	Notes
LOG_LEVEL	INFO	FATAL, ERROR, WARN, INFO, DEBUG	Logging level used by application.
STREAM_INPUT	persistent://assure1/graph/sink	Text, 255 characters	Apache Pulsar topic path. Topic at end of path may be any text value.
BATCH_DELAY	0	Integer	Time to wait in seconds before processing a batch. If set to 0, messages will be processed one-by-one.
MAX_BATCH_SIZE	1000	Integer	Maximum number of messages that can be present in a batch.

Graph Sink Autoscaling Configuration

Autoscaling is supported for the MS microservice. See Configuring Autoscaling for general information and details about the standard autoscaling configurations.

The Graph Sink microservice also supports the additional configurations described in the following table.

Name	Default Value	Possible Values	Notes
thresholds.backlogSize	1000	Integer	The number of items that need to be in the backlog before the autoscaling starts additional processes.
thresholds.totalItemsProcessed	400	Integer	Total items processed by the microservice. If the average of total items processed in five minutes exceeds the threshold, pods will be scaled.

Graph Sink Self-Monitoring Metrics

The Graph Sink microservice exposes the self-monitoring metrics described in the following table to Prometheus.

Metric Name	Type	Description
graph_sink_backlog_size	Gauge	Number of items pending in the backlog
total_items_processed	Counter	Number of items processed
processing_time_per_item	Gauge	Processing time per item from receiving it to sending it to acknowledging it

Note:

Metric names in the database include a prefix that indicates the service that inserted them. The prefix is prom_ for metrics inserted by Prometheus. For example, total_items_processed is stored as prom_total_items_processed in the database.

Sample JSON Graph Schemas

The following examples show the expected JSON file schema formats for Graph Sink.

A single vertex schema:

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "vertex",
  "_version": "5.0.0",
  "vertex": {
    "@meta": false,
    "@method": "upsert",
    "@type": "Device",
    "keys": {
      "Name": "device-a.example.com",
      "ZoneID": 1
    },
    "properties": {
      "TimestampModified": "123456789000"
    }
  }
}

An edge-tuple schema:

{
  "@timestamp": "2021-03-30T15:32:21.690779663Z",
  "_agents": [
    {
      "@timestamp": "2021-03-30T15:32:21.690778749Z",
      "host": "pg-example.example.com",
      "id": "255c83bd-ff1e-2e10-6d92-dfecce7fba21",
      "app": "virtuora-connector",
      "version": "1.0.1",
      "input": "pg-example.example.com",
      "output": "persistent://assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "edgeTuple",
  "_version": "1.0.1",
  "edgeTuple": {
    "outVertex": {
      "@meta": false,
      "@method": "upsert",
      "@type": "Device",
      "keys": {
        "Name": "L100-107-TEST",
        "ZoneID": 1
      },
      "properties": {
        "TimestampModified": 1617118341
      }
    },
    "inVertex": {
      "@meta": false,
      "@method": "upsert",
      "@type": "Device",
      "keys": {
        "Name": "L100-107-TEST",
        "ZoneID": 0
      },
      "properties": {
        "TimestampModified": 1617118341
      }
    },
    "edge": {
      "@meta": false,
      "@method": "upsert",
      "@type": "ConnectsInterface",
      "keys": null,
      "properties": {
        "TimestampModified": 1617118341
      }
    }
  }
}

A schema for deleting a single vertex. You must specify _type and at least one key.

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "vertex",
  "_version": "5.0.0",
  "vertex": {
    "@meta": false,
    "@method": "delete",
    "@type": "Device",
    "keys": {
      "Name": "device-x.example.com",
      "ZoneID": 1
    }
  }
}

A schema for deleting by a single edge. You must specify _type and at least one unique property.

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "edge",
  "_version": "5.0.0",
  "edge": {
    "@meta": false,
    "@method": "delete",
    "@type": "HasInterface",
    "properties": {
      "connectionID": "1023"
    }
  }
}

Title and Copyright Information

Implementation Guide

G10620-01