FCOM Processor

The FCOM Processor microservice takes collected fault data from an input topic, runs it through fault common object models (FCOM) to normalize into an event structure, and sends the output to an output topic. The Event Sink microservice subscribes to the default output topic, but you can configure any topic for additional enrichment or suppression.

This microservice is part of the Event microservice pipeline. See Understanding the Event Pipeline in Unified Assurance Concepts for conceptual information.

Autoscaling is enabled by default for this microservice. You can optionally disable autoscaling when you deploy the microservice. See Configuring Autoscaling.

This microservice provides additional Prometheus monitoring metrics. See FCOM Processor Self-Monitoring Metrics.

FCOM Processor Prerequisites

Before deploying the microservice, confirm that the following prerequisites are met:

1. A microservice cluster is set up. See Microservice Cluster Setup.

2. The Apache Pulsar microservice is deployed. See Pulsar.

Deploying FCOM Processor

To deploy the microservice, run the following commands:

su - assure1
export NAMESPACE=<namespace>
export WEBFQDN=<WebFQDN>
a1helm install <microservice-release-name> assure1/fcom-processor -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

In the commands:

• <namespace> is the namespace where you are deploying the microservice. The default namespace is a1-zone1-pri, but you can change the zone number and, when deploying to a redundant cluster, change pri to sec.

• <WebFQDN> is the fully-qualified domain name of the primary presentation server for the cluster.

• <microservice-release-name> is the name to use for the microservice instance. Oracle recommends using the microservice name (fcom-processor) unless you are deploying multiple instances of the microservice to the same cluster.

You can also use the Unified Assurance UI to deploy microservices. See Deploying a Microservice by Using the UI for more information.

Changing FCOM Processor Configuration Parameters

When running the install command, you can optionally change default configuration parameter values by including them in the command with additional --set arguments. You can add as many additional --set arguments as you need.

For example:

• Set a parameter described in Default FCOM Processor Configuration by adding --set configData.<parameter_name>=<parameter_value>. For example, --set configData.LOG_LEVEL=DEBUG.

• Disable autoscaling for the microservice by adding --set autoscaling.enabled=false.

• Set an autoscaling parameter described in FCOM Processor Autoscaling Configuration by adding --set autoscaling.<parameter_name>=<parameter_value>. For example, --set autoscaling.thresholds.backlogSize=2000.

Default FCOM Processor Configuration

The following table describes the default configuration parameters found in the Helm chart under configData for the microservice.

FCOM Processor Autoscaling Configuration

Autoscaling is supported for the FCOM Processor microservice. See Configuring Autoscaling for general information and details about the standard autoscaling configurations.

The FCOM Processor microservice also supports the additional configurations described in the following table.

FCOM Processor Self-Monitoring Metrics

The FCOM Processor microservice exposes the self-monitoring metrics described in the following table to Prometheus.

FCOM Overrides

An FCOM Override is an authoritative, language-neutral format written in JSON that enables the ability to override or build upon the basic functionality across the processing steps of the FCOM Processor.

After you have written an FCOM Override and uploaded it to the FCOM_FILES_LOCATION path in SVN, you must restart the FCOM Processor microservice to load the new rules. Once messages start coming in, the defined overrides run if they match the scope or objectName defined in any override.

There are four different stages during processing where an override can run. In order, these stages are:

• pre global - This would run on all messages that are received before they are converted to an Event.

• pre object specific - This would only run on a message that has an objectName that is explicitly defined in the override before it is converted to an Event

• post object specific - This would only run on a message that has an objectName that is explicitly defined in the override after it is converted to an Event

• post global - This would run on all messages that are received after they are converted to an Event.

JSON Path

There are a handful of JSON Paths which can be used to access locations outside of the message you're currently referencing. These are:

• $.lookups

  • This path can be used reference a lookup file value. The full breakdown of the path is $.lookups.{lookupfile name}.{key}.

• $.foreach

  • This path stores key/val values only during the lifecycle of an iteration when using a foreach processor. The full key for the key value would be $.foreach.{keyVal} while the val value would be $.foreach.{valField} . Look at the foreach processor definition for more information.

• $.localmem

  • This is a local memory location. It's cleared on a per event basis, so you can set a local memory value like $.localmem.test in a pre-processor and then reference it in a post processor.

• $.globalmem

  • This is a global memory location which is accessible from all pods within a deployment. It requires redis to be running in the cluster. The values are only cleared if the redis installation is restarted or another processor removes it. You can set a global memory value using like: $.globalmem.test.

• $.error

  • This path holds an error message based off the current state. For example, if a processor throws an error during runtime and you're attempting to set a field/print the error you can reference the error by doing $.error.message in a processor.

Override Format

The following table describes the expected override format.

Example:

{
    "name": "Override Example",
    "description": "This is an example of a global FCOM override",
    "domain": "fault",
    "method": "trap",
    "scope": "post",
    "version": "v2",
    "@objectName": "GLOBAL",
    "_type": "override",
    "processors": [
        {
            "set": {
                "source": "Hello, this is an example of overriding the event Summary field",
                "targetField": "$.event.Details"
            }
        },
        {
            "set": {
                "source": "Switch is down with value: %s",
                "args": [ "$.event.Details.trap.variables.0" ],
                "targetField": "$.event.EventType"
            }
        }
    ]
}

Processors

You can use the processors described below in FCOM overrides.

append

Appends the value of source to the specific array and stores in the targetField.

Example

{
    "append": {
        "source": "Example Value",
        "array": [],
        "targetField": "$.event.NewArray"
    }
}

The value of $.event.NewArray will be [ "Example Value" ] after running.

appendToOutputStream

Immediately sends the source value to an external output stream. In-cluster Pulsar topics are supported.

Example

{
    "appendToOutputStream": {
        "source": "$.trap",
        "output": "pulsar+ssl:///assure1/event/sink"
    }
}

The value the JSON Path $.trap references will be sent to the assure1 event/sink topic within the cluster.

break

Immediately breaks from the enclosing foreach loop.

Example

{
    "break": {
    }
}

convert

Does a conversion of the value of source into the targetField based on the type of conversion.

Example

{
    "convert": {
        "source": "$.event.Count",
        "type": "inttostring",
        "targetField": "$.event.CountString",
        "ignoreFailure": true
    }
}

The value of $.event.CountString will be the value of count in a string format, if it throws an error while converting the value of $event.Count it will continue on to the next processor in the override.

copy

Copies the value of source to a target location.

Example

{
    "copy": {
        "source": "$.event.Count",
        "targetField": "$.event.CopiedCount"
    }
}

The value of $.event.CopiedCOunt will be the same as "$.event.Count"

date

Converts a source date and stores it in a target field.

Example

{
    "date": {
        "source": "",
        "offset": "-2h45m",
        "timezone": "America/New_York",
        "targetField": "$.event.CurrentTimeInEST"
    }
}

The value of $.event.CurrentTimeInEST will be the current time in EST subtracted by 2h 45m.

discard

Immediately discards a message.

Example:

{
    "discard": {
    }
}

This would immediately discard the message - which means that successive overrides won't be run nor will it be sent to the output topic, and the fcom-processor will continue on to the next message.

foreach

Loops over an array or an object (key-val map).

Example

{
    "foreach": {
        "source": "$.event.Details.trap.variables",
        "keyField": "i",
        "valField": "c",
        "then": [
            {
                "log": {
                    "source": "The index is %s and the value is %s",
                    "args": [
                        "$.foreach.i",
                        "$.foreach.c"
                    ],
                    "type": "info"
                }
            },
            {
                "if": {
                    "conditions": {
                        "and": [
                            {
                                "property": "$.foreach.i",
                                "operator": ">",
                                "value": 3
                            }
                        ]
                    },
                    "then": [
                        {
                            "break": {}
                        }
                    ],
                    "else": []
                }
            }
        ]
    }
}

This foreach loops logs The index is %s and the value is %s to the screen until it reaches the fourth index of the array. It then breaks out of the loop and stops running.

grok

Runs a pre-defined grok pattern on a string and stores the result in a target object.

Example

{
    "grok": {
        "source": "$.syslog.datagram",
        "pattern": "%LINK-5-CHANGED: Interface %{INTERFACE:interface}, changed state to %{STATUS:status}",
        "targetField": "$.syslog.variables"
    }
}

The result of $.syslog.variables will be:

"variables": {
    "interface": "someinterfacename",
    "status": "somestatusstring"
}

if

This processor runs an array of processors if the list of specified conditions are true, and runs a different set of processors if the list of specified conditions are false.

Example 1

{
    "if": {
        "conditions": {
            "and": [
                {
                    "property": "$.event.Count",
                    "operator": "==",
                    "value": 3
                }
            ]
        },
        "then": [
            {
                "log": {
                    "source": "The condition is true"
                }
            }
        ],
        "else": []
    }
}

This will result in a debug log message being printed to the screen if the event count is equal to 3.

Example 2

{
    "if": {
        "conditions": {
            "or": [
                {
                    "property": "$.event.Count",
                    "operator": "==",
                    "value": 3
                },
                {
                    "property": "$.event.Count",
                    "operator": ">=",
                    "value": 1000
                }
            ]
        },
        "then": [
            {
                "log": {
                    "source": "The condition is true"
                }
            }
        ],
        "else": []
    }
}

This will result in a debug log message being printed to the screen if the event count is equal to 3 or greater than or equal to 1000.

interpolate

Interpolates the JSON Path variables contained in the source with the values which exist in the message currently being processed.

Example

{
    "interpolate": {
        "source": "The $.event.EventType event expires in $.event.ExpireTime seconds",
        "targetField": "$.event.Summary"
    }
}

length

Calculates the length of a string or a JSON Path referencing a map/array/string and stores it in the target field.

Example

{
    "length": {
        "source": "$.trap.oid",
        "targetField": "$.localmem.oidlength"
    }
}

The result of $.localmem.oidlength (which is a per message local memory location) will be the string length of the oid.

log

Outputs a log message to stdout.

Example

{
    "log": {
        "source": "There are %d devices in the device catalog",
        "args": [ 100 ],
        "type": "info"
    }
}

This example would output an info log message of "There are 100 devices in the device catalog" to the screen.

lookup

Enables external querying against MySQL, Neo4j, or the Assure1 API.

cache

fallback

DBLookupProperties (db)

DBLookupProperties Example

{
  "lookup": {
    "source": "db",
    "properties": {
      "database": "Event",
      "variables": {
        "v1": 10
      },
      "query": "SELECT * From Events LIMIT :v1"
    },
    "cache": {
      "enabled": true,
      "object": "Event",
      "keys": [ "$.EventKey" ],
      "ttl": 1000
    },
    "targetField": "$.localmem.dbresults"
  }
}

Returns 10 events from the Event database and stores it in the local memory location $.localmem.dbresults. It also populates the redis cache with 10 json records, each having a key name of Event:{Eventkey} with a 1000 second expiration.

GDBLookupProperties (gdb)

GDBLookupProperties Example

{
    "lookup": {
        "source": "gdb",
        "properties": {
            "database": "graph",
            "variables": {
                "v1": 1
            },
            "query": "MATCH(v) WHERE v.ZoneID > $v1 RETURN v"
        },
        "cache": {
            "enabled": true,
            "object": "Vertex",
            "keys": [ "$.v.Props._id" ]
        },
        "targetField": "$.localmem.gdbresults"
    }
}

Returns all the vertices which have a zone id that is greater than 1 and stores it in the local memory location $.localmem.gdbresults. It stores each record within the result in the cache with a key name of Vertex:{vertex uuid} with a 300 second expiration.

APILookupProperties (api)

APILookupProperties Example

{
    "lookup": {
        "source": "api",
        "properties": {
            "variables": {
            },
            "endpoint": "/api/device/devices?page=1&start=0"
        },
        "cache": {
            "enabled": true,
            "object": "Device",
            "keys": [ "$.DeviceID" ]
        },
        "targetField": "$.localmem.apiresults"
    }
}

Returns all devices in the first page of the device catalog and stores it in the local memory location $.localmem.apiresults. It stores each record within the result within the cache with a key name of 'Device:{DeviceID}' and a 300 second expiration.

CacheLookupProperties (cache)

CacheLookupProperties Cypher Query Example

{
    "lookup": {
        "source": "cache",
        "properties": {
            "module": "graph",
            "key": "graph",
            "query": "MATCH(n) RETURN n"
        },
        "targetField": "$.localmem.cacheresults"
    }
}

Uses a cypher query to search the topology in the redis 'graph' key and returns the results into the local memory location $.localmem.cacheresults. Please visit https://redis.io/commands/graph.query/ for an explanation on the graph module query syntax.

CacheLookupProperties Fulltext Search Example

{
    "lookup": {
        "source": "cache",
        "properties": {
            "module": "ftsearch",
            "key": "jsonidx:device",
            "query": "@DeviceID:[129, 129]"
        },
         "fallback": {
            "source": "db",
            "properties": {
                "query": "SELECT *, INET_NTOA(IPAddress) AS IPAddress, INET6_NTOA(IPv6Address) AS IPv6Address, ST_AsGeoJson(GeoLocation) AS GeoLocation FROM Devices WHERE DeviceID = 129"
            },
            "keys": [
                "$.DeviceID"
            ],
            "object": "Device",
            "ttl": 10000
        }
        "targetField": "$.localmem.cacheresults"
    }
}

Does a fulltext search for any records in the device json repository which have a DeviceID of 129 and stores the results into the local memory location $.localmem.cacheresults . If a device isn't found in the cache, it falls back to a direct db lookup which returns a result if it exists and populates it in the cache as Device:129 with a 10000 second expiration. Please visit https://redis.io/docs/stack/search/reference/query_syntax/ for an explanation on the fulltext search module query syntax.

math

Calculates the length of a string or a JSON Path referencing a map/array/string and stores it in the target field.

Example:

{
    "math": {
        "source": "$.event.Count",
        "value": 2,
        "operation": "*",
        "targetField": "$.localmem.CountTimesTwo"
    }
}

The result of $.localmem.CountTimesTwo (which is a per message local memory location) will be the value of the event.Count times 2.

regex

Runs a regex pattern on a source value. The results, and an indication of whether there was a match, are stored in the location specified in the targetField parameter. The results are stored as a double dimensional array, and each array can contain multiple strings that match the pattern. The first element of each array is the entire matched substring and the following elements are the submatched strings within the regex pattern.

Example:

{
    "regex": {
        "source": "IPAddress: 192.0.2.1 IPAddress: 192.0.2.2",
        "pattern": "(\\w+): (\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})",
        "targetField": "$.event.RegexLocation"
    }
}

The value of $.event.RegexLocation in this example will be:

"RegexLocation": {
    "matched": true,
    "results": [["IPAddress: 192.0.2.1", "IPAddress", "192.0.2.1"], ["IPAddress: 192.0.2.2", "IPAddress", "192.0.2.2"]]
}

Example with variables:

{
    "regex": {
        "source": "Events are cleared",
        "pattern": "Events are (?P<text>.*$)",
        "targetField": ""
    }
}

The value of targetField is set to empty in order to capture the variable text from the regex pattern. The value of $text in this example will be cleared.

remove

Deletes the source key and value referenced by the JSON Path.

Example:

{
    "remove": {
        "source": "$.trap.timeTicks"
    }
}

rename

Renames the source field into the target field.

Example:

{
    "rename": {
        "source": "$.event.Details",
        "targetField": "$.event.DetailsOld"
    }
}

replace

Replaces all instances of the specified pattern in the source field and then stores the new result in the targetField. If a regex pattern is needed, it must be explicitly enabled by setting the regex field to true. If the source is an array, then it matches and replaces against each value in the array.

Example:

{
    "replace": {
        "source": "This is a test",
        "pattern": "a test",
        "replacement": "not a test",
        "targetField": "$.localmem.example"
    }
}

The value of $.localmem.example after this example runs would be This is not a test

set

Takes the source value, applies the variadic args (if any are specified), and stores the result in a target field.

Example:

{
    "set": {
        "source": "$.event.%s",
        "args": [ "Details" ],
        "targetField": "$.event.Details2"
    }
}

The value of $.event.Details2 after this example runs would be identical to the value of $.event.Details.

setOutputStream

Overrides the output stream the message is sent to at the end of processing. This is on a per-message basis, and the original output stream is reset at the beginning of processing.

Example:

{
    "setOutputStream": {
        "output": "pulsar+ssl:///assure1/event/sink"
    }
}

sort

Sorts a source array and stores the sorted result in a target field.

Example:

{
    "sort": {
        "source": "$.trap.variables.0",
        "targetField": "$.trap.sortedVariables"
    }
}

split

Splits a string by a delimiter and stores the result as a string array in the target field.

Example:

{
    "split": {
        "source": "1,2,3,4",
        "delimiter": ",",
        "targetField": "$.localmem.splitarr"
    }
}

The value of $.localmem.splitarr will be:

{
    "splitarr": ["1", "2", "3", "4"]
}

strcase

Applies a case type on the value of source and stores it in the target field.

Example:

{
    "strcase": {
        "source": "HELLO, WORLD",
        "type": "lower",
        "targetField": "$.localmem.lowercase"
    }
}

The value of $.localmem.lowercase after this example runs will be hello, world

substr

Stores a substring of a source string in a target field.

Example:

{
    "substr": {
        "source": "Hello",
        "start": 1,
        "targetField": "$.localmem.substr"
    }
}

The value of $.localmem.substr after this example runs will be ello

switch

Takes a source value and operator and checks the condition against the value of each case.

SwitchCase

Example:

 {
    "switch": {
        "source": "$.localmem.val1",
        "operator": "!=",
        "case": [
            {
                "match": 2,
                "then": [
                    {
                        "discard": {
                        }
                    }
                ]
            },
            {
                "match": 5,
                "operator": "==",
                "then": [
                    {
                        "discard": {
                        }
                    }
                ]
            }
        ],
        "default": [
            {
                "log": {
                    "type": "info",
                    "source": "Do nothing since none of the cases were met"
                }
            }
        ]
    }
}

trim

Trims a cutset from the beginning and end of a string.

Example:

{
    "trim": {
        "source": "Hello",
        "cutset": "H",
        "targetField": "$.localmem.trim"
    }
}

The value of $.localmem.trim after this example runs will be ello

FCOM Lookups

The FCOM Processor has the ability to read a lookup file and store it in memory for reference within an override. This is very useful if a set of constant values need to be used.

Example:

{
    "name": "alertTypeMap",
    "_type": "lookup",
    "lookup": {
        "1": "Fault",
        "2": "Outage",
        "3": "Overload",
        "4": "Reboot",
        "10": "Failover",
        "12": "Restore"
    }
}

Usage in an override

{
    "set": {
        "source": "$.lookups.alertTypeMap.%s",
        "args": [ "$.trap.variables.0.value" ],
        "targetField": "$.trap.varMapping"
    }
}

If the value of $.trap.variables.0.value was "10" then the value of $.trap.varMapping would equal Failover

FCOM Grok Definitions

The FCOM Processor has the ability to load a set of custom grok patterns for use in overrides. There is a list of vendor defined commonly used patterns in the vendor _grok folder within the FCOM_FILES_LOCATION path.

Example:

{
    "name": "Custom Grok Definition 1",
    "_type": "grok",
    "grok": {
        "VALUE": ".*",
        "COMMONMAC": "(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})",
        "DATA": ".*?"
    }
}

Using custom grok definitions in a FCOM override

{
    "grok": {
        "source": "$.syslog.datagram",
        "pattern": "%CISCO-LINK-5: The error message is: %{VALUE:message}",
        "targetField": "$.syslog.variables"
    }
}

The contents of $.syslog.variables would be an object containing a message field with a value of whatever string is after The error message is:.