Integrate Oracle Analytics with Oracle Cloud Infrastructure Functions

Integrate Oracle Analytics with Oracle Cloud Infrastructure (OCI) functions so that you can use them in data flows to transform data.

About Using OCI Functions in Oracle Analytics

You can transform data in Oracle Analytics using functions created in OCI. For example, you might use a language conversion function to convert English text into Spanish or German.

You register OCI functions in Oracle Analytics first, and then any Oracle Analytics user with BI Service Administrator or DV Content Author privileges can use them in data flows.

About Configuring OCI Functions to use in Oracle Analytics

Create functions in OCI Console so that you can use them to transform data in Oracle Analytics data flows.

About Registering OCI Functions in Oracle Analytics

When you register an OCI function in Oracle Analytics, if it's grayed out in the Select a Function dialog, check that it's configured with the oac-compatible tag and invoked with the correct funcMode value.
Description of oci-functions-select-function-register.png follows
Description of the illustration oci-functions-select-function-register.png

About Configuring OCI Functions to use in Oracle Analytics

OCI functions that you want to use in Oracle Analytics must have an oac-compatible tag and the function code must include the funcMode variable. Configure these settings in OCI Console:

oac-compatible - Add a free form tag to the function with the name oac-compatible and set the value to true.
Description of the illustration oci-function-tags.png
funcMode - Include the funcMode variable in your function code. Oracle Analytics sends a request to register an OCI function with funcMode=describeFunction and a request to invoke an OCI function with funcMode=executeFunction. The handler function in func.py should operate in describeFunction mode or executeFunction mode. This can be done based on the value of the input variable funcMode from the request as shown below.

Description of the illustration oci-function-mode.png

Here's the request and response format of OCI Functions for Word Count example in python.

funcMode = describeFunction
```
{ "funcMode": "describeFunction"}
```
When funcMode in the request is 'describeFunction', the function should return the function definition with status (returnCode & errorMessage), outputs(name, dataType of output column), parameters (name, description, type and so on of the input parameter), bucketName and so on as a JSON object in the following format (example of word count):
```
funcDefinition = {
    "status": {
        "returnCode": 0,
        "errorMessage": ""
    },
    "funcDescription": {
        "outputs": [
            {"name": "word_count", "dataType": "integer"}
        ],
        "parameters": [
            {"name": "textColumn", "displayName": "Text Column",
             "description": "Choose column to count words", "required": True,
             "value": {"type": "column"}}
        ],
        "bucketName": "bucket-OCI-FAAS",
        "isOutputJoinableWithInput": True
    }
}
```
Add the following code in func.py to return funcDefinition when funcMode in the request is 'describeFunction'.
```
def handler(ctx, data: io.BytesIO = None):
    response_data = ""
    try:
        body = json.loads(data.getvalue())
        funcMode = body.get("funcMode")
        if funcMode == 'describeFunction':
           response_data = json.dumps(funcDefinition)
    except (Exception, ValueError) as ex:
        response_data = json.dumps(
            {"error": "{0}".format(str(ex))})
    return response.Response(
        ctx, response_data,
        headers={"Content-Type": "application/json"}
    )
```
funcMode = executeFunction - When funcMode is 'executeFunction', the actual function logic should be executed and the response should be sent back to Oracle Analytics with the output. When the registered function is invoked from dataflow in Oracle Analytics, the request object will be in the following format with the column name in args, input(bucketName, fileName, fileExtension, fileName, method & rowID) and output(bucketName, fileName & fileExtension).
```
{
    "args":
    {
        "textColumn": "REVIEW"
    },
    "funcMode": "executeFunction",
    "input":
    {
        "bucketName": "bucket-OCI-FAAS",
        "fileExtension": ".csv",
        "fileName": "oac-fn-e99cd4fddb3844be89c7af6ea4bbeb76-input",
        "method": "csv",
        "rowID": "row_id"
    },
    "output":
    {
        "bucketName": "bucket-OCI-FAAS",
        "fileExtension": ".csv",
        "fileName": "oac-fn-e99cd4fddb3844be89c7af6ea4bbeb76-output"
    }
}
```

About Creating Functions in OCI

You can create functions using the OCI Console, command line (Fn Project CLI), or API. For example, in OCI Console, click Developer Services, then Functions and follow the on-screen instructions to create applications and one or more functions. For details, see Creating Functions in OCI Documentation. If you're creating functions for the first time, follow the steps in the end to end example Creating, Deploying, and Invoking a Helloworld Function. Any functions that you want to use in Oracle Analytics must include the free form tag oac-compatible=true (see About Configuring OCI Functions to use in Oracle Analytics above).

Tips on Creating Functions in OCI Console

Enable Logs - By default, logs are disabled for functions. To enable logs for functions, on the Applications page, click Logs, and enable the log provided for the Log Name. Under Resources, you can now select Explore Log, and drill into log entries to debug and diagnose function issues.
Include Dependencies - Include dependent packages in the requirements.txt file for the function.
Follow Indentation Rules- Follow the Python indentation rules while coding Python functions.
Use Function Response Errors - Use the error details listed on the Dataflow page to diagnose issues.
Test Functions Locally First - Before creating a function in OCI, test the function locally in your development environment to ensure that the function is syntactically and logically correct.
Test functions in OCI - Before integrating functions with Oracle Analytics, make sure that you can deploy and invoke them successfully as standalone functions in OCI.
Increase the Timeout- By default, functions have a timeout of 30 seconds. If necessary, increase the timeout setting (for example, change it to 300 seconds) in the Edit Function page in OCI Console.
Increase the Memory- By default, functions have a memory limit of 1024 MB. If necessary, increase the memory limit in the Edit Function page in OCI Console.
Optimize Space - Delete unused files immediately after use.
Beware of Network Latency - Bear in mind that network latency might cause a slight delay in overall processing.

Policies Required to Integrate OCI Functions with Oracle Analytics

To integrate Oracle Analytics with OCI functions, make sure that you have the required security policies.

The OCI user that you specify in the connection between Oracle Analytics Cloud and your OCI tenancy must have read, write, and delete permissions on the compartment containing the OCI resources you want to use. Ensure that the OCI user belongs to a user group with the following minimum OCI security policies. When you connect to an OCI tenancy from Oracle Analytics, you can use either an OCI API key or resource principal.

Note:

For resource principal, to include all Analytics instances under a compartment, specify: {request.principal.type='analyticsinstance', request.principal.compartment.id='<compartmentA_ocid>'} instead of {request.principal.id='<analytics_instance_ocid>'}.

API Key Policies	Resource Principal Policies
`Allow group <group_name> to use functions-family in compartment <compartment_name>`	`Allow any-user to use functions-family in compartment <compartment_name> where all {request.principal.id='<analytics_instance_ocid>'}`
`Allow group <group_name> to read buckets in compartment <compartment_name>`	`Allow any-user to read buckets in compartment <compartment_name> where all {request.principal.id='<analytics_instance_ocid>'}`
`Allow group <group_name> to manage objects in compartment <compartment_name> where target.bucket.name='<staging_bucket_name>'`	`Allow any-user to manage objects in compartment <compartment_name> where all {request.principal.id='<analytics_instance_ocid>', target.bucket.name='<staging_bucket_name>'}`
`Allow group <group_name> to read objectstorage-namespaces in tenancy`	`Allow any-user to read objectstorage-namespaces in tenancy where all {request.principal.id='<analytics_instance_ocid>'}`

Dynamic Group Policy	Description
`Allow dynamic-group <dynamic_group> to manage objects in compartment <compartment_name> where target.bucket.name='<staging_bucket_name>'`	Provides access to staging bucket for the dynamic group.

Example matching rule: <dynamic_group>.{resource.type = 'fnfunc', resource.compartment.id = '<compartment_ocid>'}, where <compartment_ocid> is the Oracle Cloud ID of the compartment that contains the functions.

Typical Workflow for Transforming Data Using OCI Functions

Follow these high-level tasks in the order listed below to transform data in Oracle Analytics using OCI functions.

Task	Description	More Information
Create functions in OCI (requires functions developer privileges)	In your OCI tenancy, create your functions, and make sure that they comply with the prerequisites specified for use with Oracle Analytics.	About Configuring OCI Functions to use in Oracle Analytics
Assign OCI policies	Make sure that you have the required OCI policies for connecting using API key or resource principal.	Policies Required to Integrate OCI Functions with Oracle Analytics
Connect Oracle Analytics to your OCI tenancy (requires admin or DV author privileges)	In Oracle Analytics, create a connection to your OCI tenancy.	Create a Connection to Your Oracle Cloud Infrastructure Tenancy
Register OCI functions in Oracle Analytics (requires admin or DV author privileges)	In Oracle Analytics, register your OCI functions so that you can call them from data flows.	Register OCI Functions in Oracle Analytics
Transform your data using the OCI functions (requires admin or DV author privileges)	Create a data flow, and use the Apply Custom Script step to invoke an OCI function.	Transform Data Using OCI Functions

Register OCI Functions in Oracle Analytics

Register OCI functions in Oracle Analytics so that you can use them in data flows to transform data. For example, you might register a language conversion function so that data analysts can convert English text into Spanish or German.

In Oracle Analytics, on the home page, click Page Menu, then Register Model/Function, then OCI Functions.
At the Register a Custom Function dialog, select a connection to the OCI tenancy where your functions are located.
At the Select Application dialog, select the application that contains your OCI functions.
If you're not sure, ask the person who created the functions in OCI.
At the Select a Function dialog, select a function, then click Register.
If the function you want to use is grayed out, ask the administrator to make sure that it's configured for Oracle Analytics. See About Configuring OCI Functions to use in Oracle Analytics.

You can use registered OCI functions in data flows to transform data. To verify which functions are registered, you can view registered functions on the Scripts tab of the Machine Learning page (from the Home page, click Navigator, then Machine Learning, then Scripts).