The prerequisites for a pipeline are:

• A connection is required to create a stream, except for a file stream.

• A stream is required to create a pipeline.

Configure the Coherence Cluster

Make sure that you have Coherence for Java is installed.

To configure the external cluster as client:

1. Create an XML file named cache-config.xml.

2. Copy the following XML to the file:

   <?xml version="1.0"?>
   
   <cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
      xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config
      coherence-cache-config.xsd">
   		<caching-scheme-mapping>
   			<cache-mapping>
   				<cache-name>
   					externalcache*
   				</cache-name>
   				<schema-name>
   					remote
   				</schema-name>
   			</cahce-mapping>
   		</caching-scheme-mapping>
   
   		<caching-schemes>
   			<remote-cache-scheme>
   				<scheme-name>
   					remote
   				</scheme-name>
   				<service-name>
   					ExtendTcpCacheService
   				</service-name>
   				<initiator-config>
   					<tcp-initiator>
   						<remote-addresses>
   							<socket-address>
   								<address>localhost	</address>
   								<port>9099</port>
   							</socket-address>
   						</remote-addresses>
   					</tcp-initiator>
   					<outgoing-message-handler>
   						<request-timeout>5s</request-timeout>
   					</outgoing-message-handler>
   				</initiator-config>
   			</remote-cache-scheme>
   
   			<remote-invocation-scheme>
   				<scheme-name>extend-invocation</scheme-name>
   				<service-name>ExtendTcpInvocationService</service-name>
   				<initiator-config>
   					<tcp-initiator>
   						<remote-addresses>
   							<socket-address>
   								<address>localhost</address>
   								<port>9099</port>
   							</socket-address>
   						</remote-addresses>
   					</tcp-initiator>
   					<outgoing-message-handler>
   						<request-timeout>5s</request-timeout>
   					</outgoing-message-handler>
   				</initiator-config>
   			</remote-invocation-scheme>
   		</caching-schemes>
   </cache-config>

3. Save and close the file.

4. Test the connection to the cluster.

   InvocationService service = (InvocationService) CacheFactory.getConfigurableCacheFactory().ensureService("ExtendTcpInvocationService");

   ensureService() will throw exception if there is no coherence cluster available with the given host and port.

5. Create a coherence reference using a coherence connection.

6. Register the coherence as reference.

To create a stream:

1. Navigate to Catalog.

2. Select Stream in the Create New Item menu.

3. Provide details for the following fields on the Type Properties page and click Next:

   • Name — name of the stream

   • Description — description of the stream

   • Tags — tags you want to use for the stream

   • Stream Type — select suitable stream type. Supported types are File, GoldenGate, JMS, and Kafka.

   
   Description of the illustration create_stream_type.png

4. Provide details for the following fields on the Source Details page and click Next:

   When the stream type is File:

   • File Path or URL — the location of the file that you want to upload

   • Read whole content — select this option if you want to read the whole content of the file

   • Number of events per batch — the number of events that you want to process per batch

   • Loop — select this option if you want to process the file in a loop

   • Data Format — the format of the data. The supported types are: CSV and JSON.

   When the stream type is GoldenGate:

   • Connection — the connection for the stream

   • Topic name — the topic name that receives events you want to analyze

   • Data Format — the format of the data. The supported types are: CSV, JSON, AVRO. AVRO is a data serialization system.

   When the stream type is JMS:

   • Connection — the connection for the stream

   • Jndi name — the Jndi that reads messages from topics, distributed topics, queues and distributed queues

   • Client ID — the client to be used for durable subscriber

   • Message Selector — the message selector to filter messages. If your messaging application needs to filter the messages it receives, you can use a JMS API message selector, which allows a message consumer to specify the messages it is interested in. Message selectors assign the work of filtering messages to the JMS provider rather than to the application.

     A message selector is a String that contains an expression. The syntax of the expression is based on a subset of the SQL92 conditional expression syntax. The message selector in the following example selects any message that has a NewsType property that is set to the value 'Sports' or 'Opinion':

     NewsType = ’Sports’ OR NewsType = ’Opinion’

     The createConsumer and createDurableSubscriber methods allow you to specify a message selector as an argument when you create a message consumer.

   • Subscription ID — the subscription id for durable selector

   • Data Format — the format of the data. The supported types are: CSV, JSON, AVRO, MapMessage. MapMessage is supported only for JNDI based streams.

     A MapMessage object is used to send a set of name-value pairs. The names are String objects, and the values are primitive data types in the Java programming language. The names must have a value that is not null, and not an empty string. The entries can be accessed sequentially or randomly by name. The order of the entries is undefined.

   When the stream type is Kafka:

   • Connection — the connection for the stream

   • Topic name — the topic name that receives events you want to analyze

   • Data Format — the format of the data within the stream. The supported types are: CSV, JSON, AVRO.

   
   Description of the illustration create_stream_source.png

5. Select one of the mechanisms to define the shape on the Shape page:

   • Infer Shape — detects the shape automatically from the input data stream.

     You can infer the shape from Kafka, JSON schema file, or CSV message/data file. You can also save the auto detected shape and use it later.

   • Select Existing Shape — lets you choose one of the existing shapes from the drop-down list.

   • Manual Shape — populates the existing fields and also allows you to add or remove columns from the shape. You can also update the datatype of the fields.

   
   Description of the illustration create_stream_shape.png

A stream is created with specified details.

CSV Data for Pre-defined Formats

When your data format is CSV, select a predefined format based on the variations of CSV data that differs due to the originating source of these CSV. The following table describes the CSV data for each of these predefined formats:

Capabilities of JMS Source

The capabilities of JMS Source are listed in the following table:

JMS Server Clean Up

When you create a JMS stream and select the durable subscription option (by providing client ID and subscription ID value), Oracle Stream Analytics creates the durable subscription (if not already present) when the pipeline using this stream is running. When you come out of the pipeline or unpublish the pipeline(or kill the running pipeline), the durable subscription remains on the JMS Server. It is advisable to delete the durable subscription from the JMS Server and clean up the resources, if you do not intend to publish this pipeline anymore.

A database reference is a reference to specified table in the database. With cache enabled for database reference, when the values gets pulled from database, they are maintained in coherence cache for reference from where they can be served on next request. A database reference requires a database connection.

A coherence reference can be any external cache defined in coherence cluster that can have data from an external system.

To create a reference:

1. Navigate to Catalog.

2. Select Reference in the Create New Item menu.

3. Provide details for the following fields on the Type Properties page and click Next:

   • Name — name of the reference

   • Description — description of the reference

   • Tags — tags you want to use for the reference

   • Reference Type — the reference type of the reference. The supported reference types are: Coherence and Database.

     
     Description of the illustration create_reference_type.png

4. Provide details for the following fields on the Source Details page and click Next:

   When the reference type is Coherence, enter or select appropriate values for:

   • Connection — the connection for the coherence reference

     
     Description of the illustration create_reference_source.png

   • Cache name — the name of the cache to enable caching for better performance at the cost of higher memory usage of the Spark applications. Caching is supported only for single equality join condition. When you update the cache, the application will receive the updated data very quickly.

   Coherence reference has data in key-value pairs. Key is object type and value is Map<String,Object>. Map<String,Object> is map of attribute names and values, attributes list should match with external event type. In this release, only external schema for key and value s supported.

   When the reference type is Database Table, enter or select appropriate values for:

   • Connection — the connection for the database reference

   • Enable Caching — select this option if you want to enable caching

   • Expiry Delay — the amount of time from last update that entries will be kept by the cache before being marked as expired. Any attempt to read an expired entry will result in a reloading of the entry from the configured cache store. This field is enabled only when caching is enabled.

5. Provide details for the following fields on the Shape page and click Save:

   When the reference type is Coherence:

   • Select Existing Shape — select a shape that you want to use for the reference

     Remember:

     Ensure that you do not use any of the CQL reserved words as the column names. If you use the reserved keywords, you cannot deploy the pipeline.

   • Manual Shape — select this option if you want to define your own shape

   Note:

   When you load coherence data, ensure that you include precision and scale for number type. Only when these values are specified, the join works. For example,

   NamedCache cache  = CacheFactory.getCache("externalcachetimestamp");
   
           java.math.BigDecimal big10 = new java.math.BigDecimal("10",new
   MathContext(58)).setScale(56, RoundingMode.HALF_UP);
   
           Map<String,Object> order1 = new HashMap<String, Object>();
   order1.put("strValue", "Test");
   order1.put("intervalValue", "+000000002 03:04:11.330000000");
           order1.put("orderTag", big10);
   
           cache.put(big10,order1);

   When the reference type is Database Table:

   • Shape Name — select a shape that you want to use for the reference

When the datatype of the table data is not supported, the table columns do not have auto generated datatype. Only the following datatypes are supported:

• numeric

• interval day to second

• text

• timestamp (without timezone)

• date time (without timezone)

  Note:

  The date column cannot be mapped to timestamp. This is a limitation in the current release.

A reference is created with the specified details.

Limitations of Coherence as Reference

With coherence as reference, there are a few limitations:

• You cannot test the connection

• You need to specify the cache name manually

• Only equal operator is allowed while establishing a correlation with coherence reference

• You must use manual shape

Dashboard is an analytics feature. You can create dashboards in Oracle Stream Analytics to have a quick view at the metrics.

After you have created the dashboard, it is just an empty dashboard. You need to start adding visualizations to the dashboard.

Editing a Dashboard

To edit a dashboard:

1. Click the required dashboard in the catalog.

   The dashboard opens in the dashboard editor.

   
   Description of the illustration edit_dashboard.png

2. Click the Add a new visualization icon to see a list of existing visualizations. Visualizations from the pipelines and as well as from the cube explorations appear here. Go through the list, select one or more visualizations and add them to the dashboard.

3. Click the Specify refresh interval icon to select the refresh frequency for the dashboard. This is applicable only for cube based visualizations not applicable for streaming charts created out of pipeline.

   This just a client side setting and is not persisted with the Superset Version 0.17.0.

4. Click the Apply CSS to the dashboard icon to select a CSS. You can also edit the CSS in the live editor.

   You can also see the active filter applied to the dashboard by clicking the Active dashboard filters icon. You can save the link to the dashboard or email the link to someone using the Copy the link to the clipboard and Email the link icons respectively.

5. Click the Save icon to save the changes you have made to the dashboard.

6. Hover over the added visualization, click the Explore chart icon to open the chart editor of the visualization.

   

   
   Description of the illustration explore_chart.png

   You can see the metadata of the visualization. You can also move the chart around the canvas, refresh it, or remove it from the dashboard.

   A cube exploration looks like the following:

   
   Description of the illustration cube_exploration.png

   The various options like time granularity, group by, table timestamp format, row limit, filters, and result filters add more granularity and details to the dashboard.

7. Click Save as to make the following changes to the dashboard:

   • Overwrite the visualization

   • Overwrite the current visualization with a different name

   • Add the visualization to an existing dashboard

   • Add the visualization to a new dashboard

To create a cube:

When you create druid based cube, you can explore data in it.

To explore a cube:

To create a target:

1. Navigate to Catalog.

2. Select Target in the Create New Item menu.

3. Provide details for the following fields on the Type Properties page and click Save and Next:

   • Name — name of the target

   • Description — description of the target

   • Tags — tags you want to use for the target

   • Target Type — the transport type of the target. Supported types are JMS, Kafka and Rest. The target is a sink for the output event. Each type of target is a different sink system and therefore different configuration parameters are required for different types.

     
     Description of the illustration create_target_type.png

4. Provide details for the following fields on the Target Details page and click Next:

   When the target type is JMS:

   • Connection — the connection for the target

   • Jndi name — the topic or queue name defined in Jndi to be used in the target

     
     Description of the illustration create_target_details.png

   • Data Format — select a suitable data format. This is a mandatory field. The supported data format types are: CSV and JSON.

   When the target type is Kafka:

   • Connection — the connection for the target

   • Topic Name — the Kafka topic to be used in the target

   • Data Format — select a suitable data format. This is a mandatory field. The supported data format types are: CSV and JSON.

   When the target type is REST:

   • URL — enter the REST service URL. This is a mandatory field.

   • Custom HTTP headers — set the custom headers for HTTP. This is an optional field.

   • Batch processing — select this option to send events in batches and not one by one. Enable this option for high throughput pipelines. This is an optional field.

   • Data Format — select a suitable data format. This is a mandatory field.

   Click Test connection to check if the connection has been established successfully.

   Testing REST targets is a heuristic process. It uses proxy settings. The testing process uses GET request to ping the given URL and returns success if the server returns OK (status code 200). The return content is of the type of application/json.

5. Provide details for the following fields on the Data Format page and click Next:

   When the data format type is CSV:

   • CSV Predefined Format — select a predefined CSV format. This supported formats are: Excel, InfomixUnload, InfomixUnloadCsv, MySQL, PostgreSQLCsv, PostgreSQLText.

   • Create the header row — select this option if you want to create a header row in the target.

   When the data format type is JSON:

   • Create nested json object — select this option if you want a nested json object to be created for the target

     
     Description of the illustration create_target_dataformat.png

6. Select one of the mechanisms to define the shape on the Shape page and click Save:

   • Select Existing Shape lets you choose one of the existing shapes from the drop-down list.

   • Manual Shape populates the existing fields and also allows you to add or remove columns from the shape. You can also update the datatype of the fields.

     
     Description of the illustration create_target_shape.png

A target is created with specified details.

Creating Target from Pipeline Editor

Alternatively, you can also create a target from the pipeline editor. When you click Create in the target stage, you are navigated to the Create Target dialog box. Provide all the required details and complete the target creation process. When you create a target from the pipeline editor, the shape gets pre-populated with the shape from the last stage.

Create a Manual Geo Fence

To create a manual geo fence:

1. Navigate to the Catalog page.

2. Click Create New Item and select Geo Fence from the drop-down list.

   The Create Geo Fence dialog opens.

3. Enter a suitable name for the Geo Fence.

4. Select Manually Created Geo Fence as the Type.

5. Click Save.

   The Geo Fence Editor opens. In this editor you can create the geo fence according to your requirement.

6. Within the Geo Fence Editor, Zoom In or Zoom Out to navigate to the required area using the zoom icons in the toolbar located on the top-left side of the screen.

   You can also use the Marquee Zoom tool to zoom a specific area on the map. You can mark an area using the marquee zoom and that area in map is zoomed.

7. Click the Polygon Tool and mark the area around a region to create a geo fence.

   
   Description of the illustration create_geo_fence.png

8. Enter a name and description, and click Save to save your changes.

Update a Manual Geo Fence

To update a manual geo fence:

1. Navigate to the Catalog page.

2. Click the name of the geo fence you want to update.

   The Geo Fence Editor opens. You can edit/update the geo fence here.

Search Within a Manual Geo Fence

You can search the geo fence based on the country and a region or address. The search field allows you search within the available list of countries. When you click the search results tile in the left center of the geo fence and select any result, you are automatically zoomed in to that specific area.

Delete a Manual Geo Fence

To delete a manual geo fence:

1. Navigate to Catalog page.

2. Click Actions, then select Delete Item to delete the selected geo fence.

Create a Database-based Geo Fence

To create a database-based geo fence:

1. Navigate to Catalog page.

2. Click Create New Item and then select Geo Fence from the drop-down list.

   The Create Geo Fence dialog opens.

3. Enter a suitable name for the geo fence.

4. Select Geo Fence from Database as the Type.

5. Click Next and select Connection.

6. Click Next.

   All tables that have the field type as SDO_GEOMETRY appear in the drop-down list.

7. Select the required table to define the shape.

8. Click Save.

Delete a Database-based Geo Fence

To delete a database-based geo fence:

1. Navigate to Catalog page.

2. Click Actions and then select Delete Item to delete the selected geo fence.

Display the Map Using Tile Layers

Tile layer is the base map that provides immediate geographic context. Tiles are stored in the map tile server. <ph ishcondition="Product_Family=Cloud" varref="streaming">Stream Analytics</ph><ph ishcondition="Product_Family=OnPremise" varref="osa">Oracle Stream Analytics</ph> supports two types of tile layers. Open Street Maps tile layer is a free map. And, Elocation tile layer is an Oracle tile layer. These tile layers contains huge amount of data pertaining to:

• Roads, railways, waterways, etc.

• Restaurants, shops, stations, ATMs, and more

• Walking and cycling paths

• Buildings, campuses, etc.

You can choose if you would like to see the map in Elocation tile layer or Open Street Maps tile layer. To set your preference:

1. Click the user name in the top right corner of the screen.

2. Click Preferences. The Preferences page opens.

3. Click Map.

4. Under Tile Layer, choose Open Street Maps Tile Layer option from the drop-down list.

   
   Description of the illustration tilelayer_1.png

5. Click Save. The map looks like this:

   
   Description of the illustration tilelayer_2.png

6. To display the map in Elocation tile layer, follow steps 1 to 3.

7. From the Tile Layer drop-down list, choose Elocation Tile Layer.

8. Click Save. The map looks like this:

   
   Description of the illustration tilelayer_4.png

Description of the illustration predictive_model.png

To create a pipeline:

1. Navigate to Catalog.

2. Select Pipeline in the Create New Item menu.

3. Provide details for the following fields and click Save:

   • Name — name of the pipeline

   • Description — description of the pipeline

   • Tags — tags you want to use for the pipeline

   • Stream — the stream you want to use for the pipeline

     
     Description of the illustration create_application.png

A pipeline is created with specified details.

A published pipeline will continue to run on your Spark cluster after you exit the Pipeline Editor, unlike the draft pipelines which are undeployed to release resources.

To publish a pipeline:

The topology viewer helps you in identifying the dependencies that a selected entity has on other entities. Understanding the dependencies helps you in being cautious while deleting or undeploying an entity. Oracle Stream Analytics supports two contexts for the topology — Immediate Family and Extended Family.

You can launch the Topology viewer in any of the following ways:

• Select Show topology from the Catalog Actions menu to launch the Topology Viewer for the selected entity.

  
  Description of the illustration show_topology_catalog_actions_menu.png

• Click the Show Topology icon in the Pipeline Editor.

  
  Description of the illustration show_topology_exploration_editor.png

Click the Show Topology icon at the top-right corner of the editor to open the topology viewer.By default, the topology of the entity from which you launch the Topology Viewer is displayed. The context of this topology is Immediate Family, which indicates that only the immediate dependencies and connections between the entity and other entities are shown. You can switch the context of the topology to display the full topology of the entity from which you have launched the Topology Viewer. The topology in an Extended Family context displays all the dependencies and connections in the topology in a hierarchical manner.

Immediate Family

Immediate Family context displays the dependencies between the selected entity and its child or parent.

The following figure illustrates how a topology looks in the Immediate Family.

Description of the illustration topology_viewer_immediate.png

Extended Family

Extended Family context displays the dependencies between the entities in a full context, that is if an entity has a child entity and a parent entity, and the parent entity has other dependencies, all the dependencies are shown in the Full context.

The following figure illustrates how a topology looks in the Extended Family.

Description of the illustration topology_viewer_full.png