6 Integrate Third-Party Data

Integrate data from other sources with your NetSuite data. Through this integration, you can create KPIs, metrics, cards, and dashboards using data from multiple sources.

Topics:

About Data Pipeline Options to Extract and Load

Use the information in this section to assess the data pipeline options to extract and load data.

Pipeline Advantages Limitations License and Fees
Data Flow and Data Preparation
  • Easy to use. Drag and drop data to create and schedule data flows.
  • No SQL code needed.
  • Select from a long list of connectors.
  • Intended for repetitive data requests.
  • Supports all standard databases.
  • Can support only a few gigabytes of data. Limited to 1.1 million rows.
  • Incremental: Only based on one specific field.
 Embedded in Oracle Analytics Cloud.
Oracle Cloud Infrastructure Data Integration Supports large volumes of data. Need to know how to browse data in Data Integration. Additional license and fees required.
Existing ETL tool, if any Ease of use, comfort and convenience. General limitations of connectors. Owned by customers.

About Data Mashup and Transformation Options

Use the information in this section to assess the options to mash up and transform your data.

Mash Up the Data Layer Advantages Limitations
Extensibility Frameworks
  • Robust solutions at semantic layer.
  • Best performance. Uses pre-calculated data.
  • Need to be an expert user of the Oracle BI Server Administration tool.
  • Need to understand the existing metadata repository (RPD) file.
  • Must make minor changes to the metadata repository.
Data flow to collate NetSuite repository data and third-party data
  • Can access the raw data in tables to create simple to complex calculated KPIs.
  • Provides ample scope for custom analytics.
  • Easy to extract and load data using drag and drop in data flow.
  • Need to know about the tables in Oracle Autonomous Data Warehouse.
Data flow to collate NetSuite factory schema data and third-party data Easy to extract and load data using drag and drop in data flow.
  • Need to understand Oracle Autonomous Data Warehouse and the metadata repository.
  • Granularity limited to the data that’s available.

Outline the Scope and Prepare the Warehouse for the Data

This section outlines the tasks you perform to prepare the autonomous data warehouse to receive the data.

Description of nsawa_integrate_third-party_data_figure1.png follows
Description of the illustration nsawa_integrate_third-party_data_figure1.png
You must ensure that the warehouse receives the right data in the right format by completing these tasks:
  • Identify the data sources.
  • Connect to the warehouse.
  • Create the schema.
  • Create the tables.
  1. Identify and enlist the data source tables, fields, and rows to extract from the third-party source systems.
  2. Examine the various approaches to connect to the warehouse to perform the data definition language (DDL) or data manipulation language (DML) operations and use the approach that suits your requirements.
  3. After you connect to the warehouse, create a schema or database to absorb the data. If data is coming from multiple sources, decide whether you want to create multiple schemas or databases for different third-party sources or one schema or database for all third-party sources. See Provisioning Autonomous Data Warehouse Cloud
  4. Create destination tables to store data from the incoming third-party sources.

Load Data into the Warehouse

Perform these tasks to load data into Oracle Autonomous Data Warehouse.

  1. Consider these factors when selecting the pipeline tool for populating the warehouse with data:
    1. The data volume for the initial load.
    2. The data volume for incremental loads.
    3. The frequency of incremental loads.
    4. The complexity level of required transformations.
    5. The ETL tool in place, if any.
  2. Based on the factors you considered while selecting the pipeline tool, evaluate these three options to select the tool:
    • Data Flow: This is a utility within the visualization area of Oracle NetSuite Analytics Warehouse that you can use to move data from various sources into the tables in the destination schema of Oracle Autonomous Data Warehouse. See Curate Your Data Using Data Flows.
    • Data Integration on Oracle Cloud Infrastructure: Data Integration is a fully managed, server-less, and native cloud service that helps you with common extract, load, and transform (ETL) tasks such as ingesting data from a variety of sources, cleansing, transforming, and reshaping your data, and efficiently loading it to target data sources on Oracle Cloud Infrastructure. See Data Integration.
    • Custom ETL: You can use any third-party ETL tool to move data into Oracle Autonomous Data Warehouse.
  3. When the pipeline tool is ready, use it to load the data into Oracle Autonomous Data Warehouse.

Mash Up NetSuite Data with Third-Party Data

You might want to mash up data from multiple sources.

For instance, you might take web analytics data from warehouses of customer data, extract customer information from the Dynamix CRM, standardize and cleanse data, and then mash it up with NetSuite data. When data is in the autonomous data warehouse, you can select the appropriate option for mashing up the data.Description of nsawa_integrate_third-party_data_figure3.png follows
Description of the illustration nsawa_integrate_third-party_data_figure3.png
  • Option 1: Extensibility Framework to Extend Models and Content

    You can use the Extensibility framework to extend models and content with data from other sources to your team without having to worry about upgrades. You must understand a star schema and know how to use or modify the metadata repository (RPD) file. See Customize Oracle NetSuite Analytics Warehouse and Extensibility Reference Architecture to get you started.

  • Option 2: Data Flow to Mash Up Data in Oracle Autonomous Data Warehouse

    You can use this option to leverage third-party tables in Oracle Autonomous Data Warehouse (imported using one of the pipeline approaches) and NetSuite data loaded using an Oracle-managed pipeline. You can use a data flow to form a union, join, or filter, and transform data from the NetSuite schema and the custom schema that contains imported third-party data. See Curate Your Data Using Data Flows.

Create a Card or Dashboard

When you have the third-party data in Oracle Autonomous Data Warehouse and mashed-up with NetSuite data, you can create KPIs, KPI cards, decks, and dashboards, as described in the following topics: