Table of Contents
- Title and Copyright Information
- Preface
- 1 Big Data Integration with Oracle Data Integrator
- 2 Hadoop Data Integration Concepts
-
3
Setting Up the Environment for Integrating Big Data
- 3.1 Configuring Big Data technologies using the Big Data Configurations Wizard
- 3.2 Creating and Initializing the Hadoop Data Server
- 3.3 Creating a Hadoop Physical Schema
- 3.4 Configuring the Oracle Data Integrator Agent to Execute Hadoop Jobs
- 3.5 Configuring Oracle Loader for Hadoop
- 3.6 Configuring Oracle Data Integrator to Connect to a Secure Cluster
- 3.7 Configuring Oracle Data Integrator Studio for Executing Hadoop Jobs on the Local Agent
-
4
Integrating Hadoop Data
- 4.1 Integrating Hadoop Data
- 4.2 Setting Up File Data Sources
- 4.3 Setting Up HDFS Data Sources
- 4.4 Setting Up Hive Data Sources
- 4.5 Setting Up HBase Data Sources
- 4.6 Setting Up Kafka Data Sources
- 4.7 Setting Up Cassandra Data Sources
- 4.8 Importing Hadoop Knowledge Modules
- 4.9 Creating ODI Models and Data Stores to represent Hive, HBase and Cassandra Tables, and HDFS Files
- 4.10 Password Handling in Hadoop
- 4.11 Loading Data from Files into Hive
- 4.12 Loading Data from Hive to Files
- 4.13 Loading Data from HBase into Hive
- 4.14 Loading Data from Hive into HBase
- 4.15 Loading Data from an SQL Database into Hive, HBase, and File using SQOOP
- 4.16 Loading Data from an SQL Database into Hive using SQOOP
- 4.17 Loading Data from an SQL Database into HDFS File using SQOOP
- 4.18 Loading Data from an SQL Database into HBase using SQOOP
- 4.19 Validating and Transforming Data Within Hive
- 4.20 Loading Data into an Oracle Database from Hive and File
- 4.21 Loading Data into an SQL Database from Hbase, Hive, and File using SQOOP
- 4.22 Loading Data from Kafka to Spark Processing Engine
- 5 Executing Oozie Workflows
-
6
Using Query Processing Engines to Generate Code in Different Languages
- 6.1 Query Processing Engines Supported by Oracle Data Integrator
- 6.2 Setting Up Hive Data Server
- 6.3 Creating a Hive Physical Schema
- 6.4 Setting Up Pig Data Server
- 6.5 Creating a Pig Physical Schema
- 6.6 Setting Up Spark Data Server
- 6.7 Creating a Spark Physical Schema
- 6.8 Generating Code in Different Languages
- 7 Working with Spark
- 8 Working with Unstructured Data
- 9 Working with Complex Datatypes and HDFS File Formats
-
A
Hive Knowledge Modules
- A.1 LKM SQL to Hive SQOOP
- A.2 LKM SQL to File SQOOP Direct
- A.3 LKM SQL to HBase SQOOP Direct
- A.4 LKM File to SQL SQOOP
- A.5 LKM Hive to SQL SQOOP
- A.6 LKM HBase to SQL SQOOP
- A.7 LKM HDFS File to Hive Load Data
- A.8 LKM HDFS File to Hive Load Data (Direct)
- A.9 IKM Hive Append
- A.10 IKM Hive Incremental Update
- A.11 LKM File to Hive LOAD DATA
- A.12 LKM File to Hive LOAD DATA Direct
- A.13 LKM HBase to Hive HBASE-SERDE
- A.14 LKM Hive to HBase Incremental Update HBASE-SERDE Direct
- A.15 LKM Hive to File Direct
- A.16 XKM Hive Sort
- A.17 LKM File to Oracle OLH-OSCH
- A.18 LKM File to Oracle OLH-OSCH Direct
- A.19 LKM Hive to Oracle OLH-OSCH
- A.20 LKM Hive to Oracle OLH-OSCH Direct
- A.21 RKM Hive
- A.22 RKM HBase
- A.23 IKM File to Hive (Deprecated)
- A.24 LKM HBase to Hive (HBase-SerDe) [Deprecated]
- A.25 IKM Hive to HBase Incremental Update (HBase-SerDe) [Deprecated]
- A.26 IKM SQL to Hive-HBase-File (SQOOP) [Deprecated]
- A.27 IKM Hive Control Append (Deprecated)
- A.28 CKM Hive
- A.29 IKM Hive Transform (Deprecated)
- A.30 IKM File-Hive to Oracle (OLH-OSCH) [Deprecated]
- A.31 IKM File-Hive to SQL (SQOOP) [Deprecated]
-
B
Pig Knowledge Modules
- B.1 LKM File to Pig
- B.2 LKM Pig to File
- B.3 LKM HBase to Pig
- B.4 LKM Pig to HBase
- B.5 LKM Hive to Pig
- B.6 LKM Pig to Hive
- B.7 LKM SQL to Pig SQOOP
- B.8 XKM Pig Aggregate
- B.9 XKM Pig Distinct
- B.10 XKM Pig Expression
- B.11 XKM Pig Filter
- B.12 XKM Pig Flatten
- B.13 XKM Pig Join
- B.14 XKM Pig Lookup
- B.15 XKM Pig Pivot
- B.16 XKM Pig Set
- B.17 XKM Pig Sort
- B.18 XKM Pig Split
- B.19 XKM Pig Subquery Filter
- B.20 XKM Pig Table Function
- B.21 XKM Pig Unpivot
-
C
Spark Knowledge Modules
- C.1 LKM File to Spark
- C.2 LKM Spark to File
- C.3 LKM Hive to Spark
- C.4 LKM Spark to Hive
- C.5 LKM HDFS to Spark
- C.6 LKM Spark to HDFS
- C.7 LKM Kafka to Spark
- C.8 LKM Spark to Kafka
- C.9 LKM SQL to Spark
- C.10 LKM Spark to SQL
- C.11 LKM Spark to Cassandra
- C.12 RKM Cassandra
- C.13 XKM Spark Aggregate
- C.14 XKM Spark Distinct
- C.15 XKM Spark Expression
- C.16 XKM Spark Filter
- C.17 XKM Spark Input Signature and Output Signature
- C.18 XKM Spark Join
- C.19 XKM Spark Lookup
- C.20 XKM Spark Pivot
- C.21 XKM Spark Set
- C.22 XKM Spark Sort
- C.23 XKM Spark Split
- C.24 XKM Spark Table Function
- C.25 IKM Spark Table Function
- C.26 XKM Spark Unpivot
- D Component Knowledge Modules
- E Considerations, Limitations, and Issues