About pipelines

Accelerate your data replication solutions with OCI GoldenGate Pipelines.

OCI GoldenGate pipelines let you configure real time data replication flows that support mission critical use cases such as analytics, operational reporting, and hybrid cloud integrations. You can create a pipeline to stream data directly from operational databases to analytical environments without traditional extract, transform, load (ETL) processes. For example, you can set up a pipeline to directly replicate data from Autonomous AI Transaction Processing to Autonomous AI Lakehouse with minimal configuration.

Note:

Because pipelines are powered by GoldenGate deployments, you may experience downtime at times when a deployment upgrade occurs.

About ZeroETL pipelines

ZeroETL is an Extract, Transform, and Load (ETL) design pattern for continuous, metadata driven data movement. ZeroETL Mirror pipelines stream database changes with transactional awareness and applies them directly to a target database.

Benefits to using ZeroETL Mirror pipelines include:

  • Fewer pipelines to build and maintain: Replace hand-coded ETL processes and brittle schedules with a managed pipelines powered by OCI GoldenGate.
  • Faster delivery of analytics and applications: Near real-time mirrors keep targets in sync for dashboards, AI/ML features, and microservices.
  • Better data quality and consistency: Transactionally aware replication reduces drift and downstream reprocessing.
  • Lower operational risk: Managed orchestration, monitoring, and error handling reduce on-call toil.
  • Standards, security, and governance: Centralized connections, credentials management, and auditability align with enterprise controls.

Examples that you can use ZeroETL Mirror pipelines for include:

  • Real-time analytics and business intelligence: Mirror operational data into analytics databases, such as Autonomous AI Lakehouse, for fresh dashboards without nightly ETL.
  • Artificial Intelligence (AI)/Machine Learning (ML) feature stores and Retrieval Augmented Generation (RAG): Keep features synchronized across Autonomous AI Databases or Oracle AI Databases to power low-latency inference.
  • Multicloud application data fabric: Keep data consistent across regions and clouds to support distributed applications and regional data access patterns.
  • Modernization and migration: Stand up a synchronized target during migration to reduce cutover risk and downtime.

Create pipeline resources to get started.

Pipeline concepts

The following concepts are essential for working with pipelines:

  • Pipeline: a predefined component that enables real time, minimal latency data movement from a source operational database to a target lakehouse or analytics platform without the need for traditional ETL steps.
  • Pipeline type: A common data replication use case.
  • Capture: A process that runs against the source connection and extracts, or captures, data. Also known as an Extract process.
  • Apply: A process that delivers data to a target database or technology, otherwise known as the Replicat process.

Supported connection types for Pipelines

OCI GoldenGate Pipelines currently support the following Oracle Databases:

You can also create a connection in the Create pipeline workflow.