Cassandra provides a number of column data types and most of these data types are supported by the Cassandra Handler.

Supported Cassandra Data Types

ASCII
BIGINT
BLOB
BOOLEAN
DATE
DECIMAL
DOUBLE
DURATION
FLOAT
INET
INT
SMALLINT
TEXT
TIME
TIMESTAMP
TIMEUUID
TINYINT
UUID
VARCHAR
VARINT

Unsupported Cassandra Data Types

COUNTER
MAP
SET
LIST
UDT (user defined type)
TUPLE
CUSTOM_TYPE

Supported Database Operations

INSERT
UPDATE (captured as INSERT)
DELETE

The Cassandra commit log files do not record any before images for the UPDATE or DELETE operations. So the captured operations never have a before image section. Oracle GoldenGate features that rely on before image records, such as Conflict Detection and Resolution, are not available.

Unsupported Database Operations

TRUNCATE
DDL (CREATE, ALTER, DROP)

The data type of the column value in the source trail file must be converted to the corresponding Java type representing the Cassandra column type in the Cassandra Handler. This data conversion introduces the risk of a runtime conversion error. A poorly mapped field (such as varchar as the source containing alpha numeric data to a Cassandra int) may cause a runtime error and cause the Cassandra Handler to abend. You can view the Cassandra Java type mappings at:

DataStax Documentation

It is possible that the data may require specialized processing to get converted to the corresponding Java type for intake into Cassandra. If this is the case, you have two options:

• Try to use the general regular expression search and replace functionality to format the source column value data in a way that can be converted into the Java data type for use in Cassandra.

  Or

• Implement or extend the default data type conversion logic to override it with custom logic for your use case. Contact Oracle Support for guidance.

It is important to understand how data maps from the metadata definition in the source trail file are mapped to the corresponding keyspace and table in Cassandra. Source tables are generally either two-part names defined as schema.table,or three-part names defined as catalog.schema.table.

The following table explains how catalog, schema, and table names map into Cassandra. Unless you use special syntax, Cassandra converts all keyspace, table names, and column names to lower case.

The Cassandra Handler pushes operations to Cassandra using either the asynchronous or synchronous API. In asynchronous mode, operations are flushed at transaction commit (grouped transaction commit using GROUPTRANSOPS) to ensure write durability. The Cassandra Handler does not interface with Cassandra in a transactional way.

Supported Database Operations

INSERT
UPDATE (captured as INSERT)
DELETE

The Cassandra commit log files do not record any before images for the UPDATE or DELETE operations. So the captured operations never have a before image section. Oracle GoldenGate features that rely on before image records, such as Conflict Detection and Resolution, are not available.

Unsupported Database Operations

TRUNCATE
DDL (CREATE, ALTER, DROP)

Insert, update, and delete operations are processed differently in Cassandra than a traditional RDBMS. The following explains how insert, update, and delete operations are interpreted by Cassandra:

• Inserts: If the row does not exist in Cassandra, then an insert operation is processed as an insert. If the row already exists in Cassandra, then an insert operation is processed as an update.

• Updates: If a row does not exist in Cassandra, then an update operation is processed as an insert. If the row already exists in Cassandra, then an update operation is processed as insert.

• Delete:If the row does not exist in Cassandra, then a delete operation has no effect. If the row exists in Cassandra, then a delete operation is processed as a delete.

The state of the data in Cassandra is idempotent. You can replay the source trail files or replay sections of the trail files. The state of the Cassandra database must be the same regardless of the number of times that the trail data is written into Cassandra.

Oracle GoldenGate allows you to control the data that is propagated to the source trail file in the event of an update. The data for an update in the source trail file is either a compressed or a full image of the update, and the column information is provided as follows:

Compressed

For the primary keys and the columns for which the value changed. Data for columns that have not changed is not provided in the trail file.

Full Image

For all columns, including primary keys, columns for which the value has changed, and columns for which the value has not changed.

The amount of information about an update is important to the Cassandra Handler. If the source trail file contains full images of the change data, then the Cassandra Handler can use prepared statements to perform row updates in Cassandra. Full images also allow the Cassandra Handler to perform primary key updates for a row in Cassandra. In Cassandra, primary keys are immutable, so an update that changes a primary key must be treated as a delete and an insert. Conversely, when compressed updates are used, prepared statements cannot be used for Cassandra row updates. Simple statements identifying the changing values and primary keys must be dynamically created and then executed. With compressed updates, primary key updates are not possible and as a result, the Cassandra Handler will abend.

You must set the control properties gg.handler.name.compressedUpdates and gg.handler.name.compressedUpdatesfor so that the handler expects either compressed or full image updates.

The default value, true, sets the Cassandra Handler to expect compressed updates. Prepared statements are not be used for updates, and primary key updates cause the handler to abend.

When the value is false, prepared statements are used for updates and primary key updates can be processed. A source trail file that does not contain full image data can lead to corrupted data columns, which are considered null. As a result, the null value is pushed to Cassandra. If you are not sure about whether the source trail files contains compressed or full image data, set gg.handler.name.compressedUpdates to true.

CLOB and BLOB data types do not propagate LOB data in updates unless the LOB column value changed. Therefore, if the source tables contain LOB data, set gg.handler.name.compressedUpdates to true.

Primary key values for a row in Cassandra are immutable. An update operation that changes any primary key value for a Cassandra row must be treated as a delete and insert. The Cassandra Handler can process update operations that result in the change of a primary key in Cassandra only as a delete and insert. To successfully process this operation, the source trail file must contain the complete before and after change data images for all columns. The gg.handler.name.compressed configuration property of the Cassandra Handler must be set to false for primary key updates to be successfully processed.

The following are the configurable values for the Cassandra Handler. These properties are located in the Java Adapter properties file (not in the Replicat properties file).

To enable the selection of the Cassandra Handler, you must first configure the handler type by specifying gg.handler.name.type=cassandra and the other Cassandra properties as follows:

The Cassandra Handler connection to the Cassandra Cluster can be secured using user name and password credentials. These are set using the following configuration properties:

gg.handler.name.username 
gg.handler.name.password

To configure SSL, the recommendation is to configure the SSL properties via the Datastax Java Driver configuration file and point to the configuration file via the gg.handler.name.DatastaxJSSEConfigPath property. See https://docs.datastax.com/en/developer/java-driver/4.14/manual/core/ssl/ for the SSL settings instructions.

datastax-java-driver {
  advanced.ssl-engine-factory {
    class = DefaultSslEngineFactory

    # This property is optional. If it is not present, the driver won't explicitly enable cipher
    # suites on the engine, which according to the JDK documentations results in "a minimum quality
    # of service".
    // cipher-suites = [ "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA" ]

    # Whether or not to require validation that the hostname of the server certificate's common
    # name matches the hostname of the server being connected to. If not set, defaults to true.
    // hostname-validation = true

    # The locations and passwords used to access truststore and keystore contents.
    # These properties are optional. If either truststore-path or keystore-path are specified,
    # the driver builds an SSLContext from these files. If neither option is specified, the
    # default SSLContext is used, which is based on system property configuration.
    // truststore-path = /path/to/client.truststore
    // truststore-password = password123
    // keystore-path = /path/to/client.keystore
    // keystore-password = password123
  }
}

The Cassandra Handler first interrogates the target Cassandra database to determine whether the target Cassandra keyspace exists. If the target Cassandra keyspace does not exist, then the Cassandra Handler abends. Keyspaces must be created by the user. The log file must contain the error of the exact keyspace name that the Cassandra Handler is expecting.

Next, the Cassandra Handler interrogates the target Cassandra database for the table definition. If the table does not exist, the Cassandra Handler either creates a table if gg.handler.name.ddlHandling includes the CREATE option or abends the process. A message is logged that shows you the table that does not exist in Cassandra.

If the table exists in Cassandra, then the Cassandra Handler reconciles the table definition from the source trail file and the table definition in Cassandra. This reconciliation process searches for columns that exist in the source table definition and not in the corresponding Cassandra table definition. If it locates columns fitting this criteria and the gg.handler.name.ddlHandling property includes ADD, then the Cassandra Handler adds the columns to the target table in Cassandra. Otherwise, it ignores these columns.

Next, the Cassandra Handler searches for columns that exist in the target Cassandra table but do not exist in the source table definition. If it locates columns that fit this criteria and the gg.handler.name.ddlHandling property includes DROP, then the Cassandra Handler removes these columns from the target table in Cassandra. Otherwise those columns are ignored.

Finally, the prepared statements are built.

You can capture all of the new change data into your Cassandra database, including the DDL changes in the trail, for the target apply. Following is the acceptance criteria:

AC1: Support Cassandra as a bulk extract 
AC2: Support Cassandra as a CDC source
AC4: All Cassandra supported data types are supported
AC5: Should be able to write into different tables based on any filter conditions, like Updates to Update tables or based on primary keys
AC7: Support Parallel processing with multiple threads 
AC8: Support Filtering based on keywords
AC9: Support for Metadata provider
AC10: Support for DDL handling on sources and target
AC11: Support for target creation and updating of metadata.
AC12: Support for error handling and extensive logging
AC13: Support for Conflict Detection and Resolution
AC14: Performance should be on par or better than HBase

When the classpath that is intended to include the required client libraries, a ClassNotFound exception appears in the log file. To troubleshoot, set the Java Adapter logging to DEBUG, and then run the process again. At the debug level, the log contains data about the JARs that were added to the classpath from the gg.classpath configuration variable. The gg.classpath variable selects the asterisk (*) wildcard character to select all JARs in a configured directory. For example, /usr/cassandra/cassandra-java-driver4.9.0/*:/usr/cassandra/cassandra-java-driver-4.9.0/lib/*.

For more information about setting the classpath, see Setting Up and Running the Cassandra Handler and Cassandra Handler Client Dependencies.

When running the Cassandra handler, you may experience a com.datastax.driver.core.exceptions.WriteTimeoutException exception that causes the Replicat process to abend. It is likely to occur under some or all of the following conditions:

• The Cassandra Handler processes large numbers of operations, putting the Cassandra cluster under a significant processing load.

• GROUPTRANSOPS is configured higher than the value of 1000 default.

• The Cassandra Handler is configured in asynchronous mode.

• The Cassandra Handler is configured with a consistency level higher than ONE.

When this problem occurs, the Cassandra Handler is streaming data faster than the Cassandra cluster can process it. The write latency in the Cassandra cluster finally exceeds the write request timeout period, which in turn results in the exception.

The following are potential solutions:

• Increase the write request timeout period. This is controlled with the write_request_timeout_in_ms property in Cassandra and is located in the cassandra.yaml file in the cassandra_install/conf directory. The default is 2000 (2 seconds). You can increase this value to move past the error, and then restart the Cassandra node or nodes for the change to take effect.

• Decrease the GROUPTRANSOPS configuration value of the Replicat process. Typically, decreasing the GROUPTRANSOPS configuration decreases the size of transactions processed and reduces the likelihood that the Cassandra Handler can overtax the Cassandra cluster.

• Reduce the consistency level of the Cassandra Handler. This in turn reduces the amount of work the Cassandra cluster has to complete for an operation to be considered as written.

asm-9.1.jar
asm-analysis-9.1.jar
asm-commons-9.1.jar
asm-tree-9.1.jar
asm-util-9.1.jar
config-1.4.1.jar
esri-geometry-api-1.2.1.jar
HdrHistogram-2.1.12.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-core-asl-1.9.12.jar
jackson-databind-2.12.2.jar
java-driver-core-4.12.0.jar
java-driver-query-builder-4.12.0.jar
java-driver-shaded-guava-25.1-jre-graal-sub-1.jar
jcip-annotations-1.0-1.jar
jffi-1.3.1.jar
jffi-1.3.1-native.jar
jnr-a64asm-1.0.0.jar
jnr-constants-0.10.1.jar
jnr-ffi-2.2.2.jar
jnr-posix-3.1.5.jar
jnr-x86asm-1.0.2.jar
json-20090211.jar
jsr305-3.0.2.jar
metrics-core-4.1.18.jar
native-protocol-1.5.0.jar
netty-buffer-4.1.60.Final.jar
netty-codec-4.1.60.Final.jar
netty-common-4.1.60.Final.jar
netty-handler-4.1.60.Final.jar
netty-resolver-4.1.60.Final.jar
netty-transport-4.1.60.Final.jar
reactive-streams-1.0.3.jar
slf4j-api-1.7.26.jar
spotbugs-annotations-3.1.12.jar

asm-7.1.jar
asm-analysis-7.1.jar
asm-commons-7.1.jar
asm-tree-7.1.jar
asm-util-7.1.jar
commons-collections-3.2.2.jar
commons-configuration-1.10.jar
commons-lang-2.6.jar
commons-lang3-3.8.1.jar
config-1.3.4.jar
esri-geometry-api-1.2.1.jar
gremlin-core-3.4.8.jar
gremlin-shaded-3.4.8.jar
HdrHistogram-2.1.11.jar
jackson-annotations-2.11.0.jar
jackson-core-2.11.0.jar
jackson-core-asl-1.9.12.jar
jackson-databind-2.11.0.jar
java-driver-core-4.9.0.jar
java-driver-query-builder-4.9.0.jar
java-driver-shaded-guava-25.1-jre-graal-sub-1.jar
javapoet-1.8.0.jar
javatuples-1.2.jar
jcip-annotations-1.0-1.jar
jcl-over-slf4j-1.7.25.jar
jffi-1.2.19.jar
jffi-1.2.19-native.jar
jnr-a64asm-1.0.0.jar
jnr-constants-0.9.12.jar
jnr-ffi-2.1.10.jar
jnr-posix-3.0.50.jar
jnr-x86asm-1.0.2.jar
json-20090211.jar
jsr305-3.0.2.jar
metrics-core-4.0.5.jar
native-protocol-1.4.11.jar
netty-buffer-4.1.51.Final.jar
netty-codec-4.1.51.Final.jar
netty-common-4.1.51.Final.jar
netty-handler-4.1.51.Final.jar
netty-resolver-4.1.51.Final.jar
netty-transport-4.1.51.Final.jar
reactive-streams-1.0.2.jar
slf4j-api-1.7.26.jar
spotbugs-annotations-3.1.12.jar
tinkergraph-gremlin-3.4.8.jar