OGG-Based Incremental Ingestion
Hybrid Source Synchronization for Oracle Databases
Infoworks DataFoundry supports the following mechanisms to synchronize data from Oracle using Change Data Capture:
- Log-based synchronization using Oracle DB logs
- Query-based synchronization using timestamps or BatchIDs
- Hybrid Source synchronization using third party replication tools such as Golden Gate.
Hybrid Source synchronization operates by directly loading the source table(s) via Infoworks DataFoundry initial bulk loading. Once the full table is ingested using Infoworks DataFoundry, you can configure Hybrid synchronization to read the CDC records from an Oracle table maintained by tools such as Oracle Golden Gate.
Configuring a Table for OGG
Open the table configuration page for which the table must be configured for OGG and perform the following:
- Select Ingest Type as Ogg-based Incremental Load.
- Select the Natural Key for the table.
- Select split-by key if any.
- In the Synchronization Configuration section, enter the archival server details like JDBC URL, username, password and schema name. Captured_table_name is the name of the table in the target server to which the source table is mapped.
The other configurations include storage format, partition table on, no.of secondary partitions, no.of reducers and number of merge jobs to run in parallel which are same as any normal ingestion job.
Advanced Configurations at table Level
The Table Configurations page allows you to configure any configurations related to the table. If the configuration is not found at the table level, Infoworks obtains it from the source Advanced Configuration or at global level.
Fetching Metadata
- OGG_REFRESH_COL: The name of the column in the target table through which the incremental records will be fetched.
- DML_COL: The name of the column in the target table to identify if an incremental record is an INSERT, UPDATE or DELETE. DML_COLvalues supported by Infoworks are INSERT, SQLCOMP UPDATE, DELETE.
Running Bulk Initial Load
Once all the configurations are done for the ogg-table, add the table in a Table Group and run Initialize and Ingest Now for loading the history data from the source table.
Running Incremental Load Job
Run Ingest now to fetch the change data from the golden gate target table in the archival server. Infoworks reduces multiple records for the same natural key to single record and place all the history transactions for the records in the history table, if history view is enabled.