Log-Based Incremental Ingestion

Prerequisite

Ensure that transactional logs are enabled on the source database.

Overview

Log-based incremental ingestion allows ingesting data with lesser load on the source databases, as compared to other ingestion methods. The incremental data is read from the log statements (Data Manipulation Language - DMLs) stored in database log files (archival log in Oracle). Tables in the log-based incremental ingestion require natural keys to be configured like query-based CDC. As incremental data is read from logs, deletion of records is handled from source tables. Infoworks DataFoundry supports Oracle, MSSQL, Sybase IQ and DB2 log-based CDC.