Teradata Ingestion

The types of synchronization that can be performed on the table.

The options include Full Load, Timestamp-Based Incremental Ingestion, Query Based Incremental Ingestion, Batch ID Based Incremental Ingestion.

The option to crawl large tables by breaking into smaller chunks. The smaller chunks can be crawled in parallel. Select the column to Perform Segmented Load On. To derive a column, check the Use substring from field option and select the Extract function.

NOTE: New columns can be extracted only for date, datetime, timestamp column types. The possible extraction options are year, month, year-month, month-day and day-num-in-month.

After completing the segmented load, ensure to set this value to Completed to submit incremental load or full load jobs.

Source Configuration

The key to identify the row uniquely. This key is used to identify and merge incremental data with the existing data on the target. Distribution of data into secondary partitions for a table on target will be computed based on the hashcode value of the natural key.

This field is mandatory for incremental ingestion tables.

NOTE: The value of natural key column cannot be updated when updating a row on source; all the components of the natural key are immutable.

The option to crawl the table in parallel with multiple connections to the database.

Split-by column can be an existing column in the data or derived from an existing column. Any column for which minimum and maximum values can be computed can be a split-by key (numbers, date, datetime, timestamp).

To derive a column, check the Use substring from field option and select the Extract function.

Synchronization Configuration

The source column to be used as update watermark.

NOTE: It can also be the source column to be used as insert watermark.

The type of the query based ingestion. The options include SCD Type 1 and SCD Type 2.

NOTE: SCD Type 1 always overwrites the existing record in case of updates.

This column used for fetching of delta.

This column must be a numeric column. Source datatype and target datatype must be equal.

Target Configuration

The option to partition the data in target. The partition column also can be derived for date, datetime and timestamp column. This will further partition the data. A hierarchy of partitions are supported with both normal partitions and derived partitions.

NOTES: Ensure that the partition columns data is immutable. You can also provide a combination of normal and derived partitions in the hierarchy.

The number of reducers for the ingestion map reduce job. Increasing the number of reducers helps reduce the ingestion duration.

This will help in processing the MR jobs faster. This will be effective with the combination of partition key and number of secondary partitions. In any data processing using MR job, this will help in bringing the parallelism based on the data distribution across number of primary partitions and secondary partitions on Hadoop.