Delimited File Export
This feature exports pipelines and sources to HDFS or cloud storage in CSV format with UTF-8 encoding.
Field Description
- Export Type: Select Full Export/Incremental Export.
- Target: Select the database type to be exported as Delimited File.
- Export File Systems: Select the storage system to export data as S3.
- Account Type: Authentication mechanism based on the storage system. The System Default option uses the credentials globally available in the system or available to the Infoworks user.
- Export Location: Absolute URI of the location where data will be exported. This path must contain the file scheme, bucket name (if applicable) and absolute path to the export directory. Following are few examples,
- s3://bucket/path/to/export
- hdfs://path/to/export
- hdfs://namenode-ip.fqdn/path/to/export
- gs://bucket/path/to/export
- wasb[s]://@.blob.core.windows.net/path/to /export
The final path of export will be a sub directory under the specified path based on the export type. For full export, the final path will be s3://bucket/path/to/export/full/csv. For incremental export, the final path will be s3://bucket/path/to/export/cdc/csv/, where timestamp refers to export job time.
NOTE: Full export overwrites existing full export data in the same path, while incremental data always writes to a new path using the timestamp in the path.
- Export Columns: List of columns to be exported.
- Enable Partition: Option to enable partitioning.
- Select Partition Column: Option to select partition column when partitioning is enabled.
- Field Delimiter: Single character input for field delimiter. The default value is ,
- Escape Character: Single character input to escape other control characters in column values. The default value is \
- Collection Item Separator: Field separator for struct, array and map complex data types. The default value is #
- Map Key Terminator: Key value separator for map data type. The default value is :
- Null Values: Substitution for null column values. The default value is \N
Limitations
- Partition Key: Only one partition key can be used per export job. The data type of the partition key column must be primitive and not complex (like map, array). Number of partitions supported depends on the Hive settings. For more details, see the hive.exec.max.dynamic.partitions.pernode configuration.
- Default Partition Name: All data with null partition values will be moved to a directory with default partition name. Special characters must be avoided - semicolons will be removed. The default value is HIVE_DEFAULT_PARTITION.
- Delimited file parameters: The parameters like field separator, escape character, etc must be a single UTF-8 character; multiple characters are not supported. Back slash in these options must be escaped. Quote characters are not supported, but quotes can still be added as part of the pipeline.
Troubleshooting
IW Constants and Configurations
- export_fs_data_overwrite: The default value is false. Forcefully overwrites the existing target path.
WARNING: All existing data will be deleted.