Sybase Log-Based Incremental Ingestion

Through Sybase log-based approach, data is ingested into Hive and is continuously synchronized with the database using the DBTRAN utility provided by Sybase. This utility must be set up on the edge node where Infoworks DataFoundry services are running.

Log files from the Sybase server are stored in a temporary location on edge node, that can be configured. The corresponding folder must be created on the edge node.

Configurations

To support log-based data synchronization in Sybase, the following global level parameters must be configured in the admin configuration page:

  • SYBASE_EDGENODE_TEMP_FOLDER: The directory where the database log file must be stored.
  • SYBASE_INSTALLATION_PATH: The folder where the DBTRAN utility is available on the edge node.
  • SYBASE_CLIENT_SOURCING_PATH: The path for sourcing Sybase client.

Optional Configurations

Following are the optional source-level parameters:

  • LOG_FILE_NAME: The name of the log file that will be stored in the temporary folder created on the edge node. The default value is log. If the log file name is different, specify the corresponding name (without .log).
  • FIRST_CDC_OFFSET: During first incremental ingestion, Infoworks DataFoundry reads and parses the complete log data present in the log file. To fetch only the incremental data, specify the offset from where the data must be fetched.

Enabling Logs in Sybase Server

Sybase IQ includes IQINIT admin utility used to create database. When the database is created with this utility, the transactional logs for the database will be automatically stored in the log file. You must have the access to the file system to create the .db file. You can disable the transactional logs by explicitly mentioning the -f option when running the start_iq command line utility.

Following is the command to create database and to enable logging in Sybase:

NOTE: In this command, database called automation_db is created using the database credentials.

Copy

Known Issues

  • Derived Splitby and Derived Partition are not supported.
  • Schema Synchronization is not supported.
  • Split-by key will be used only during full-load but not during CDC.
Type to search, ESC to discard
Type to search, ESC to discard
Type to search, ESC to discard
Next to read:
Apache Ignite Ingestion

For more details, refer to our Knowledge Base and Best Practices!

For help, contact our support team!

(C) 2015-2022 Infoworks.io, Inc. and Confidential

On This Page
Sybase Log-Based Incremental IngestionConfigurationsOptional ConfigurationsEnabling Logs in Sybase Server