Configuration

Enabling ECB in Infoworks DF

By default, the ECB feature is enabled on the Infoworks DF (from version 2.6.0). You can enable or disable this feature by setting the ENABLE_ECB admin constant key to true or false respectively.

Configuring ECB Agent in Infoworks DF

After installing and enabling the ECB Agent, perform the following to create agent profile on the Infoworks DF cloud installation:

NOTE: Only a user with admin privileges can create an agent profile.

Login to the Infoworks DF and click to the Admin menu.
Click the ECB Agent icon.

NOTE: If the icon is not available, verify the version of the Infoworks DF installed.

Click the New ECB Agent button to create an agent profile.
Enter the following details:
Name: Name of the agent profile for identification.
Endpoint: HTTP/S address of the ECB Agent, specifically the address of the agent job service. The default port is 8090.
Staging Directory: Full path to directory on the on-premise host where ECB agent is installed. This is used as temporary storage before publishing to Infoworks cloud installation. For example, /opt/temp/ecb.
Auth Key: The Auth Key password used in the agent during installation.
Cloud Type: Cloud provider for the cloud storage services.

Cloud Provider Specific Options:

Path: The path where data is staged on the cloud. This path is a combination of the bucket name (or container name in Azure blob Store) and an absolute path inside the bucket for exclusive use by ECB.
Credential Type: Varies based on the cloud provider. You can use the system default credentials or provide custom credentials like Access Keys for AWS, GCP Service Account Credentials File, Credential file with Azure Connection String, etc.
Click the Test Connection button to validate the settings. If the test fails, an error message will be displayed.
Click the Save button to add the agent.

Authentication Setup

Agent

Add the following keys in the $ECB_HOME/conf/job.verticle.json file:

ecb.username: Username for ECB agent auth.
ecb.password: Password to be used for ECB Agent.

SSL/TLS Configuration

For details on SSL Certificate generation, see Appendix.

Agent

Add the following keys in the $ECB_HOME/conf/job.verticle.json file:

ecb.server.keystore.path: Path to the Server keystore (.jks) file.
Use either of the following types of certificates:

Self-signed certificate: Client certificate generated by the user using keytool. Certificate Authority (CA) signed certificate: Certificate issues by the user from CA.

ecb.server.keystore.password: Password for the keystore file.
ecb.password: Password to be used for the ECB Agent. This must be same as the password in the ECB Agent page_._
ecb.username: (Optional) username for the ECB Agent access. The default value is iw-user.

Cloud

Add the following keys to enable SSL:

ecb_truststore_filepath: Path to the ECB Truststore file. The truststore includes the fingerprints of all the trusted authorities (Certified Authorities). To enable the default truststore, set the path to the cacerts folder.
ecb_truststore_password: Password for the ECB truststore.
enable_ssl: The value is set to true by default, to enable SSL authentication.
ecb_trust_anonHost: This value is set to false by default, to not trust anonymous host. Used to internally configure the host name verifier.
ecb.username: (Mandatory) username for the ECB Agent access. The default value is iw-user.

Starting Agent

Start the agent using the following command: bash $ECB_HOME/bin/start.sh

Usage

DFI Ingestion

To ingest data from files mounted on an on-premise system onto cloud installation of the Infoworks DF using ECB, perform the following:

Identify the node where the source data files are stored
Ensure the ECB Agent is installed and configured on this machine (agent).
Identify the Infoworks cloud installation on which the data will be ingested (cloud).

The Infoworks DF on the cloud is the main interface to setup and execute ingestion jobs.

RDBMS Ingestion

To ingest data from RDBMS source into cloud installation of the Infoworks DF using ECB, perform the following:

Identify the node where the source data files are stored.
Ensure the ECB Agent is installed and configured on the machine (agent).
Identify the Infoworks cloud installation on which the data will be ingested (cloud).

Currently, Oracle, Teradata and SQL Server sources are supported. The synchronization types supported are Full load, BatchID and Timestamp. For details, see RDBMS Ingestion.

The Infoworks DF on the cloud is the main interface to setup and execute ingestion jobs.

XML Ingestion

To ingest data from XML sources into cloud installation of the Infoworks DF using ECB, perform the following:

Identify the node where the source data files are stored
Ensure the ECB Agent is installed and configured on this machine (agent).
Identify the Infoworks cloud installation on which the data will be ingested (cloud).

The Infoworks DF on the cloud is the main interface to setup and execute ingestion jobs.

Reference Video

The demo video of Enabling ECB Agent for XML Ingestion is available here.

JSON Ingestion

To ingest data from JSON sources into cloud installation of the Infoworks DF using ECB, perform the following:

Identify the node where the source data files are stored
Ensure the ECB Agent is installed and configured on this machine (agent).
Identify the Infoworks cloud installation on which the data will be ingested (cloud).

The Infoworks DF on the cloud is the main interface to setup and execute ingestion jobs.

Reference Video

The demo video of Enabling ECB Agent for JSON Ingestion is available here.

Creating ECB Source

Prerequisites

Ensure that the ECB Agent is configured on the cloud as mentioned above.
Note the name with which the agent is configured.

To create an ECB source, perform the following:

Create a source (DFI, Oracle, Teradata) on cloud using the Infoworks DF.
Click the Sources menu, click the required source.
Click the Settings icon. The Source Settings page is displayed.

Perform the following to configure the source created as ECB source:

Select the From Filesystem option.
Source Base Path: enter the absolute path to the data files on the agent.
ECB Agent: select the agent name.

Click the Add Entry button in the File Mappings section and add the table entries.
Configure the table as shown in the sample below:

Metadata Crawl

Perform the following after configuring the source:

Navigate to the Source Configuration page.
Click the Recrawl Metadata button.

Data Crawl

For DFI data crawl full load and incremental load, see the Delimited File Ingestion section in the Data Ingestion and Synchronization chapter.

Known Issue

When ingesting Avro file format, disable publisher compression by setting the following property in the ecb.properties file: publisher.enable.compression=false.

Last updated on