Release Notes 3.1.2

Date: 4 AUG 2020

New Features and Enhancements

Component: Data Ingestion and Synchronization

• Support to Ingest User-defined Subsets of Data for Oracle Sources: Users can now specify filter conditions to extract a subset of data from Oracle source tables. For more details, see Filter Query for Oracle Sources.

Component: Data Transformation

• Support for Redshift Target Export: Redshift is now supported as a target in transformation pipelines. Data can be exported to Redshift, and kept incrementally synchronized via a fast and parallelized process. Infoworks DataFoundry also optimizes the Redshift table for query patterns through user-configurable settings for distribution and sort styles.

IMPORTANT If spark.masteris set to_YARN_ in the/conf/conf.properties file, remove * at the end of the iw_``udfs_dir configuration value (iw_udfs_dir=/opt/infoworks/dt/udfs/*).

Component: Admin and Operations

• Support for Workflow Lists: Workflow list displays an admin view of all the workflows, workflow runs, and associated tasks, with the ability to search and sort based on user requirements. This allows the admin to process bulk operations on all the workflows, created by all the users. For more details, see Workflow List.

Component: Export

• Support Text Enclosing Character for Delimited Export to CSV: Infoworks DataFoundry now supports delimited file export to CSV with character enclosing capabilities to automatically wrap the data using the “ character. For more details, see Delimited File Export.

Component: Platform

• The version of EMR supported for Infoworks DataFoundry is now 5.28.1. For more details, see EMR Deployment.

Installation

Refer Installation and Configuration to install Infoworks DataFoundry 3.1.2.

Upgrading to This Release

To upgrade your current Infoworks DataFoundry version, execute the following commands on the edge node:

NOTE: Before starting the upgrade, ensure that no Infoworks jobs are running.

• Run the following command: source $IW_HOME/bin/env.sh
• Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DataFoundry is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks DataFoundry version using the following commands: ./update.sh -v <version_number>

NOTE: For machines without certificate setup, --certificate-check parameter can be entered as false as described in the following syntax: ./update.sh -v <version_number> --certificate-check <true/false>. The default value is true. If you set it to false, this performs insecure request calls. This is not a recommended setup.

NOTES:

For HDP, CentOS/RHEL6, replace <version_number> with 3.1-hdp-rhel6

For HDP, CentOS/RHEL7, replace <version_number> with 3.1-hdp-rhel7

For MapR, CentOS/RHEL6, replace <version_number> with 3.1-rhel6

For MapR, CentOS/RHEL7, replace <version_number> with 3.1-rhel7

For Azure, replace <version_number>with 3.1-azure

For GCP, replace <version_number>with 3.1-gcp

NOTE: If MongoDB is not managed locally, the MongoDB server must be updated to the latest version (4.0) manually.

For EMR, replace <version_number>with 3.1-emr

If the base version is below version 2.7.0, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version

NOTE: If you are upgrading from Infoworks DataFoundry versions 2.8 or 2.9 to Infoworks DataFoundry version 3.1, then run the following commands:

• source /bin/env.sh
• cd $IW_HOME/scripts
• wget https://infoworks-setup.s3.amazonaws.com/3.1/conf_migrate_azure_310.sh
• chmod +x conf_migrate_azure_310.sh
• ./conf_migrate_azure_310.sh

Post-upgrade Procedure

Modifications must be made in the $IW_HOME/conf/conf.properties file. The properties must be modified when upgrading Infoworks DataFoundry from previous versions to 3.1 version.

NOTE: New installations of Infoworks DataFoundry 3.1 works automatically, without the modifications.

Environments: HDP 2.x, GCP, EMR

Step 1

• Navigate to the $IW_HOME/conf/conf.properties file.
• Remove $IW_HOME/lib/shared/* from the df_batch_classpath key value.
• Ensure that the additional : at the end of the value is removed.

Step 2

• Navigate to the $IW_HOME/conf/conf.properties file.
• Remove $IW_HOME/lib/shared/* from the df_tomcat_classpath key value.
• Ensure that the additional : at the end of the value is removed.
• Stop and start the transformation service using the following commands: source $IW_HOME/bin/env.sh; $IW_HOME/bin/stop.sh df; $IW_HOME/bin/start.sh df

Environment: HDP 3.1.4

Post-installation Steps for HDP 3.1.4

• Navigate to the $IW_HOME/conf/conf.properties file.
• Append hive.optimize.index.filter=false for hiveConfigurationVariables.

Custom Jar Files

From Infoworks DataFoundry version 2.7.1, during an upgrade the $IW_HOME/lib/extras directory and the subdirectories are preserved to place the custom jar files. Users having custom jar files must manually place them in these folders. For more details, see Custom Jar Files.

PAM

For the PAM, see Product Availability Matrix.