Release Notes 2.7.2

Date: 05 JUL 2019

New Features and Enhancements

Component: Data Ingestion and Synchronization

• IPD-7554 - REST API Ingestion: A new ingestion type, REST API Ingestion, has been added.
• IPD-7520 - Ability to Use Source in Pipeline after Metadata Crawl: Data Transformation pipelines can now use data sources immediately after the data source has successfully completed a metadata crawl. Previously, the sources could only be used in pipelines after both metadata and data crawl had been completed. This is helpful in rapid prototyping of data transformations pipelines, without the need of ingesting data. Also, artifact promotion of data transformation pipelines from development to production environment is supported without data being ingested.
• IPD-7633 - Edit Schema in Streaming Ingestion: For streaming enabled tables, users can now enter timestamp and date formats of incoming messages in the Edit Schema page. This feature can be used to provide the details of the messages that are part of the stream, facilitating use cases where the messages in the stream have different format from the initial load file/RDBMS.

Component: Data Transformation

• IPD-7560 - Snowflake Target Support: The output of data transformation pipelines can now be incrementally synchronized to Snowflake. The new Snowflake Target supports merge, overwrite and append modes for incremental synchronization. NOTE: Snowflake targets are only supported in pipelines that use Spark as execution engine.
• IPD-7312 - Pipeline Executor Preconfiguration: Static configurations used for different execution engines can now be set in the df_executor_configs.yaml configuration file located in the $IW_HOME/conf folder. This file can be modified to change the statements based on the permission model in user environment.
• IPD-7558 - Support for Executing Pipelines via Apache Livy: Support for submission of Spark pipelines via Apache Livy has been added. This allows parallel execution of more jobs without adding load on the edge node.
• IPD-8651 - Primary Partition Column Support in Pipeline: In pipeline target, all columns, irrespective of the data type can now be used for primary partition selection.

NOTE: Using higher cardinality column as primary partition causes out-of-memory issues.

Component: Orchestration

• IPD-7573 - Workflow Deletion: Users can now delete workflows via the user interface and remove unwanted workflows.
• IPD-7579 - Workflow Status Auto Refresh: The workflow state and task state are now refreshed automatically without the use of the Refresh button to provide fresher information.

Component: Platform

• IPD-7639 - Metadata Database Upgrade: The metadata repository has been upgraded from MongoDB 3.6 to 4.0 to enhance the security of the application.

Component: Cloud

• IPD-7587 - Split-by and Boundary Query for ECB RDBMS Source: Ingestion via Enterprise Cloud Bridge can now be parallelized based on the split-by column and boundary query in RDBMS source for faster transfer.
• IPD-7590 - Support for ECB Agent in XML and JSON Source: XML and JSON sources can now be ingested via Enterprise Cloud Bridge for reliable transfer of the files from on-premise agent to Infoworks DataFoundry cloud installation.
• IPD-7672 - Support for ECB Agent in SQL Server Source: SQL Server sources can now be ingested via Enterprise Cloud Bridge for reliable transfer of the data from on-premise database to Infoworks DataFoundry cloud installation, without exposing the database to a public network.

Component: Admin and Operations

• IPD-7576 - Support to View Admin Jobs: A new page, Admin Jobs, has been added to display the details of the jobs run by admin users.
• IPD-7568 - Bulk User Addition from Active Directory: Admin can now create accounts for bulk users and access their details. Artifact access can be defined for these users, even if they have not logged in to Infoworks DataFoundry earlier.

Bug Fixes

• IPD-7655 - Workflow Time Zone Issue: The workflow build chart displayed incorrect job times. This issue has been fixed. The browser time and time zone will now be displayed in the chart.
• IPD-7772 - Date Issue in DFI Ingestion: Incorrect date values were displayed after performing delimited file ingestion. This issue has been fixed.
• IPD-7698 - Pipeline Version Deletion Error in Workflow: Workflow execution failed with an error, when the workflow included a pipeline version that was deleted. A warning message indicating the impact of the pipeline version on workflows, is now displayed when the user tries to delete a pipeline version.
• IPD-7252 - Issue in Refreshing Distinct Values: When performing segmented load, the Refresh Distinct Values button did not work as expected. The button now performs the documented behaviour.
• IPD - 7701 - Incorrect Pipeline Version During Workflow Run: When running a workflow which included a pipeline with more than 10 versions, an incorrect pipeline version was executed. This has been fixed in this release.

Installation

Refer Installation to install Infoworks ADE 2.7.2.

Upgrading to This Release

To upgrade your current Infoworks ADE version, execute the following commands on the edge node:

NOTE: Before starting the upgrade, ensure that all Infoworks services are running and no Infoworks jobs are running.

• Run the following command: source $IW_HOME/bin/env.sh
• Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks ADE is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
• Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
• Upgrade the Infoworks ADE version using the following commands: ./update.sh -v <version_number>

NOTES:

For CentOS/RHEL6, replace <version_number> with 2.7.2

For CentOS/RHEL7, replace <version_number> with 2.7.2-rhel7

For Azure, replace <version_number>with 2.7.2-azure

For GCP, replace <version_number>with 2.7.2-gcp

For EMR, replace <version_number>with 2.7.2-emr

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

• updates the metadata DB binaries
• sets up feature compatibility version