Upgrade Procedure

NOTE: Before you upgrade, read the Release Notes for the updates in the version.

Prerequisites

Ensure that the Mongo DB is up and running.
If Mongo HA is enabled, ensure that the Mongo DB is in primary or secondary state on all nodes. If the Mongo HA node is in unrecoverable state, perform the procedure mentioned in Resync Member of Replica Set.
Cleanup the logs that are not required, using the following script. You can create a cron job to schedule this in your Infoworks environment.
If Infoworks is configured with SSL-enabled LDAP Auth, ensure to add the LDAP server certificate to the Java Keystore on the edge node before performing the upgrade.

Upgrade Procedures

The upgrade logs will be available in <path_to_Infoworks_home>/platform/iw-installer-metadata/logs/upgrade.log.

CentOS/RHEL6

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

Run the following command: source $IW_HOME/bin/env.sh
Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
Upgrade the Infoworks DF version using the following commands: ./update.sh -v <version-number>

NOTE: For machines without certificate setup, --certificate-check parameter can be entered as false as described in the following syntax: ./update.sh -v <version_number> --certificate-check=<true/false>. The default value is true. If you set it to false, this performs insecure request calls. This is not a recommended setup.

To exclude a particular service, use the following command: --exclude-services cube-engine . For example, to exclude Cube engine, use ./upgrade.sh -v <version_number> --exclude-services cube-engine

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

updates the metadata DB binaries
sets up feature compatibility version

CentOS/RHEL7

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

Run the following command: source $IW_HOME/bin/env.sh
Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
Upgrade the Infoworks DF version using the following commands: ./update.sh -v <version-number>-rhel7

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

updates the metadata DB binaries
sets up feature compatibility version

AZURE

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

Run the following command: source $IW_HOME/bin/env.sh
Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
Upgrade the Infoworks DF version using the following commands: ./update.sh -v <version-number>-azure

NOTE: If you are upgrading from Infoworks DataFoundry versions 2.8 or 2.9 to Infoworks DataFoundry version 3.1, then run the following commands:

source /bin/env.sh
cd $IW_HOME/scripts
wget https://infoworks-setup.s3.amazonaws.com/3.1/conf_migrate_azure_310.sh
chmod +x conf_migrate_azure_310.sh
./conf_migrate_azure_310.sh

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

updates the metadata DB binaries
sets up feature compatibility version

GCP

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

Run the following command: source $IW_HOME/bin/env.sh
Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
Upgrade the Infoworks DF version using the following commands: ./update.sh <version-number>-gcp

If the user wants to upgrade to Infoworks DataFoundry version 3.1.0 from any of the lower versions, run the following commands:

wget https://infoworks-setup.s3.amazonaws.com/upgrade_gcp_310.sh
chmod +x upgrade_gcp_310.sh
./upgrade_gcp_310.sh

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

updates the metadata DB binaries
sets up feature compatibility version

EMR

NOTE: Before starting the upgrade, ensure that MongoDB is running and no Infoworks jobs are running.

To upgrade the Infoworks DF version, execute the following commands on the edge node:

Run the following command: source $IW_HOME/bin/env.sh
Navigate to the scripts directory using the following command: cd $IW_HOME/scripts; where, $IW_HOME is the directory where the Infoworks DF is installed. If scripts folder is not available (2.4.x, 2.5.x, 2.6.x base versions), create scripts folder in $IW_HOME.
Download the update script using the following command: wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
Upgrade the Infoworks DF version using the following commands: ./update.sh -v <version-number>-emr

If the base version is below version 2.7.2, the upgrade procedure upgrades Metadata DB (Mongo) from 3.6 to 4.0 version. The upgrade of metadata DB includes the following:

updates the metadata DB binaries
sets up feature compatibility version

Upgrade Considerations

If metadata backup is not successful, the upgrade will be discontinued.
If jobs are run during upgrade, the upgrade will be discontinued.
If any step fails during upgrade, the Infoworks upgrade process will automatically rollback to the base version, without any inconsistency.
The HDFS temp directory cache will be cleared after the upgrade.

NOTE: Rollback is automatically handled during the upgrade process.

Silent Upgrade Procedure

To perform the upgradation offline, follow these steps on the edge node:

Step 1: Run the following command: source $IW_HOME/bin/env.sh

Step 2: Get the installer tar ball locally.

Step 3: Extract the installer by running the following command:tar -xf deploy_<version_number>.tar.gz

Step 4: Place it in the IW_HOME/scripts folder: IW_HOME/scripts/iw-installer

Step 5: Get the Infoworks DataFoundry tar ball.

Step 6: Place it in the required location using the following commands:

mkdir IW_HOME/temp/downloads

cp infoworks-x.tar.gz IW_HOME/temp/downloads

Step 7: In IW_HOME/bin/auto-update-flags file set auto_mongo_upgrade_download="n".

Step 8: Contact support to download the Mongo binary file (mongodb-4.0.9.tar.gz).

Step 9: Place the file in the following folder: ${IW_HOME}/temp/mongo-upgrade-4.0/mongodb-4.0.9.tar.gz

Step 10: Navigate to installer directory using the following command: cd IW_HOME/scripts/iw-installer

Step 11: Upgrade the Infoworks DF version using the following commands: ./update.sh -v <version-number>

Post Upgrade

After upgrading the Infoworks services, perform the following post-upgrade tasks:

Verify if all the Infoworks services are running
Obtain the Infoworks license key from Infoworks support (for upgrade from versions below 2.5.0)

Verifying Infoworks Services

Perform the following:

Trigger a simple Ingestion, Data Transformation and Workflow job.
Verify that the jobs are running correctly.

Obtaining Infoworks License Key

An admin can perform the following License Management operations from the Infoworks DF:

Add license keys
Update license keys
View features added to the product

Adding License Key

To add a new license key follow the below steps:

Login to the Infoworks DF.
Navigate to Admin > License Manager. The Infoworks License page is displayed which includes the following sections:

Summary: Summary of the Infoworks License

License Violation: The list of license compliance anomalies. For details see the License Violations section below.
Installation Id: Unique installation ID for each server
License Type: Type of license like dl_basic, dl_enterprise, etc.
License Starts: Timestamp when the license was activated.
License Ends: Timestamp when the license expires.

Features - Details of features added to the product

License Feature: list of features.
Value: value assigned to each feature. The features that are activated using the license key are displayed as Yes.

Install Your New Infoworks License: text box to paste the license key.

Copy the Installation ID and mail it to Infoworks. The license key will be sent to you via mail.
Copy the license key and paste in the Install Your New Infoworks License text box.
Click Install New License. The license will be installed.

To update the license key, follow the same procedure. Adding a new license key will replace the existing license.

License Violations

Following is the list of license violations:

Server not having a license.
License being expired or not started.
Registered users not being licensed.
Usage of unlicensed source types.
Usage of unlicensed feature (Data Transformation, Cube, etc).

License Warning Messages

The license compliance anomalies are displayed in the License Management page and warnings are displayed in the respective source, pipeline, workflow and cube pages.

All the license violations are displayed in the License Management page.
When you use an unlicensed source type, a warning is displayed for the particular Source Type in the Manage Source page.
When you edit a pipeline without the required license, a warning is displayed in the pipeline editor page.
When you edit a workflow without the required license, a warning is displayed in the workflow editor page.
When you edit a cube without the required license, a warning is displayed in the cube editor page.

Post-upgrade Procedure

Modifications must be made in the $IW_HOME/conf/conf.properties file. The properties must be modified when upgrading Infoworks DataFoundry from previous versions to 3.1 version.

NOTE: New installations of Infoworks DataFoundry 2.8.0 works automatically, without the modifications.

Environment: HDP 2.x

Step 1

Navigate to the $IW_HOME/conf/conf.properties file.
Remove $IW_HOME//lib/parquet-support/* from the iw_jobs_classpath key value.
Ensure that the additional : at the end of the value is removed.

Step 2

Navigate to the $IW_HOME/conf/conf.properties file.
If the base version is below 2.8, replace the Hive client libraries (like /usr/hdp/current/hive-client/lib/…) in the iw_jobs_classpath key value with /usr/hdp/current/hive-client/lib/*.
Ensure that the additional : at the end of the value is removed.

Environments: HDP 2.x, GCP, EMR

Step 3

Navigate to the $IW_HOME/conf/conf.properties file.
Remove $IW_HOME/lib/shared/* from the df_batch_classpath key value.
Ensure that the additional : at the end of the value is removed.

Step 4

Navigate to the $IW_HOME/conf/conf.properties file.
Remove $IW_HOME/lib/shared/* from the df_tomcat_classpath key value.
Ensure that the additional : at the end of the value is removed.
Stop and start the transformation service using the following commands: source $IW_HOME/bin/env.sh; $IW_HOME/bin/stop.sh df; $IW_HOME/bin/start.sh df

Custom Jar Files

From Infoworks DataFoundry version 2.7.1, during an upgrade the $IW_HOME/lib/extras directory and the subdirectories are preserved to place the custom jars. Users having custom jars must manually place them in these folders.

The custom jar files or libraries related to ingestion, data transformation, cubes, export, Hive crawls, streaming, and cloud must be placed in the following folders respectively:

Ingestion and Cube: $IW_HOME/lib/extras/ingestion
Data Transformation: $IWHOME/lib/extras/dt
Export: $IW_HOME/lib/extras/export
Hive Crawl: $IW_HOME/lib/extras/hive-crawl
Streaming: $IW_HOME/lib/extras/streaming
Cloud: $IW_HOME/lib/extras/cloud

It is ensured that these folders are preserved during the Infoworks upgrade process. Any jar files placed outside these folders are deleted during the upgrade process.

Following are the folders to classpath key mapping:

iw_jobs_classpath: $IW_HOME/lib/extras/ingestion
df_batch_classpath, df_tomcat classpath: $IW_HOME/lib/extras/dt
iw_export_classpath: $IW_HOME/lib/extras/export
iw_hive_schema_crawl_classpath: $IW_HOME/lib/extras/hive-crawl
spark_streaming_job_classpath: $IW_HOME/lib/extras/streaming
iw_ecb_client_classpath: $IW_HOME/lib/extras/cloud

Post Infoworks DataFoundry 2.7.1 migration, these folders are placed at the beginning of their respective classpath keys.

Ensure to place all the custom jars in the respective folders. After Infoworks DataFoundry 2.7.1 upgrade, you must manually place all the custom jars added in different folders (in older versions) to the above mentioned folders, to ensure compatibility.

Last updated on