Infoworks High Availability Setup

Prerequisites

Ensure the following:

At least three node cluster is set up.
All nodes are identical.
Primary edge node is setup with Infoworks.

Obtain the latest HA scripts from the artifactory using the following commands:

cd <IW_HOME>/bin
rm -rf infoworks-ha-ansible
wget <link-to-download>; reach out to your Infoworks support representative to get the link to download and replace with the link.
tar -xvf ha_scripts.tar.gz

Recommended Steps

Following are the recommended steps:

Filesync Script Run

After first-time setup of Infoworks on primary.

HA Setup

Mongo HA Setup: After first-time setup of Infoworks on primary.
Postgres, RabbitMQ, Platform setup HA script run: After first-time setup of Infoworks on primary.

Failover

Manual HA for Infoworks service: Only if an Infoworks service goes down on primary. These steps must be followed if any Infoworks service goes down on primary edge node. This procedure brings up the entire Infoworks and infrastructure service set on standby node.
Postgres-Failover script: Only if Postgres goes down on primary. This script will setup the environment and configurations for a standby node to be brought up as new master node for Postgres.

NOTE: For details on setting a three-node cluster for HA compatibility, see How to Setup HA with Three Nodes for Infoworks.

FileSync

Prerequisites

IW_USER must be present on remote machines (secondary nodes).
SSH_USER must have su permissions to IW_USER.
Infoworks must be setup on primary node.
Infoworks home directory must be created and write permissions must be provided for IW_USER on secondary nodes.

Procedure

Navigate to the <IW_HOME>/bin/infoworks-filesync-ansible folder.
Run the following command: ./setup-iw-filesync.sh

The required files will be automatically copied to remote hosts using Infoworks tool.

Mongo HA

Steps to Setup HA

Prerequisites

IW_USER must be present on remote machines (secondary nodes).
SSH_USER must be have su permissions to IW_USER.
Any running instances of Mongo on the remote nodes must be stopped.
Infoworks must be setup on the primary node.
Infoworks Home directory must be created and write permissions must be provided for IW_USER on secondary nodes.
Secondary nodes must be identical to the primary node (part of the same cluster).

If <IW_HOME/resources/mongodb> already exists on secondary nodes, ensure that all three nodes have same Mongo HA credentials:

In <IW_HOME>/resources/mongodb/conf/mongodb.conf, check the following key on all nodes and ensure they are identical: replSet, keyFile.
Ensure that keyFile specified as the value in mongodb.conf exists and is identical on all nodes.

If <IW_HOME/resources/mongodb> does not exist on secondary nodes, it will be created during Mongo HA setup.

Ensure that in the <IW_HOME>/resources/mongodb/conf/mongodb.conf file, the keys replSet and keyFile are uncommented.
Ensure that the keyFile specified as the value in mongodb.conf exists.

NOTE: Ensure that no Infoworks jobs are running. Running jobs will fail during HA setup.

Procedure

Navigate to the <IW_HOME>/bin folder on the primary edge node.
Run the following command: ./mongo-ha-setup.sh

The following occurs:

Pre-execution

A backup of the conf.properties file will be created in the <IW_HOME>/temp/conf.properties.YYYY-MM-DD-HH-MM file.
All Infoworks and infrastructure services will be stopped.
A tar file for the Mongo directory will be created in the <IW_HOME>/temp/<DATE>-pre directory. (Optional)
A prompt will be displayed to enter the path to pem file, ssh user, and mongo port which defaults to 27017.

User Creation

An oplogger user will be created with read access to local database.
An iw-ha admin user will be created, if it does not exist.

Replica-Set Creation

NOTE: Ensure that the replica set configurations, replSet and keyFile, exist with correct data in the Mongo configuration file (<IW_HOME>/resources/mongodb/conf/mongodb.conf) on the secondary and tertiary nodes while the HA setup is in progress.

A backup tar of Mongo directory will be created in the <IW_HOME>/temp/<DATE> directory.
A prompt will be displayed to enter the IPs/hostnames of the other two nodes.
Backup tar will be copied from the primary node to the secondary nodes and Mongo will be started on the secondary nodes.
When Mongo is up and running on the remote nodes, replica set is initiated and remote nodes are added to the replica set.

Post-execution

Remove the backup in the <IW_HOME>/temp/<DATE> folder.
Restart Infoworks services using the following steps:

Navigate to <IW_HOME>/bin.

Run the following commands:

./start.sh all

./start.sh orchestrator

Validation

On the primary node:

Navigate to <IW_HOME>/bin and run the following command: ./status.sh all

MongoDB Replica will display the status of mongo servers running in the cluster.

On successful installation, the replica set will be online. This can be verified using the following steps:

Navigate to <IW_HOME>/resources/mongodb/bin.

Getting <MONGO_PASSWORD>:

Find encrypted <MONGO_PASSWORD> using the value of metadbPassword from <IW_HOME>/conf/conf.properties.
Navigate to <IW_HOME>/apricot-meteor/infoworks_python/infoworks/bin/.
Find mongodb password by executing the following command: ./infoworks_security.sh -decrypt -p <MONGO_PASSWORD>security.sh -decrypt -p <MONGO_PASSWORD>

Execute the following command: ./mongo admin --host $host -u iw-ha -p $password with relevant parameters.

where, $host is the primary node hostname or ip and $password is <MONGO_PASSWORD>.

In the mongo prompt, execute the following command: rs.status()
The replicaSet information will be displayed.
Exit from prompt using the following command: exit

NOTE: If the Mongo HA node is in unrecoverable state, perform the procedure mentioned inResync Member of Replica Set.

Starting/Stopping/Monitoring Mongo HA

Ensure that the following files are available in <IW_HOME>/bin directory:

NOTE: Ensure that the latest version of mongoha_start.sh and mongoha-start-stop.sh files are downloaded.

mongo-ha-reset.sh - used to reset the Mongo HA node to non-HA node. The script does not impact the other remote machines.
mongo-ha-setup.sh, mongoha_start.sh - used to setup the Mongo HA.
mongo-ha-start-stop.sh - used to start/stop Mongo remotely from the edge node.
Usage: mongo-ha-start-stop.sh {host} {stop/start}
mongo_start.sh mongo_stop.sh - used to stop/start mongo locally. Available in the <IW_HOME>/resources/mongodb/bin directory.
status.sh - used to monitor the status of all Infoworks services including Mongo replica. Check for the MongoDB Replica parameter.

Postgres, RabbitMQ, Platform HA

Steps to Setup HA

Infoworks version 2.7.0 (or older)

Prerequisites

Ensure the following:

IW_USER is present on remote machines (secondary nodes).
Secondary nodes are identical to the primary node (part of the same cluster).
SSH_USER has su permissions to IW_USER.
Infoworks is setup on primary edge node.
All Infoworks and infrastructure services are stopped on secondary nodes.
Infoworks directory structure is created and identical on all nodes of cluster.

Procedure

Execute the following steps on the primary node

If Postgres HA is required,

Navigate to <IW_HOME>/bin
Run the following command: ./start.sh postgres

Navigate to the``/bin/infoworks-ha-ansiblefolder.

Run the following command: ./setup-iw-ha.sh

Setup HA for RabbitMQ, Postgres and Platform services to true, as required.
Enter the host details for the selected HA services.
Enter the ssh details for the hosts.
The required files will be automatically copied to remote hosts using Infoworks tool.
All Infoworks services are stopped.
Enter the su password. If password is not set, press Enter.
Installation procedure is started.
After successful installation, HA is setup for the selected services and Infoworks services are started.

NOTES:

While entering host details, ensure that the correct order of preference is followed from top to bottom in decreasing priority (topmost node will act as active node until there is a failure).
Postgres master node IP must be Infoworks primary node IP.
Ensure that the hostnames are uncommented.

Infoworks version 2.7.x+

Prerequisites

Ensure the following:

IW_USER is present on remote machines (secondary nodes).
Secondary nodes are identical to the primary node (part of the same cluster).
SSH_USER has su permissions to IW_USER.
Infoworks is setup on primary edge node.
All Infoworks and infrastructure services are stopped on secondary nodes.
Infoworks home directory is created and write permissions are provided for IW_USER.

Procedure

If Postgres HA is required,

Navigate to <IW_HOME>/bin.
Run the following command: ./start.sh postgres

Navigate to the <IW_HOME>/bin/infoworks-filesync-ansible folder.

Run the following command: ./setup-iw-filesync.sh

The required files will be automatically copied to the remote hosts using Infoworks tool.

Navigate to the <IW_HOME>/bin/infoworks-ha-ansible folder.

Run the following command: ./setup-iw-ha.sh

Setup HA for RabbitMQ, Postgres and Platform services to true, as required.
Enter the host details for the selected HA services.
Enter the ssh details for the hosts.
All Infoworks services are stopped.
Enter the su password. If password is not set, press Enter.
Installation procedure is started.

After successful installation, HA is setup for the selected services and Infoworks services are started.

NOTE: Ensure that postgres master and orchestrator services are running on the same machine.

FileSync Manual Steps

Copy Infoworks basics (if<IW_HOME>and its file structure is not yet present on secondary nodes)

On primary node:

source $IW_HOME/bin/env.sh
Stop all infoworks services

stop.sh all

stop.sh orchestrator

Remove existing tar temp directory: rm -rf $IW_HOME/temp/IW_HA
Create backup tar directory: mkdir -p $IW_HOME/temp/IW_HA/{{item}}

For the following items:

bin
conf
apricot-meteor
resources
platform
scripts

Create tar of Required binary and conf dependencies:

cd $IW_HOME

tar --ignore-failed-read -czf $IW_HOME/temp/IW_HA/{{item}}/{{item}}.tar.gz {{item}}

For the following items:

bin
conf
apricot-meteor
scripts

Create tar of required python dependencies:

cd$IW_HOME/resources

tar --ignore-failed-read -czf $IW_HOME/temp/IW_HA/resources/{{item}}.tar.gz {{item}}

For the following items:

python27

Copy all files from $IW_HOME/temp/IW_HA on primary node to $IW_HOME/temp/IW_HA on secondary node

On secondary node:

Check if iw user has write access to IW_HOME.

Remove existing backup

rm -rf $IW_HOME/{{item}}.bak || true

For the following items:

bin
conf
apricot-meteor
scripts
resources/python27

Backup existing resources to backup: mv $IW_HOME/{{item}} $IW_HOME/{{item}}.bak || true

For the following items:

bin
conf
apricot-meteor
scripts
resources/python27

Create Resources directory if not exists: mkdir -p $IW_HOME/resources/ || true

Create Infoworks log directory: mkdir -p $IW_HOME/logs/ || true

Extract common dependent packages: tar xzf $IW_HOME/temp/IW_HA/resources/{{item}}.tar.gz -C

$IW_HOME/resources/

For the following items:

bin
conf
apricot-meteor
scripts

Extract python resources tar to required path: tar xzf $IW_HOME/temp/IW_HA/resources/{{item}}.tar.gz -C

$IW_HOME/resources/

For the following items:

python27

Infoworks HA Failover

Mongo HA

On failure of Mongo on primary, secondary node will automatically be promoted. No action is required.

Postgres, RabbitMQ, Platform HA

On failure of RabbitMQ or platform services on primary, secondary node will automatically be promoted. No action is required.

Steps for Postgres Failover

Prerequisites

setup-iw-ha.sh must have been run previously for Postgres. This will result in the system to be in the following state:

In all the nodes, conf.properties will have the following values (values are representational):

postgresha=y
postgres_host=172.30.1.5,172.30.1.6,172.30.1.7postgres_
port=3008,3008,3008

Postgres service on primary is running as master:

Navigate to <IW_Home>/bin.
Execute the following command: ./status.sh
Expected output:
Postgres [master] RUNNING

Postgres service on secondary nodes are running as standby:

Navigate to <IW_Home>/bin.
Execute the following command: ./status.sh
Expected output:
Postgres RUNNING

Procedure

Stop Postgres on all standby nodes using the following command: /bin/stop.sh postgres

In the one of the cluster nodes, run the following steps:

Navigate to the <IW_Home>/bin/infoworks-ha-ansible folder.
Run the following command: ./postgres-failover.sh
The Postgres will run in standalone mode.
Enter the ssh details for the hosts (ensure to specify a working standby node as master in hosts file).
Enter the su password. If password is not set, press Enter.

The following steps will occur:

Stops Postgres server on existing postgres master.
Gets replication lag from all standby nodes.
Identifies most synched standby host.
Promotes Postgres on most synced standby as master.
Updates conf.properties on all hosts.
Stops Postgres on all nodes except current master.
Restarts orchestrator services on all nodes.

Validation

On successful completion,

Postgres will be running on the new master and service will be shown as Postgres [master].
Postgres will be stopped on other nodes.

NOTE: Newly promoted master node can be identified during the ansible script run of postgres-failover.sh (as shown), namely MOST_SYNCED_STANDBY.

In the current master (promoted) node, setup the Postgres HA again using the following steps:

Execute the following steps on the newly promoted master node.

Navigate to the <IW_Home>/bin/infoworks-ha-ansible folder.

Run the following command: ./setup-iw-ha.sh
Setup HA only for Postgres and make it false for RabbitMQ and Platform.
Enter the host details for the selected HA services.
Enter the ssh details for the hosts.
The required files will be automatically copied to remote hosts using Infoworks tool.
All Infoworks services are stopped.
Enter the su password. If password is not set, press Enter.
Installation procedure is started.

After successful installation, HA is setup for the selected services and Infoworks services are started.

NOTE: While entering host details, newly promoted master is provided under [postgres-master] and the old master with the arbiter node are given under [postgres-standby].

Infoworks Services

Manual Steps for Failover

These steps can be used if one of the following Infoworks services goes down on the primary node:

User Interface
Hangman
Governor
Rest API
Query Service
Orchestrator
Cube Engine
Data Transformation

Prerequisites

Ensure the following:

IW_USER is present on remote machines (secondary nodes).
The remote machine is identical to the existing edge node (must be part of the cluster).
SSH_USER has su permissions to IW_USER.
Infoworks directory structure is created and identical on all nodes of cluster.
All Infoworks and infrastructure services are stopped on secondary nodes.

Infoworks version 2.7.0 (or older)

Sync files from the primary node to secondary nodes using the steps mentioned in the following section: FileSync Manual Steps.

Restart all services

Stop all Infoworks services on primary node.
Restart all of Infoworks services on secondary node.

Restart Postgres, platform, RabbitMQ

If HA has been setup for one of these services previously, perform the following steps to bring up the service on secondary node:

<IW_HOME>/bin/infoworks-ha-ansible
Run the following command: ./setup-iw-ha.sh and setup HA for that service again.

NOTE: Ensure that the secondary node IP/hostname is provided as the top most node in the hosts file.

If HA is not setup for any of these services, restart service normally on secondary node.

NOTE: Ensure that postgres master and orchestrator services are running on the same machine.

Infoworks 2.7.x+

Navigate to the <IW_HOME>/bin/infoworks-filesync-ansible folder.
Run the following command: setup-iw-filesync.sh

The required files will be automatically copied to remote hosts using Infoworks tool.

Restart all infoworks services

Stop all services on primary node (ui governor hangman restapi queryservice scheduler df orchestrator cube-engine).
Restart Infoworks services (ui governor hangman restapi queryservice scheduler df orchestrator cube-engine) on secondary node.

Restart Postgres, platform, RabbitMQ

If HA has been setup for one of these services previously, perform the following steps to bring up the service on secondary node:

<IW_HOME>/bin/infoworks-ha-ansible
Run the following command: ./setup-iw-ha.sh and setup HA for that service again.

NOTE: Ensure that the secondary node’s IP/hostname is provided as the topmost node in the hosts file.

If HA is not setup for any of these services, restart service normally on secondary node.

NOTE: Ensure that at any time, only one instance of hangman is running across all nodes.__Ensure that Postgres master and orchestrator services are running on the same machine.

Service Specific Instructions

Rest API

Change RestAPI in calls:

All external calls to Infoworks Rest API from outside Infoworks must change the REST API host to {{seconday_node_ip}}.
Internal calls to Infoworks Rest API from within Infoworks will automatically point to {{seconday_node_ip}}. No action required.

Validation

In secondary node, execute the following command:

$sudo netstat -tulpn | grep 2999

Expected output:

tcp 0 0 0.0.0.0:2999 0.0.0.0:* LISTEN 25920/gunicorn: mas

Ensure that port is listening on 0.0.0.0

DF and QueryServices:

Ensure that dependencies are setup:

Tomcat must be setup on secondary node.

Ensure that JAVA_HOME is set properly in secondary node.
Run the following command: echo $JAVA_HOME
If JAVA_HOME is not set, set it as an environment variable (for example, export JAVA HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el75.x86 64)

Validate whether tomcat is setup correctly:

Run start.sh {{service_name}} on secondary node.
Run on secondary node: ps auwwx | grep catalina.startup.Bootstrap
Expected output: several Java processes are printed.

Orchestrator

If Postgres HA has been setup on the cluster, ensure that orchestrator services are running on the same machine as Postgres master.

Last updated on