Hive ML Pipeline Failure in HDP 3.1

Issue

Exception occurs when Advanced Analytics Nodes are used in Hive pipelines.

Cause

Caused by the following exception:

java.util.concurrent.ExecutionException: java.lang.NumberFormatException: For input string: “0000001_0000”

Solution

For HDP 3.1, the following configuration must be added to pipeline advanced configuration, for ML nodes to work:

• Key: df_batch_hive_settings

• value: hive.strict.managed.tables=false; hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager; hive.optimize.index.filter=false; hive.compactor.initiator.on=false; hive.compactor.worker.threads=0; hive.metastore.create.as.acid=false; hive.create.as.insert.only=false;

NOTE: This procedure is applicable for Infoworks DataFoundry versions above 2.9.0.