Issue
Configuring Infoworks to make Hive connections via Zookeeper.
Target Audience
- Infoworks Admin
- Hadoop Cluster Admin
Cause
Infoworks is configured to make direct connection to a single Hiveserver host using port 10000. So, the ingestion and pipeline jobs depend on this Hiveserver running. HDP clusters include an option to avail HA for Hive through Zookeeper. In this setup, if one Hiveserver is down, Zookeeper routes to another server that is running.
Solution
Following are the steps to configure Infoworks with Zookeeper:
- Configure Infoworks to connect to Hive via Zookeeper URL instead of the Hiveserver URL.
Following is the Zookeeper URL format:
jdbc:hive2://< ZOOKEEPER-QUORUM> /;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Sample URL:
jdbc:hive2://zk0-infowo.rvsskpbly.gx.internal.cloudapp.net:2181,zk1-infowo.vslebpd5rmbefg1anloj6969nica.gx.internal.cloudapp.net:2181,zk4-infowo.vslebpd5rmbefg1anloj12nica.gx.internal.cloudapp.net:2181/default; serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
- The part of URL in bold is the Zookeper quorum and the other part is the extra information. These two must be separated for IW, so that the URL is created properly.
- These properties must be set in the Admin page as follows:
hiveConnectionExtraProperties=serviceDiscoveryMode=zooKeeper; zooKeeperNamespace=hiveserver2
hive=jdbc:hive2://zk0-infowo.rvsskpbly.gx.internal.cloudapp.net:2181,zk1-infowo.vslebpd5rmbefg1anloj12nica.gx.internal.cloudapp.net:2181,zk4-infowo.vslebpd5rmbefg1anloj12nica.gx.internal.cloudapp.net:2181/default
Applicable Versions
- 2.3.X and above