Spark Pipeline Issues

Issue

The SparkContext ClassNotFound Exception is displayed.

Cause

This issue can be due to jars missing from classpath.

Solution

To run spark pipeline, classpath must contain spark jars.

Following are the steps to add spark jobs to classpath.

• Run the following command: source $IW_HOME/bin/env.sh

• Open the conf.properties file from the $IW_HOME/conf folder.

• Search for df_batch_classpath and add $SPARK_HOME/jars/* to the list of jars.

• Insert <SPARK_HOME>/jars/antlr-4.7.jar:<rest of the classpath> at the beginning of the df_batch_classpath and df_tomcat_classpath values.

For Spark 2.3, perform the following:

• Add the antlr jar at the beginning of the classpath.

• Add the jackson-databind jar at the beginning of the dt_interactive_classpath value. For example, <SPARK_HOME>/jars/jackson-databind-2.6.7.1.jar:<classpath>.