How to Read Hive Table Data Ingested in Parquet Format through Spark Shell

Issue

Reading Hive table data ingested in parquet format through Spark shell.

Cause

Infoworks can ingest data in Parquet format in Hive. The data will be stored recursively in nested HDFS directories and if user tries to read the data through Spark shell, no results will be displayed.

Solution

To read the hive table data stored in recursive directories in HDFS through Spark shell, you must set the following configurations in the df_spark-defaults.conf file in the $IW_HOME/conf directory and then run the Spark shell command:

spark.sql.hive.convertMetastoreParquet false spark.sql.parquet.writeLegacyFormat true spark.mapreduce.input.fileinputformat.input.dir.recursive true spark.hive.mapred.supports.subdirectories true spark.mapred.input.dir.recursive true spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive true spark.sql.crossJoin.enabled true

Spark Shell Command

spark-shell -- properties-file <absolute_path_for_df_spark-defaults.conf file>


Next to read:
How to Set HDFS Path for Pipeline

For more details, refer to our Knowledge Base and Best Practices!

For help, contact our support team!

(C) 2015-2022 Infoworks.io, Inc. and Confidential

On This Page
How to Read Hive Table Data Ingested in Parquet Format through Spark ShellIssueCauseSolution