Pipeline Executor Preconfiguration

Static configurations used for different execution engines are set in the df_executor_configs.yaml configuration file located in the $IW_HOME/conf folder. This file can be edited for pipelines to work with different versions and configurations of Hive or Spark.

Sample df_executor_configs.yaml File

    
​x
 
#Executor Configurations​---​batchEngine: "SPARK"​udfStatements:​- "DROP FUNCTION IF EXISTS iwrowid"​- "CREATE TEMPORARY FUNCTION iwrowid AS 'io.infoworks.awb.udfs.RowIdUDF'"​- "DROP FUNCTION IF EXISTS iwltrim"​- "CREATE TEMPORARY FUNCTION iwltrim AS 'io.infoworks.awb.udfs.LTrimUDF'"​- "DROP FUNCTION IF EXISTS iwrtrim"​- "CREATE TEMPORARY FUNCTION iwrtrim AS 'io.infoworks.awb.udfs.RTrimUDF'"​- "DROP FUNCTION IF EXISTS iwtrim"​- "CREATE TEMPORARY FUNCTION iwtrim AS 'io.infoworks.awb.udfs.TrimUDF'"​- "DROP FUNCTION IF EXISTS iwuuid"​- "CREATE TEMPORARY FUNCTION iwuuid AS 'io.infoworks.awb.udfs.UuidUDF'"​- "DROP FUNCTION IF EXISTS iwsplit"​- "CREATE TEMPORARY FUNCTION iwsplit AS 'io.infoworks.awb.udfs.SplitUDF'"​- "DROP FUNCTION IF EXISTS iwdateloop"​- "CREATE TEMPORARY FUNCTION iwdateloop AS 'io.infoworks.awb.udfs.LoopUDF'"​- "DROP FUNCTION IF EXISTS iwgenrows"​- "CREATE TEMPORARY FUNCTION iwgenrows AS 'io.infoworks.awb.udfs.GenRowsUDF'"​- "DROP FUNCTION IF EXISTS iwtime_similarity"​- "CREATE TEMPORARY FUNCTION iwtime_similarity AS 'io.infoworks.awb.udfs.TimeSimilarityUDF'"​- "DROP FUNCTION IF EXISTS SeqId"​- "CREATE TEMPORARY FUNCTION SeqId AS 'io.infoworks.awb.udfs.SeqIdUDF'"​propertyStatements:​partitionedTargetProperties:​- "SET hive.exec.dynamic.partition = true"​- "SET hive.exec.dynamic.partition.mode = nonstrict "​- "SET hive.optimize.sort.dynamic.partition=true"​---​batchEngine: "HIVE"​udfStatements:​- "ADD JAR ${iw_hdfs_udfs_path}/df-shared.jar"​- "DROP FUNCTION IF EXISTS iwrowid"​- "CREATE TEMPORARY FUNCTION iwrowid AS 'io.infoworks.awb.udfs.RowIdUDF'"​- "DROP FUNCTION IF EXISTS iwltrim"​- "CREATE TEMPORARY FUNCTION iwltrim AS 'io.infoworks.awb.udfs.LTrimUDF'"​- "DROP FUNCTION IF EXISTS iwrtrim"​- "CREATE TEMPORARY FUNCTION iwrtrim AS 'io.infoworks.awb.udfs.RTrimUDF'"​- "DROP FUNCTION IF EXISTS iwtrim"​- "CREATE TEMPORARY FUNCTION iwtrim AS 'io.infoworks.awb.udfs.TrimUDF'"​- "DROP FUNCTION IF EXISTS iwuuid"​- "CREATE TEMPORARY FUNCTION iwuuid AS 'io.infoworks.awb.udfs.UuidUDF'"​- "DROP FUNCTION IF EXISTS iwsplit"​- "CREATE TEMPORARY FUNCTION iwsplit AS 'io.infoworks.awb.udfs.SplitUDF'"​- "DROP FUNCTION IF EXISTS iwdateloop"​- "CREATE TEMPORARY FUNCTION iwdateloop AS 'io.infoworks.awb.udfs.LoopUDF'"​- "DROP FUNCTION IF EXISTS iwgenrows"​- "CREATE TEMPORARY FUNCTION iwgenrows AS 'io.infoworks.awb.udfs.GenRowsUDF'"​- "DROP FUNCTION IF EXISTS iwtime_similarity"​- "CREATE TEMPORARY FUNCTION iwtime_similarity AS 'io.infoworks.awb.udfs.TimeSimilarityUDF'"​- "DROP FUNCTION IF EXISTS SeqId"​- "CREATE TEMPORARY FUNCTION SeqId AS 'io.infoworks.awb.udfs.SeqIdUDF'"​propertyStatements:​commonProperties:​- "SET hive.enforce.bucketing=true"​- "SET hive.optimize.bucketingsorting=false"​partitionedTargetProperties:​- "SET hive.exec.dynamic.partition=true"​- "SET hive.exec.dynamic.partition.mode=nonstrict"​- "SET hive.optimize.sort.dynamic.partition=true"
Copy

For troubleshooting, see Troubleshooting YAML Executor Configuration.

Last updated on