Pipeline Executor Preconfiguration

Static configurations used for different execution engines are set in the df_executor_configs.yaml configuration file located in the $IW_HOME/conf folder. This file can be edited for pipelines to work with different versions and configurations of Hive or Spark.

Sample df_executor_configs.yaml File

#Executor Configurations

---

batchEngine: "SPARK"

udfStatements:

- "DROP FUNCTION IF EXISTS iwrowid"

- "CREATE TEMPORARY FUNCTION iwrowid AS 'io.infoworks.awb.udfs.RowIdUDF'"

- "DROP FUNCTION IF EXISTS iwltrim"

- "CREATE TEMPORARY FUNCTION iwltrim AS 'io.infoworks.awb.udfs.LTrimUDF'"

- "DROP FUNCTION IF EXISTS iwrtrim"

- "CREATE TEMPORARY FUNCTION iwrtrim AS 'io.infoworks.awb.udfs.RTrimUDF'"

- "DROP FUNCTION IF EXISTS iwtrim"

- "CREATE TEMPORARY FUNCTION iwtrim AS 'io.infoworks.awb.udfs.TrimUDF'"

- "DROP FUNCTION IF EXISTS iwuuid"

- "CREATE TEMPORARY FUNCTION iwuuid AS 'io.infoworks.awb.udfs.UuidUDF'"

- "DROP FUNCTION IF EXISTS iwsplit"

- "CREATE TEMPORARY FUNCTION iwsplit AS 'io.infoworks.awb.udfs.SplitUDF'"

For troubleshooting, see Troubleshooting YAML Executor Configuration.

Last updated on