Workflow Configuration Migration
For a successful workflow configuration migration, the sources and tablegroups, pipelines and domains, cubes and domains in the exported file must exactly match the names in the environment where the workflow configuration is being imported. This feature:
- Allows you to map tablegroup to any tablegroup in the source.
- Allows you to map pipeline to any pipeline in the domain.
- Allows you to map cube to any cube in the domain.
Exporting Workflow Configuration
To export a workflow configuration, follow these steps:
- Click the Domains menu and click the domain where the workflow is created.
- On the Summary page of the domain, click the workflow for which configuration must be migrated.
- Click the Settings icon.
- On the Settings page, locate the Configuration Migration section as shown below:
- Click Export. The workflow configuration gets downloaded.
Importing Workflow Configuration
To import pipeline configuration, follow these steps:
- Click the Domains menu and click the domain where the workflow is created.
- On the Summary page of the domain, click the workflow for which configuration must be migrated.
- Click the Settings icon.
- In the Settings page, locate the Configuration Migration section.
- Click Browse and select the required workflow configuration file.
- Click Upload. The following page is displayed:
This page displays the tablegroup, pipeline, cube mapping between the imported file and the tablesgroups in the source, and pipeline and cube in the domain. If a tablegroup, pipeline or cube mapping is not available, an error is displayed next to the mapping.
- Select the required tablegroup from the Tablegroup in this Source drop-down list to map it with the tablegroup in the uploaded configuration.
- Select the required pipeline from the Pipeline in this Domain drop-down list to map it with the pipeline in the uploaded configuration.
- Select the required cube from the Cubes in this Domain drop-down list to map it with the cube in the uploaded configuration.
- Click Import Configurations. The screen with a success message is displayed.
Limitation
- Importing a workflow which is already opened in a workflow editor is not supported.
- Modifying the design of workflow or importing workflow configuration with a different design when an instance of the workflow is already running might lead to intermittent failures.
- Existing workflow metrics for all workflow runs prior to 2.6.0 will not be available.
Orchestrator Log Files Cleanup
-
Stop the orchestrator services using the following command:
$W_HOME/bin/stop.sh orchestrator
- Ensure that no orchestrator services are running using the following command:
ps -ef | grep airflow``ps -ef | grep orchestrator - If the processes is still running, use the following command to kill the process:
pkill -f airflow; pkill -f orchestrator
Cleanup the log directories using the following commands:
- To delete the complete logs for all historical dates:
rm -rf $W_HOME/orchestrator-engine/logs/scheduler/* - To delete the logs for specific historical dates:
rm -rf $W_HOME/orchestrator-engine/logs/scheduler/<date folders to remove> - Cleanup the log files using the following commands:
rm -rf $W_HOME/orchestrator-engine/airflow-scheduler.log``rm -rf $W_HOME/orchestrator-engine/airflow-scheduler.out