Data Transformation
How can we view the code/query submitted for Spark pipelines running on the cluster?
In Spark, the query cannot be printed since data frame is used to submit the job. The plan is printed in the logs for all the transformations. Also, you can view the logical plan in SQL Tab in the Spark UI.