To change Spark work directory:

1. Stop Spark service:
   1. sudo systemctl stop spark-slave.service
   2. sudo systemctl stop spark-master.service
2. Create work directory under u02:
   1. Navigate to /u02
   2. sudo mkdir spark. Here spark is the work folder name, you can create folder with the name of your choice.
   3. chmod 777 spark, to change permission.
3. Edit spark-env.sh
   1. Navigate to SPARK_HOME/conf, and edit spark-env.sh
   2. Add SPARK_WORKER_DIR=/u02/spark at the end of the file spark-env.sh to point to newly created folder under /u02
4. Start Spark service:
   1. sudo systemctl start spark-master.service
   2. sudo systemctl start spark-slave.service

   You will see the application and driver data ( files and logs) under /u02/spark when you publish the pipeline again.