To change Spark log rollover based on time:

1. Stop Spark service:
   1. sudo systemctl stop spark-slave.service
   2. sudo systemctl stop spark-master.service
2. Edit spark-env.sh
   1. Navigate to SPARK_HOME/conf, and comment or delete SPARK_WORKER_OPTS variable and its value.
3. Edit spark-defaults.conf by adding the below lines:
   • spark.executor.logs.rolling.maxRetainedFiles 7
   • spark.executor.logs.rolling.strategy time
   • spark.executor.logs.rolling.time.interval minutely

   Note:

   You can change spark.executor.logs.rolling.time to daily, hourly, minutely. This is to enable log rollover based on time.
4. Start Spark service:
   1. sudo systemctl start spark-master.service
   2. sudo systemctl start spark-slave.service

   You will see the application and driver data ( files and logs) under /u02/spark when you publish the pipeline again.