SuiteCloud Processors Priority Scheduling Examples

The two examples in this section, Default Priority Scheduling Example and Varying Priority Scheduling Example, each show three diagrams. The two sets of diagrams compare the same 18 jobs as they are handled by:

Pre-2017.2 Scheduling Queues
2017.2 SuiteCloud Processors: All queues are removed
2017.2 SuiteCloud Processors: Some queues are removed; jobs 1 -3, 15, and 18 still use queues

This table shows the duration of each job submitted. As demonstrated in the examples, the processing environment has no impact on the amount of time required to complete each job after processing starts.

Note:

Each map/reduce stage is handled by a minimum of one job. Therefore, the duration listed for map/reduce jobs is per stage. For example, the reduce stage jobs 11 and 12 have a combined duration of 5. This means that the sum of the duration of jobs 11 and 12 is 5. When one of the jobs is canceled or not created, the combined duration becomes the duration of the remaining job from the pair.

Job	Duration
1	5
2	2
3	2
4	6
5 +6	2
7 +8	1
9 +10	5
11 +12	5
13 +14	2
15	3
16	3
17	3
18	2

Default Priority Scheduling Example

This example assumes default priority settings (standard priority) and default preferences. It also assumes that each job executes in full without yielding.

The map/reduce deployment in the first diagram has a Queues value of 2 and 3. The map/reduce deployment in the second and third diagrams has a Concurrency Limit value of 2.

Note:

With the introduction of SuiteCloud Processors, the Queues field is replaced with the Concurrency Limit field on map/reduce deployments.

Diagram Key

The black circles are scheduled script jobs. Each scheduled script is processed by a single job.
The red circles are map/reduce jobs. In this example, all of the map/reduce jobs belong to a single map/reduce task. The map/reduce jobs include a label that indicates the map/reduce stage being processed.
The jobs in columns labeled Q1, Q2, and Q3 are for deployments that still use queues. The jobs in columns without a label are for deployments that no longer use queues.
P1, P2, and P3 represent the processors that are available in the processor pool.

Note:
Although processors are labeled for this example, they are not technically individual entities. Also, a real account would never have only three processors available to it. The minimum number of processors available to an account with SuiteCloud Plus is listed at SuiteCloud Plus Settings.
The numbers within each circle represent the time stamp on the job submission, with 1 being the first job submitted. The example also uses this number as a unique identifier for each job.
Arrows represent dependencies. For example, in the Scheduling Queues diagram, job 2 cannot start until job 1 is complete.
- Black arrows represent queue dependencies. Jobs within a queue must be processed in the order they were submitted.
- Red arrows represent dependencies imposed by the map/reduce algorithm.
- Green arrows indicate access to processors.

Scheduling Queues Only

SuiteCloud Processors – All Queues Removed

SuiteCloud Processors – Some Queues Removed

An example of the SuiteCloud Processors default priority scheduling.

SuiteCloud Processors Priority Scheduling Examples section of the SuiteCloud Processors page

Time Slot	Scheduling Queues Only	SuiteCloud Processors – All Queues Removed	SuiteCloud Processors – Some Queues Removed
101	Jobs 1, 4, and 6 start.	Jobs 1, 2, and 3 start.	Jobs 1, 4, and 5 start.
102	Job 6 cancels job 5. The getInput stage cannot be processed by multiple jobs, so the first getInput job to start (job 6) automatically cancels all others in the queues.	—	—
103	Job 6 completes. The next job in Q3, job 8, starts.	Jobs 2 and 3 complete. Jobs 4 and 5 start.	Job 5 completes. There is no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.
104	Job 8 completes. The map stage can be processed by multiple jobs. But since the other map stage job, job 7, is unable to start (the job utilizing its processor, job 4, is not yet complete), job 8 completes the entire map stage. Job 7 is no longer needed, so it is canceled. The next job in Q3, job 10, starts.	—	Job 7 completes. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Job 9 starts.
105	Job 10 cancels job 9. The shuffle stage cannot be processed by multiple jobs, so the first shuffle job to start (job 10) automatically cancels all others in the queues.	Job 5 completes. There is no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.	—
106	Job 1 completes. The next job in Q1, job 2, starts.	Job 7 completes. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Job 1 completes. Jobs 9 and 15 start. Jobs 11 -13 are dependent on job 9. There is no job 10 or job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs and jobs 13 and 14 are both summarize stage jobs. The shuffle and summarize stages cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple shuffle and summarize stage jobs are no longer needed.	Job 1 completes. The next job in Q1, job 2, starts.
107	Job 4 completes. Q2 is blocked. The next job in Q2, job 11, cannot start. It is dependent on job 10, and job 10 is not complete.	—	Job 4 completes. The first job in Q3, Job 15, starts.
108	Job 2 completes. The next job in Q1, job 3, starts.	—	Job 2 completes. The next job in Q1, job 3, starts.
109	Job 10 completes. Job 11 can now start and Q2 is no longer blocked. The next job in Q3, job 12, starts. These are both reduce stage jobs. The reduce stage can be processed by multiple jobs.	Jobs 4 and 15 complete. Jobs 16 and 17 start.	Job 9 completes. There is no job 10 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs. The shuffle stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple shuffle stage jobs are no longer needed. Job 11 starts.
110	Job 3 completes. The next job in Q1, job 16, starts.	—	Jobs 3 and 15 complete. There are no additional jobs to process in Q1. Jobs 12 and 16 start.
111	Job 11 completes. Q2 is blocked. The next job in Q2, job 13, cannot start. It is dependent on job 12, and job 12 is not complete.	Job 9 completes. Job 11 starts.	—
112	Job 12 completes. Job 13 can now start and Q2 is no longer blocked. The next job in Q3, job 14, can also start. But these are both summarize stage jobs, and the summarize stage cannot be processed by multiple jobs. Job 13 starts first, so job 14 is canceled. Since job 14 is canceled, the next job in Q3, job 15, starts	Jobs 16 and 17 complete. Jobs 12 and 18 start.	Jobs 11 and 12 complete. Jobs 13 and 17 start.
113	Job 16 completes. The next job in Q1, job 17, starts.	—	Job 16 completes. The next job in Q3, Job 18, starts.
114	Job 13 completes. There are no additional jobs to process in Q2.	Jobs 11, 12, and 18 complete. Job 13 starts	Job 13 completes. There is no job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 13 and 14 are both summarize stage jobs. The summarize stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple summarize stage jobs are no longer needed.
115	Job 15 completes. The next job in Q3, job 18, starts.	—	Jobs 17 and 18 complete. There are no additional jobs to process.
116	Job 17 completes. There are no additional jobs to process in Q1.	Job 13 completes. There are no additional jobs to process.	—
117	Job 18 completes. There are no additional jobs to process in Q3.	—	—

Varying Priority Scheduling Example

This example assumes default preferences. It also assumes that each job executes in full without yielding.

The map/reduce deployment in the first diagram has a Queues value of 2 and 3. The map/reduce deployment in the second and third diagrams has a Concurrency Limit value of 2.

Note:

With the introduction of SuiteCloud Processors, the Queues field is replaced with the Concurrency Limit field on map/reduce deployments.

The second and third diagrams vary from Default Priority Scheduling Example as follows:

In the second diagram, jobs 15 and 18 are high priority. Job 4 is low priority. The remaining jobs are standard priority.
In the third diagram, job 4 is low priority. Jobs 15 and 18 in the third diagram are standard priority.

Diagram Key

The black circles are scheduled script jobs. Each scheduled script is processed by a single job.
The red circles are map/reduce jobs. In this example, all of the map/reduce jobs belong to a single map/reduce task. The map/reduce jobs include a label that indicates the map/reduce stage being processed.
The circles with blue fill are high priority. The circles with gray fill are low priority. The circles with white fill are standard priority.
The jobs in columns labeled Q1, Q2, and Q3 are for deployments that still use queues. The jobs in columns without a label are for deployments that no longer use queues.
P1, P2, and P3 represent the processors that are available in the processor pool.

Note:
The processors are labeled for this example. Although technically, processors are not distinguished as individual entities. Also, a real account would never have only three processors available to it. The minimum number of processors available to an account with SuiteCloud Plus is listed at SuiteCloud Plus Settings.
The numbers within each circle represent the time stamp on the job submission, with 1 being the first job submitted. The example also uses this number as a unique identifier for each job.
Arrows represent dependencies. For example, in the Scheduling Queues diagram, job 2 cannot start until job 1 is complete.
- Black arrows represent queue dependencies. Jobs within a queue must be processed in the order they were submitted.
- Red arrows represent dependencies imposed by the map/reduce algorithm.
- Green arrows indicate access to processors.

Scheduling Queues Only

SuiteCloud Processors – All Queues Removed

SuiteCloud Processors – Some Queues Removed

An example of the SuiteCloud Processors varying priority scheduling.

Time Slot	Scheduling Queues Only	SuiteCloud Processors – All Queues Removed	SuiteCloud Processors – Some Queues Removed
101	Jobs 1, 4, and 6 start.	Jobs 15, 18, and 1 start. Jobs 15 and 18 are designated as high priority so they are assigned to processors first (in the order of submission). Job 1 is the first standard priority job submitted so it is the next job assigned to a processer.	Jobs 1, 5, and 15 start. Job 4 is designated low priority, so it is processed after all other available standard priority jobs. All other standard priority jobs submitted before job 15 have dependencies that are blocking them.
102	Job 6 cancels job 5. The getInput stage cannot be processed by multiple jobs, so the first getInput job to start (job 6) automatically cancels all others in the queues.	—	—
103	Job 6 completes. The next job in Q3, job 8, starts.	Job 18 completes. Job 2 starts.	Job 5 completes. There is no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.
104	Job 8 completes. The map stage can be processed by multiple jobs. But since the other map stage job, job 7, is unable to start (the job utilizing its processor, job 4, is not yet complete), job 8 completes the entire map stage. Job 7 is no longer needed, so it is canceled. The next job in Q3, job 10, starts.	Job 15 completes. Job 3 starts.	Jobs 7 and 15 complete. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Jobs 9 and 16 start.
105	Job 10 cancels job 9. The shuffle stage cannot be processed by multiple jobs, so the first shuffle job to start (job 10) automatically cancels all others in the queues.	Job 2 completes. Job 5 starts. Job 4 is designated low priority, so it is processed after all other available standard priority jobs.	—
106	Job 1 completes. The next job in Q1, job 2, starts.	Jobs 1 and 3 complete. Jobs 16 and 17 start.	Job 1 completes. The next job in Q1, job 2, starts.
107	Job 4 completes. Q2 is blocked. The next job in Q2, job 11, cannot start. It is dependent on job 10, and job 10 is not complete.	Job 5 completes. There is no job 6 with SuiteCloud Processors. In the Scheduling Queues example, jobs 5 and 6 are both getInputstage jobs. The getInput stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple getInput stage jobs are no longer needed. Job 7 starts.	Job 16 completes. Job 17 starts.
108	Job 2 completes. The next job in Q1, job 3, starts.	Job 7 completes. Since job 7 completes the entire map stage, job 8 is no longer needed. Job 8 is canceled. Job 9 starts.	Job 2 completes. The next job in Q1, job 3, starts.
109	Job 10 completes. Job 11 can now start and Q2 is no longer blocked. The next job in Q3, job 12, starts. These are both reduce stage jobs. The reduce stage can be processed by multiple jobs.	Jobs 16 and 17 complete. Job 4 starts. Job 4 is designated as low priority, but all the remaining standard priority jobs are dependent on job 9.	Job 9 completes. There is no job 10 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs. The shuffle stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple shuffle stage jobs are no longer needed. Job 11 starts.
110	Job 3 completes. The next job in Q1, job 16, starts.	—	Jobs 3 and 17 complete. There are no additional jobs to process in Q1. Job 12 starts. The next job in Q3, Job 18, starts.
111	Job 11 completes. Q2 is blocked. The next job in Q2, job 13, cannot start. It is dependent on job 12, and job 12 is not complete.	—	—
112	Job 12 completes. Job 13 can now start and Q2 is no longer blocked. The next job in Q3, job 14, can also start. But these are both summarize stage jobs, and the summarize stage cannot be processed by multiple jobs. Job 13 starts first, so job 14 is canceled. Since job 14 is canceled, the next job in Q3, job 15, starts	—	Jobs 11, 12 and 18 complete. There are no additional jobs to process in Q3. Jobs 13 and 4 start.
113	Job 16 completes. The next job in Q1, job 17, starts.	Job 9 completes. There is no job 10 with SuiteCloud Processors. In the Scheduling Queues example, jobs 9 and 10 are both shuffle stage jobs. The shuffle stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple shuffle jobs are no longer needed. Jobs 11 and 12 start.	—
114	Job 13 completes. There are no additional jobs to process in Q2.	—	Job 13 completes. There is no job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 13 and 14 are both summarize stage jobs. The summarize stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple summarize stage jobs are no longer needed.
115	Job 15 completes. The next job in Q3, job 18, starts.	Jobs 4 and 12 complete.	—
116	Job 17 completes. There are no additional jobs to process in Q1.	Job 11 completes. Job 13 starts.	—
117	Job 18 completes. There are no additional jobs to process in Q3.	—	—
118	—	Job 13 completes. There is no job 14 with SuiteCloud Processors. In the Scheduling Queues example, jobs 13 and 14 are both summarize stage jobs. The summarize stage cannot be processed by multiple jobs. Since this map/reduce deployment is not using queues, multiple summarize stage jobs are no longer needed. There are no additional jobs to process.	Job 4 completes. There are no additional jobs to process.

Default Priority Scheduling Example

Diagram Key

Varying Priority Scheduling Example

Diagram Key

Related Topics