Configuration Options for Handling Map/Reduce Interruptions
A map/reduce script can be interrupted at any time. If the application server is disrupted, it'll immediately stop the script's execution. Additionally, an uncaught error, although it does not cause the script to stop executing, stops the current function invocation, even if it's not complete.
The system has two configuration options to fine-tune your script's response to interruptions. For details, see the following sections:
retryCount
The retryCount option only applies to map and reduce stages. This option lets you configure your script to restart the map or reduce function for any key/pairs that were left in an uncertain state following an interruption. including application server restarts and uncaught errors.
The effect of this setting varies slightly depending on whether the script was disrupted by application server restart or an uncaught error. For example:
-
When an application server restart occurs, the script cannot identify the exact key-value pair that was being processed when the interruption occurred. However, the script can identify the pairs that had been flagged for processing but were not yet marked as complete. In the event of a retry, the script retries processing for all of these pairs.
-
When an uncaught error occurs, the script can identify the exact key-value pair that was being processed when the interruption occurred. When you're using the retryCount option, the script invokes the map or reduce function again for that specific key-value pair.
Valid values are 0 to 3. The number you choose for the retryCount setting is the number of retries permitted for each key-value pair that was left in an uncertain state after an interruption. For example, suppose you have retryCount set to 2, and an error interrupts a map job. In this case, the script would restart the function for the key-value pair that was being processed when the error was thrown. If the second function invocation for that pair was also interrupted by an error, the script would invoke the function for that key-value pair another time. However, if an error was thrown during the third attempt, the script would not retry processing again, because the retryCount setting permits only two retries. When the job moves on to a new key-value pair, two retries are available again.
The retryCount setting is optional. You can set a value for it in your map/reduce script's return statement, as described in Adding a Configuration Option to a Script.
If you don't use retryCount, the behavior differs depending on whether the script was disrupted by an error or an application server restart. For example:
-
When retryCount isn't used and an application server restart interrupts the script, the system restarts processing for all key-value pairs that were left in an uncertain state.
-
When retryCount isn't used and an error interrupts the map or reduce function, the system does not restart processing for the key-value pair that was left in an uncertain state.
See also System Response After an Uncaught Error and System Response After an Application-Server Disruption.
When an uncaught error occurs, the system's response also depends on the exitOnError option's value. However, the script evaluates and reacts to the retryCount setting first.
exitOnError
exitOnError is used when an uncaught error occurs in the map or reduce stage. It's evaluated after the retryCount option.
When exitOnError is set to true, the script exits the stage after both of the following occur:
-
An uncaught error occurs.
-
All retries permitted by the retryCount option have been used.
This setting has no impact on the system’s behavior after an application server restart. It's used only when an uncaught error is thrown.
The exitOnError setting is optional. You can set its value by adding it to the return statement in your map/reduce script, as described in Adding a Configuration Option to a Script. Possible values are:
-
false — The script continues processing in the current stage. (This is also the behavior used when the option isn't added to the script.)
-
true — The script exits the stage and goes immediately to the summarize stage.
See also System Response After an Uncaught Error.
Adding a Configuration Option to a Script
If you want to use the retryCount or exitOnError option, add them to the script’s return statement in a config block. For example:
// Add additional code.
...
return {
config:{
retryCount: 3,
exitOnError: true
},
getInputData: myGetInputData,
map: myMap,
reduce: myReduce,
summarize: mySummarize
};
...
// Add additional code.