Map/Reduce Script Error Handling

A map/reduce script can be interrupted by:

Interruptions can leave parts of the script's work unfinished. The system has measures to handle this issue.

After an application-server disruption, the system automatically restarts the script, regardless of which stage was in progress when the failure occurred. For instance, if the failure occurred during the map or reduce stage, the map or reduce job restarts. In this case, the default behavior is that the restarted job invokes the map or reduce function again for all key-value pairs that were previously flagged for processing but not yet marked complete. You can modify this behavior using the retryCount option.

After an uncaught error is thrown, the behavior varies depending on the stage. If an error occurs during getInputData, the script moves to the summarize stage and doesn't restart getInputData. If an uncaught error is thrown during the map or reduce stage, the default behavior is that the current function invocation ends, even if incomplete, and the map or reduce job moves on to other key-value pairs that require processing. You can configure the script to re-run the map or reduce function for the pair that caused the error. You manage the system’s behavior in this case by using the retryCount and exitOnError options.

To fully understand the system response to interruptions, see System Response After a Map/Reduce Interruption.

In general, your script should include logic that checks to see whether a restart has occurred. If the script has been restarted, it should prevent duplicate processing. For more information, see Adding Logic to Handle Map/Reduce Restarts.

For additional explanation of how each stage responds to a restart, see Execution of Restarted Map/Reduce Stages.

Related Topics

General Notices