About This Recipe
This recipe demonstrates how to use the parking lot pattern for handling downstream throttling and errors with an example use case of importing employee bank details from an external system to a downstream payroll system. An Oracle ATP database table is used for implementing the parking lot pattern.
To use the recipe, you must install the recipe and configure the connections and other resources within the recipe. In this example, employee bank details are received from an external system such as Oracle HCM Cloud in a CSV file. The CSV file is then downloaded to a local directory on FTP Server or File Server. Subsequently, the employees bank details are registered in batches to an Oracle ATP database table and parked there for a specific time period before further processing. Thus, the integration flow has a chance to throttle the number of batches processed concurrently. The ready batches are picked up in a staged manner for processing and the processed data is finally updated to a downstream payroll system.
Overview of the recipe's integration flows:
- The Request Persister integration (SR_BulkDownload_RequestPersister_ATP) reads the CSV file received from external system and writes it to a local input directory in FTP. It reads the batch configurations (such as, group_id and group_type) and writes the metadata to the Oracle ATP database table. The registered batches are parked in the Parking Lot table in ATP for a specific period of time.
- The Scheduled Dispatcher integration (SR_ScheduledDispatcher_CSVBatch) is scheduled to run at required frequency. At every run, it fetches the batch requests from the Parking Lot table in ATP and dispatches them to the Async Batch Processor integration for processing. Thus, it triggers the Async Bach Processor integration to process the batches that are submitted for processing.
- The Async Processor integration (SR_OneWay_Processor_HCM_To_Payroll) processes all the submitted batch requests. Also, it updates the batch status (for example, if a batch is successfully processed, it updates the status to PROCESSED) in the Batch Statistics table in ATP. On successful processing, it invokes another integration (for example, SR_MOCK_EmpBankAccountProvider integration in this recipe ) to send the processed information to a downstream application via REST API calls. Any errors received in the response from the downstream application are written into an Error folder in FTP. These payload errors are also written to a Payload Error Record table in ATP.
- The Resubmission Processor integration (SR_ScheduledDispatcher_PayrollErrors) is used to resubmit any batch requests failures/errors that were fixed. It fetches any record from the Payload Error Record table that are in READY state and re-submits them for processing.
System and Access Requirements
-
Oracle Integration 3
-
A secure FTP (sFTP) server or File Server
-
An FTP client to access the sFTP server
-
Oracle ATP
-
An account on Oracle ATP with the Administrator role
Recipe Schema
This section provides an architectural overview of the recipe.
Description of the illustration parking-lot-pattern.png
A CSV file with employee bank details from an upstream application such as Oracle HCM Cloud is downloaded to an FTP Server or File Server. The first integration flow (Request Persister) in Oracle Integration registers the batches in the ATP database. Each batch request is parked in the ATP database parking lot table for a certain period of time so the integration flow has a chance to throttle the number of batches processed concurrently. The second scheduled integration flow (Scheduled Dispatcher) in Oracle Integration fetches the batches from the parking lot table in ATP at a date and time of your choice, and dispatches the batches for further processing. The second integration triggers the third integration flow (Async Processor) which processes the dispatched batches. The processed batches are then sent to a downstream application such as a Payroll system via a REST API call. Any errors that are received as response from the downstream application are rectified and then resubmitted for processing by the fourth integration flow (Resubmission Processor) in Oracle Integration.