About Pipeline Parameters
The pipeline parameters apply to all functional areas.
- Data Refresh Schedule: Specify the frequency and when you want the incremental data load to happen. While specifying the timezone, the recommendation is to use city names to handle the daylight savings. For example, instead of selecting timezone such as EST or PST, select Europe/Bucharest or America/Los_Angeles. In this case, the data refresh process calculates the value mentioned in the Time field based on the local time irrespective of daylight savings.
-
Initial Extract Date: Initial extract date is used when you extract data for a full load. After extracting the data for a functional area, avoid changing the initial extract date. If you need to change the initial extract data, then after changing the date, reset the data warehouse and reactivate the functional areas. See Reset the Data Warehouse.
You can specify an absolute date from which to load the transaction data or a relative period to load transaction data within the effective period. The system uses the absolute date to process and load transactional data created after the initial extract date to the warehouse. It reduces the initial data load volume, however, with time the historical data volume can keep on increasing. In such a scenario, specify a relative period because with this approach, you can refresh data for the moving window of the specified period. For example, if the relative period is 3 years, the effective extract date moves every day to consider only data for 3 years. As part of application upgrade, the system cleans up the data warehouse based on the relative initial extract date that you specify. The Relative Initial Extract Date option enables you to store only the required amount of data in the data warehouse; thereby improving performance and avoiding additional costs due to large amount of historical data in the data warehouse.