Schedule Frequent Data Refreshes for Functional Areas

You can schedule frequent refreshes of the applicable functional areas, prioritized datasets, and warehouse tables to ensure that the incremental data in these is the latest.

About Scheduling Frequent Data Refreshes

For Oracle NetSuite Analytics Warehouse, you can schedule frequent data refreshes for functional areas and warehouse tables using the Frequent Data Refresh Schedule tab on the Pipeline Settings page.

The frequent data refresh capability is designed for refreshing primarily transactional data; not for a high volume of derived datasets that require aggregations, snapshots, or complex transformation logic. When you refresh certain functional areas and warehouse tables, be sure you understand which tables you can refresh or not because the selective refresh of some tables could lead to functional inconsistencies when combining data from multiple subject areas.

When you select the functional areas for a frequent refresh, you can't refresh the data pipeline for the applicable functional area using the Refresh Data option on the Data Configuration page. Oracle NetSuite Analytics Warehouse processes the data augmentations as part of the incremental loads. If you change a data augmentation after including it in the frequent data refresh schedule, then you must remove that data augmentation and let the next incremental refresh finish. Otherwise, the frequent data refresh might fail. After the incremental refresh is complete, you can add the updated data augmentation back to the frequent data refresh schedule.

Frequent Data Refresh Scenarios

Here are few examples of situations when you would want to run frequent data refreshes:

  • During the last five days of the month or at the end of every quarter, general ledger data changes quickly with updated entries happening more than once a day. In this scenario, a financial analyst wants to add the most recent and accurate data during the day into the analysis and report generation process. The frequency of refresh needs to be every 4 hours, or 6 times a day.
  • A retailer has strict guidelines for returns or refunds on any purchase orders. They need the most recent order data for computation and report generation at the beginning of a fiscal quarter. In this example, to account for time differences between Europe, East Coast, and West Coast of the U.S., a financial analyst needs to refresh data every 8 hours, or 3 times a day.

Performance Considerations for Frequent Data Refresh

Review and consider the following to ensure that frequent data refreshes work as expected.

Performance of frequent data refreshes depends on the following:
  • Size of data.
  • Data change such as what data has changed, and which pipeline gets triggered.
  • Number of extracted records that may result in very different number of published records. For example, in one scenario, 44 extracted records resulted in 1060 published records in 70 minutes and 395 extracted records resulted in 55 published records in 35 minutes.

The frequent data refresh process isn't executed in the following scenarios:

  • In the 180-minute window before the scheduled start of the daily incremental data refresh.
  • If any release upgrade is in progress.
  • Until the previous frequent data refresh process is completed. You can set a maximum frequency of 1 hour. If it takes more than 1 hour to complete the refresh, the next frequent data refresh process starts at the next hour.

Follow these guidelines to configure frequent data refresh of warehouse tables; that is, those in datasets:

  • Ensure that you know the exact names of the datasets to refresh.
  • Specify no more than 20 datasets for each run.
  • Determine any dependencies and include the applicable tables because dependencies aren't automatically included.

Schedule Frequent Refreshes of Data with Redesigned Refresh Mechanism (Preview)

Utilizing a redesigned data refresh mechanism, named Frequent Data Refresh V2, you can schedule frequent data refreshes of the applicable modules (functional areas and custom data configurations) and warehouse tables to refresh the data during the day at a higher frequency in between incremental data refreshes.

Topics:

About Scheduling Frequent Data Refreshes Using a New Refresh Mechanism (Preview)

Utilizing the redesigned frequent data refresh mechanism, you can schedule frequent data refreshes of the applicable modules (functional areas and custom data configurations) and warehouse tables to refresh the data during the day at a higher frequency in between incremental data refreshes.

The new frequent data refresh mechanism is called Frequent Data Refresh V2. During refresh, Frequent Data Refresh V2 doesn't update the base tables. Instead, it utilizes “Delta” tables to store the data that has changed and “Lambda” views for data that incorporates base data from incremental data refresh and the changes up to the point when the frequent data refresh happened. See Frequent Data Refresh Delta Tables and Lambda View Details (Preview).

Frequent Data Refresh V2 updates data in warehouse tables more efficiently, enabling potentially higher frequency data refreshes. The frequent data refresh process automatically adds the FDR Flag to the relevant subject areas. To include the frequently refreshed data in your reports, filter the data using the FDR Flag.

With Frequent Data Refresh V2, you can select applicable modules to refresh applicable functional areas and custom data configurations. For the selected modules, data is refreshed for all the required tables, and the Frequent Data Refresh flag is added in the relevant subject areas. You can also select specific tables, including data augmentation tables, to refresh. You may need to make changes to these tables if you want to use them in your customizations. See Manage Frequently Refreshed Data in the Workbooks.

Note:

After release Platform 25.R3.P5, the Enable Features page won't display options to enable or disable “Frequent Refresh” and “Frequent Refresh Tables” features. You can continue using the original version of the Frequent Data Refresh capability or upgrade to Frequent Data Refresh V2. For more information on upgrading to Frequent Data Refresh V2, see Upgrade to Frequent Data Refresh V2 (Preview).
Frequent Data Refresh Delta Tables and Lambda View Details (Preview)

The redesigned refresh mechanism utilizes “Delta” tables to store the data that has changes and “Lambda” views for data that incorporates base data from incremental data refresh and the changes up to the point when the Frequent Data Refresh happened

With Frequent Data Refresh V2, you'll see the following tables in the warehouse. If you plan to use these tables directly for any downstream processing such as custom ETL, you'll need to understand the schema of these tables and what data they contain.
  • Base Table – Data as of the last incremental refresh. This table is unchanged each time frequent data refresh runs.
  • Delta Table – On each Frequent Data Refresh run, includes inserts and updates in source.
  • Delta_Delete Table – On each Frequent Data Refresh run, includes deletes and updates in source.
  • Lambda View = Base – Delta_Delete + Delta

Note:

The data fetched by the Lambda View is filtered by the "Oldest Data to Report" option set in Frequent Data Refresh V2 configuration. Lambda Views are only for viewing purposes; you must not use them for downstream integrations. See About Scheduling Frequent Data Refreshes Using a New Refresh Mechanism (Preview).
Frequent Data Refresh Module and Table Statuses (Preview)

You can view the status of individual modules or tables on the Frequent Data Refresh page.

These statuses offer insight into whether related data has been recently published, changed, or is available for update. The following table explains the different possible Status messages and their meanings.

Status Definition
Eligible You can select the module for inclusion in Frequent Data Refresh. Once you publish the Frequent Data Refresh definition, this status changes to Published.
Not Published If you've selected any modules for Frequent Data Refresh and then saved your progress, this status indicates that it's been selected since the last publish.
Publish in progress Frequent Data Refresh is running, and the user interface isn't editable.
Published The module is up-to-date as of the last Frequent Data Refresh run, and a Frequent Data Refresh publish isn't running.
Modified If you've changed a table since the last save or publish. You should double check your in-progress changes. For example, you select TABLE_A, and click Save. You later return to work on your Frequent Data Refresh configuration and unselect TABLE_A. TABLE_A’s Status is Modified until you Save or Publish again, at which point it becomes Eligible.
Invalid If a functional area that used to be was part of your Frequent Data Refresh configuration is disabled, the related module is Invalid until the next Publish runs. The data related to the recently disabled functional area is deleted during the next publish.
Frequent Data Refresh Upgrade Considerations (Preview)

Existing users of the Frequent Data Refresh functionality can upgrade to the latest Frequent Data Refresh version, but should be aware of changes to the configuration and behavior after upgrade.

If you have been using Frequent Data Refresh before Platform 25.R3, you can continue using the original version or upgrade to the latest Frequent Data Refresh version. Before upgrading you should consider the following:
The following existing configurations for Frequent Data Refresh are migrated when upgrading to the latest Frequent Data Refresh version:
  • Activated Functional Areas
  • Tables from Activated Data Augmentations
  • Recommended and Certified Tables
  • Not-certified Tables (migrated as non-certified)
The following aren't migrated when upgrading to the latest Frequent Data Refresh version:
  • Non-Activated Functional Areas
  • Tables from not-Activated Data Augmentations
  • Tables from Data Augmentations not supported in Frequent Data Refresh V2 - Dimension Alias and Data Transformation, Data Augmentations with Versioned Data Sets, Data Augmentations without Incremental Key column, Data Augmentations without Last Update Data (LUD) column

Upgrade to Frequent Data Refresh V2 (Preview)

As a service administrator, you can upgrade to use the latest version of frequent data refresh.

  1. In Oracle Fusion Data Intelligence Console, click Enable Features under Application Administration.
  2. On the Enable Features page, under the Preview Features tab, expand the Pipeline Features section, and enable Frequent Data Refresh V2.
  3. Return to the Oracle Fusion Data Intelligence Console and click Data Configuration, then Pipeline Settings.
  4. Note the banner message about a new Frequent Data Refresh. Click Upgrade.
  5. Fill out the Warning dialog with the name of the person performing the upgrade, and the name of the person authorizing the upgrade. Note that these details can be reviewed in the Audit Log.
  6. Click Upgrade. You can't edit the Pipeline Settings page while the upgrade is in progress.
  7. When the upgrade completes, you see the latest user interface and can configure Frequent Data Refresh. See Configure Frequent Data Refresh V2 (Preview).
  8. After upgrade, view the audit log for “Frequent Refresh V2 Upgrade” to understand what configurations got migrated to FDR V2. Ensure that you inspect and validate the modules included in the FDR V2 configuration.
    You must also update the refresh schedule and publish.

Configure Frequent Data Refresh V2 (Preview)

Configure the frequency of data refresh by selecting the refresh schedule, modules, and data tables.

You can specify data to be refreshed at specific times in a day; up to a maximum of three time per day. For example, you could set refreshes at 10AM, 2PM, and 5PM because that's when you want the intra-day data to be refreshed.

Ensure that Frequent Refresh is enabled in Pipeline Features under the Preview Features tab on the Enable Features page prior to scheduling frequent data refreshes. See Make Preview Features Available.

  1. On the Oracle Fusion Data Intelligence Console click Data Configuration, then Pipeline Settings.
  2. Click the Frequent Data Refresh tab.
  3. Under Schedule Information, click Update to change the schedule.
  4. In Update Schedule, specify these:
    1. Timezone: Select your time zone.
    2. In Active Schedules, click the Add Schedule icon, enter a time, and click Update.

    Update Schedule dialog

  5. To refresh specific modules, view the Modules tab, and select the desired function areas and custom data configurations. If you don’t see an expected functional area, first ensure that it is activated. You must also include any functional areas on which the selected functional areas depend on. For Fusion Accounting Hub, additional steps are required to enable it for Frequent Data Refresh. See Enable Frequent Data Refresh Modules for Fusion Accounting Hub (Preview)
    1. Enter the name of a Functional Area, Application, or Offering in the Search field to find a desired entry to select for Frequent Data Refresh. You can select three functional areas for Frequent Data Refresh.
  6. To refresh specific tables, select the Data Tables tab and click to select the desired tables. If you don't see an expected table, make sure you've activated the related subject area..
    1. Filter by warehouse table name and use the Select Entity dropdown to find specific tables.
    2. Use the filters to only show Module tables, Custom Data Configuration tables, Data Augmentation tables, or Not Certified tables, as well as Selected Tables/All Tables, as needed, to find specific items to be included for Frequent Data Refresh.
    3. Select all applicable tables you want to be updated during each Frequent Data Refresh run.
    4. Click Add Table to add tables of type Recommended, Certified or non-Certified. See Table Certification Values for Frequent Data Refresh (Preview) for more details. If you add a Non-certified table, it'll be indicated in the Table Certification column. Certain tables, such as tables that contain aggregate and snapshots, aren't allowed for Frequent Data Refresh. If you attempt to add such tables, you'll get an error.
    5. If you need to share the details of the Frequent Data Refresh configuration, click the Download icon to download a list of selected Data Modules and Tables as CSV.
  7. Optionally edit the Oldest Data to Report field. This option affects the data fetched by the Lambda View. The default value is 4 months. Increasing the value may impact performance on reports and any custom processing you may have based on Lambda views. See Frequent Data Refresh Delta Tables and Lambda View Details (Preview).
  8. To save your progress at any point during configuration, click the Save button.
  9. When you're finished making changes, click Publish. Each change to Frequent Data Refresh settings includes a Warning dialog that requires you to enter the name of the person making changes as well as the name of the person authorizing the change.
  10. See the Status column for information on specific tables or modules. See Frequent Data Refresh Module and Table Statuses (Preview).
  11. View the Audit log to review Frequent Data Refresh configuration details. See View the Audit Log.

Enable Frequent Data Refresh Modules for Fusion Accounting Hub (Preview)

You can control the specific subledgers eligible for Frequent Data Refresh in custom data configurations, such as Oracle Fusion Accounting Hub. If you know that certain subledgers only update weekly, for example, you can exclude them from Frequent Data Refresh to minimize using unnecessary processing resources.

  1. In Oracle Fusion Data Intelligence Console, click Data Configuration under Application Administration.
  2. On the Data Configuration page, under Configurations, click Custom Data Configurations. Click the Actions menu (3 dots) and select Edit.
  3. Click Eligible for frequent data refresh next to specific subledgers you want included in Frequent Data Refresh.

Schedule Frequent Refreshes of Data with Original Refresh Mechanism

You can schedule frequent data refreshes to ensure that incremental data in functional areas, prioritized datasets, warehouse tables, and custom data configurations is the latest.

The original frequent data refresh capability, named Frequent Data Refresh V1, is designed for refresh of base tables that capture the transactional data; it isn't meant for derived datasets that require aggregations, snapshots, or complex transformation logic.

If you want to select the warehouse tables created by the custom data configurations that were generated prior to release Platform 25.R3, then you must regenerate the applicable custom data configurations for these tables to be visible for selection. From release Platform 25.R3, the warehouse tables created by the custom data configurations are available for a frequent data refresh under the Frequent Refresh tab.

This frequent data refresh capability is designed for refresh of base tables that capture the transactional data; it isn't meant for derived datasets that require aggregations, snapshots, or complex transformation logic. Such processing creates data latency that doesn't support high volume of frequent data refresh. For Oracle NetSuite Analytics Warehouse, you can schedule frequent data refreshes for functional areas that are visible in the Frequent Data Refresh Schedule tab on the Pipeline Settings page.

For frequent data refreshes, the semantic model isn't updated. Jobs that update the semantic model don't run as part of data augmentations; they run only during incremental loads for data augmentations.

Topics:

Configure Frequent Data Refresh with Original Refresh Mechanism

You can schedule frequent data refreshes of the applicable functional areas, prioritized datasets, and warehouse tables to ensure that the incremental data in these is the latest.

Ensure that Frequent Refresh is enabled in Pipeline Features under the Generally Available Features tab on the Enable Features page prior to scheduling frequent data refreshes. See Enable Generally Available Features.

  1. Sign in to your service.
  2. In Oracle NetSuite Analytics Warehouse Console, click Data Configuration under Application Administration.
  3. On the Data Configuration page, under Configurations, click Pipeline Settings.
  4. On the Pipeline Settings page, click Frequent Data Refresh Schedule.

    Frequent Refresh tab

  5. In the Frequent Data Refresh Schedule tab, in Refresh Frequency (in hours), select the number of hours after which you want the data refresh to happen.
  6. Enter a start time in Start time.
  7. In Last refresh date, view the date when the data was last refreshed.
  8. If you've enabled the "Frequent Refresh Tables" preview feature and saved your selection of the warehouse tables, then select the Perform frequent refresh for selected warehouse tables toggle. Click the "warehouse tables" hyperlink to view or update your selection of the warehouse tables in the Frequent Refresh Tables tab.
  9. Select the check boxes for the functional areas with the data you want refreshed at the specified interval.
  10. Click Save.