1 About Loading Rated Events into the BRM Database

Learn about loading rated events into the Oracle Communications Billing and Revenue Management (BRM) database from Elastic Charging Engine (ECE) using Rated Event Loader (RE Loader).

Topics in this chapter:

How Rated Events are Loaded Into the BRM Database

The RE Loader can be used by both the Pipeline Manager and ECE to load rated events into the BRM database. There are different methods of loading the records, depending on the event types, whether the records are coming from Pipeline Manager or ECE, and your business needs.

Methods of Transferring Rated Events to BRM

There are several methods of transferring rated events to BRM. The original method is the most flexible, supporting all events from both ECE and Pipeline Manager. There is also the Rated Event Manager (RE Manager), which has higher performance than the original method and includes five different modes of operation, equating to five different methods of transferring rated events to BRM. The RE Manager has a plug-in for the Rated Event Formatter (RE Formatter) in ECE and a daemon in BRM. The RE Manager can be used only for ECE (not for Pipeline Manager), and it can be used for all types of events from ECE except for suspense management events.

The following methods of transferring rated events are available:

Original Method

The original method can be used for all types of rated events, and for files from both ECE and Pipeline Manager. It transfers files to an intermediate file system where they are picked up by BRM.

The process when this method is used with ECE is:

  1. In the ECE system, the Elastic Charging Server (ECS) publishes rated events to an Oracle NoSQL database or Oracle Database.
  2. RE Formatter uses the BRM call detail record (CDR) plug-in to write the rated events to a CDR file and send the file to a specified directory.
  3. Rated Event Loader (RE Loader) retrieves the CDR file from the directory and loads the rated events into the BRM database. RE Formatter and RE Loader can run on the same system or different systems, as long as they can both access the CDR file directory.

The following chapters contain information that is relevant to using this processing method:

About Loading Prerated Events Using Pipeline Manager

Prerated events are events that have been rated by Pipeline Manager prior to being loaded into the BRM database. Basic steps of pipeline rating and event loading include:

  1. Pipeline Manager rates events associated with call detail records (CDRs).

    For information on how Pipeline Manager rates events, see "How Events Are Rated by Using Pipeline Manager" in BRM Setting Up Pipeline Pricing.

  2. Pipeline Manager creates an output file for each service type and places them in one or more output directories.

    You configure the number and location of your pipeline output directories by using the pipeline EXT_OutFileManager module.

  3. RE Loader loads the prerated events.

    For information, see "Pipeline Manager RE Loader Process Overview".

About Loading Rerated Events Using Pipeline Manager

It is possible to discover pricing or rating configuration errors after events have been rated by Pipeline Manager and loaded into the BRM database. When this occurs, you rerate any incorrectly rated events and reload them into the BRM database.

When you need to rerate and reload pipeline-rated events, you must:

  1. Extract events that need rerating from the BRM database by using the Event Extraction Tool.

    Note:

    Event Extraction Manager is included in the RE Loader installation.

  2. Rerate those events by using Pipeline Manager. Pipeline Manager backs out the previous rating changes and then rerates the events.

  3. Reload the rerated events by using RE Loader.

    For information, see "Pipeline Manager RE Loader Process Overview".

DIRECT Mode

This mode loads the data directly from ECE into the regular tables in the BRM database. This is the most direct method, but it uses more bandwidth than the ZIP_DB option. The process when this mode is used is:

  1. In the ECE system, ECS publishes rated events to an Oracle NoSQL database or Oracle Database.
  2. RE Formatter uses the ECE RE Manager Plug-in to write the rated events to temporary CDR files. The RE Manager Loader inserts the CDRs directly into the BRM database.
  3. The RE Manager Loader passes information to the RE Manager Updater, which updates the balances in the BRM database.

The following chapters contain information that is relevant to using this processing method:

ZIP_DB Mode

This mode creates a ZIP file and stores it in the /batch/rel object in the database, where a BRM process loads it into the regular tables. The process when this mode is used is:

  1. In the ECE system, ECS publishes rated events to an Oracle NoSQL database or Oracle Database.
  2. RE Formatter uses the ECE RE Manager Plug-in to write the rated events to temporary CDR files. The RE Manager compresses the CDRs and inserts the ZIP file directly into a new table in the BRM database.
  3. The RE Manager Zip Transfer Processor in BRM reads the ZIP file and updates the RE Manager Loader and the RE Manager Updater.
  4. The RE Manager Loader inserts the records into the BRM Database and the RE Manager Updater updates the balances.

The following chapters contain information that is relevant to using this processing method:

ZIP_FILE Mode

This mode compresses the CDRs in a transaction into a single file before transferring it to an intermediate file system where it is picked up by BRM. The process when this mode is used is:

  1. In the ECE system, ECS publishes rated events to an Oracle NoSQL database or Oracle Database.
  2. RE Formatter uses the ECE RE Manager Plug-in to write the rated events to temporary CDR files. The RE Manager compresses the CDRs and creates the ZIP file on the specified file system.
  3. The RE Manager Directory Processor in BRM reads the ZIP file and updates the RE Manager Loader and the RE Manager Updater.
  4. The RE Manager Loader inserts the records into the BRM Database and the RE Manager Updater updates the balances.

The following chapters contain information that is relevant to using this processing method:

CDR Mode

This mode is similar to the original method. It transfers individual files to an intermediate file system, where they are picked up by BRM. You might choose this method if you wanted to read or perform some action on the files before they were picked up by BRM. The process when this mode is used is:

  1. In the ECE system, ECS publishes rated events to an Oracle NoSQL database or Oracle Database.
  2. RE Formatter uses the ECE RE Manager Plug-in to write the rated events to temporary CDR files. RE Manager in CDR Mode writes the rated events to CDR files and creates the files in a specified directory.
  3. The RE Manager Directory Processor in BRM reads the CDR files and updates the RE Manager Loader and the RE Manager Updater.
  4. The RE Manager Loader inserts the records into the BRM Database and the RE Manager Updater updates the balances.

The following chapters contain information that is relevant to using this processing method:

Event Streaming Mode

This mode streams events to and from a Kafka topic. The process when this mode is used is:

  1. In the ECE system, RE Formatter publishes rated events to one or more Kafka topics. These topics can be subscribed to from external systems, such as analytics and reporting systems.
  2. The RE Manager Stream Processor in BRM reads the messages from the Kafka stream and updates the RE Manager Loader and the RE Manager Updater.
  3. The RE Manager Loader inserts the records into the BRM Database and the RE Manager Updater updates the balances.

The following chapters contain information that is relevant to using this processing method:

Rated Event Loading

RE Manager Loader and RE Loader load rated events directly into the BRM database, bypassing the Connection Manager (CM) and Data Manager (DM). RE Loader then updates account balances, billing items, and journals in the BRM database.

RE Loader uses a partitioned database and inserts pre-rated events into separate partitions allocated for delayed events. The events are called "delayed" because they are rated before they are loaded, and there is a delay between the two actions.

Note:

You must partition your database when loading pre-rated events. See "Partitioning Database Tables" in BRM System Administrator's Guide.

By default, RE Loader loads the events shown in Table 1-1. However, you can configure RE Loader to load custom events. See "Adding New Types of Events for RE Loader to Load".

Table 1-1 Services and Events Loaded by Default

Service Event

GPRS (General Packet Radio Service)

/event/delayed/session/gprs

GSM (Global System for Mobile Communication)

/event/delayed/session/telco/gsm

Pipeline Manager RE Loader Process Overview

RE Loader processes output files generated from Pipeline Manager. You send these files to RE Loader manually through a command-line utility or automatically by the Batch Controller.

After RE Loader receives a pipeline output file, it:

  1. Checks the event header to determine the storable class type and whether the file contains prerated, rerated, or discount events.

  2. Parses the event data record (EDR) data into multiple temporary files, one for each BRM database table to be loaded, and places the files in a temporary directory.

  3. Loads events from each temporary file into the BRM database by using multiple Oracle SQL Loader utility sqlldr processes.

  4. Calls stored procedures to update the account balances, bill items, and journals.

  5. Logs the session information in its log file (rel.pinlog).

About Running Multiple RE Loader Processes

To achieve better loading performance, the sample Batch Controller configuration file (SampleBatchControllerInfranet.properties) is set to run three RE Loader processes in parallel. This means that when you schedule RE Loader to run automatically, Batch Controller starts up to three instances of the RE Loader batch handler. Each instance of the RE Loader batch handler starts an instance of the RE Loader utility.

If you configure RE Loader to run manually, you can start multiple processes from the command line.

Note:

If you run multiple RE Loader processes in parallel with Pipeline Manager, configure Pipeline Manager to delete empty output streams. For more information, see "Configuring Pipeline Manager to Delete Empty Output Streams".

Figure 1-1 shows an example of three RE Loader batch handler and pin_rel utility processes running to load EDR files. Each utility loads one file into the BRM database:

Figure 1-1 Example of Three Batch Handlers and pin_rel Utilities


Description of Figure 1-1 follows
Description of "Figure 1-1 Example of Three Batch Handlers and pin_rel Utilities"

The following actions are performed when multiple RE Loader processes are configured:

  1. Batch Controller starts an RE Loader batch handler for each new file it detects in the output directory, up to the maximum number of RE Loader batch handler processes configured.

  2. Each RE Loader batch handler process starts an RE Loader utility process.

  3. The RE Loader utility processes the events in the file in three phases:

    • Preprocessing

    • Loading

    • Account balance, bill item, and journal updates

    During the loading phase, the database tables are locked so that only one RE Loader process can load events at one time.

  4. Each RE Loader process polls the other processes to see whether they are currently in the loading phase and waits its turn to load events. When the first process is loading, the second process performs prepreprocessing tasks while waiting its turn. When the first process finishes loading, the second process loads while the first process performs account balance, bill item, and journal updates for the events it loaded:


    Description of rlo_multi_processes.gif follows
    Description of the illustration rlo_multi_processes.gif

  5. When an RE Loader process has completed all three phases, it is free to process another file.

The files are loaded in sequence, one directory at a time. If the number of RE Loader processes is set to 3, only three handler processes can run at one time for all directories.

Setting the Optimal Number of RE Loader Processes

Because there are three phases to processing EDR files, the optimal number of RE Loader processes to configure is at least three. If you want to configure more than three processes, you should test your RE Loader performance. Because only one RE Loader process can load events at a time, having more than three means those that have finished the preprocessing phase wait their turn to load events. Depending on the size of your EDR files and the time it takes to load events, configuring four or five processes might save time.

You configure the number of RE Loader processes to run in parallel by setting the number of RE Loader batch handlers to start in the Batch Controller configuration file. For more information, see "Configuring RE Loader to Run Automatically by Using Batch Controller".

Configuring Pipeline Manager to Delete Empty Output Streams

If you run multiple RE Loader processes, you should configure Pipeline Manager to delete empty output streams.

Pipeline Manager generates files based on the number of output streams that are running. If some of the output streams are empty, the pipeline can produce empty files, which causes an error when RE Loader attempts to load the empty files.

To produce only one output stream, set the DeleteEmptyStream pipeline registry entry to True. This is the default. For more information, see "Configuring Output for Rated Events in BRM Setting Up Pipeline Rating and Discounting and OUT_GenericStream" in BRM Pipeline Manager Reference.

About Backing Up RE Loader Files

By default, RE Loader skips redo generation when loading files into the BRM database. This optimizes loading performance, but it can cause you to lose data if your system shuts down ungracefully.

To prevent data loss when your system shuts down:

  • Make full backups of the BRM database on a regular basis.

  • Archive all successfully loaded files until you make a full database backup.

You can re-enable redo generation, at the cost of loading performance, by modifying the RE Loader control files. See "Configuring Whether to Perform Redo Generation".

About Using RE Loader in a Multischema System

If you use a multischema system, you must set up the following for each BRM database schema in your system:

  • RE Loader instance. Each instance of RE Loader must also have its own set of RE Loader processing directories.

  • (Running RE Loader automatically only) An instance of Batch Controller and the RE Loader batch handler.

About Configuring Rated Event Loading

To retrieve and load rated events: