During the first phase of the upgrade process, Site 1 processes usage requests while you upgrade the software in Site 2.

To switch off Site 2:

1. Stop all usage requests to Site 2 and direct all usage requests to Site 1.

2. Stop the connection from BRM on Site 1 to ECE on Site 2. To do so:

   • On ECE Site 2, stop the EM Gateway.

   • On BRM Site 1, remove any connection to the EM Gateway on Site 2.

3. On ECE Site 1, mark Site 2 as failed.

   This stops ECE from forwarding rating requests to Site 2 for subscribers with Site 2 as their preferred site. Usage requests are now rated on Site 1.

4. Disable the federation from ECE Site 1 to Site 2.

   Note:

   Keep the Site 2 to Site 1 federation active to drain any remaining data from Site 2.

5. Check that the federation backlog from Site 2 to Site 1 has been cleared.

   All traffic to Site 2 is stopped. Ensure all data from Site 2 has been synchronized with Site 1. The Coherence Federation Metrics should show as IDLE instead of YIELDING.

6. On Site 2, check that all rated events in the ECE cache have been extracted by running the following with the query.sh utility:

   ./query.sh
   Coherence Command Line Tool
   CohQl> select value() from AggregateObjectUsage

   If successful, the query displays zero entries. See "query" in ECE Implementing Charging for information about the utility's syntax and parameters.

7. On Site 2, use SQL*Plus to check that all Site 2 rated events have been extracted from the persistence database and are present in BRM:

   sqlplus pin@databaseName
   Enter password: password
   
   SQL> select count(*) from ratedevent_site2Name

   where databaseName is the service name or database alias of the BRM database, and password is the password for the pin user.

   Note:

   The Site 2 persistence database might contain some Site 1 events. After the Site 2 to Site 1 federation process is stopped, these events are not extracted or purged. However, they will be processed on Site 1. You can ignore these events because they get purged when you re-create the Site 2 persistence database later.

8. Stop the federation process from Site 2 to Site 1.

   All data is now synchronized with Site 1. Site 2 is isolated and ready for the upgrade.

To uninstall BRM and ECE from Site 2:

1. Uninstall BRM 12.0 Patch Set 3 cloud native from Site 2:

   helm uninstall BrmReleaseName --namespace BrmNameSpace

2. Uninstall ECE 12.0 Patch Set 3 cloud native from Site 2:

   helm uninstall EceReleaseName --namespace BrmNameSpace

where:

• BrmReleaseName is the release name for oc-cn-helm-chart and is used to track this installation instance.

• EceReleaseName is the release name for oc-cn-ece-helm-chart and is used to track this installation instance.

• BrmNameSpace is the namespace in which to create BRM Kubernetes objects for the BRM Helm chart.

These commands delete all the resources associated with the chart's last release and the release history.

To upgrade your BRM database schema in Site 2 to release BRM 15.0.x:

1. Download and extract the BRM 15.0.x cloud native database initializer Helm chart (oc-cn-op-job-helm-chart) from Oracle Software Delivery Cloud (https://edelivery.oracle.com).

   See "Downloading Packages for the BRM Cloud Native Helm Charts and Docker Files" for more information.

2. Copy your BRM 12.0 Patch Set 3 wallet files (ewallet.p12 and cwallet.sso) from the dm-oracle pod's /oms/wallet/client directory to the BRM 15.0.x Helm chart's oc-cn-init-db-helm-chart/existing_wallet/ directory.

3. Create an override-init-db-15.yaml file for the 15.0.x version of oc-cn-init-db-helm-chart.

4. In your override-init-db-15.yaml file, set the following keys:

   • ocbrm.is_upgrade: Set this to true.

   • ocbrm.existing_rootkey_wallet: Set this to true.

   • ocbrm.wallet.client: Set this to the password for the client wallet. This value must match that of your 12.0 Patch Set 3 release.

   • ocbrm.wallet.server: Set this to the password for the server wallet. This value must match that of your 12.0 Patch Set 3 release.

   • ocbrm.wallet.root: Set this to the password for the root wallet. This value must match that of your 12.0 Patch Set 3 release.

5. Enter this command from the helmcharts directory to upgrade the database schema. Ensure you run the 15.0.x oc-cn-init-db-helm-chart Helm chart with a new release name and namespace.

   helm install newInitDbReleaseName oc-cn-init-db-helm-chart --namespace newInitDbNameSpace --values oldOverrideValues --values override-init-db-15.yaml

   where:

   • newInitDbReleaseName is the new release name for the 15.0.x version of oc-cn-init-db-helm-chart.

   • newInitDbNameSpace is the new namespace for the 15.0.x version of oc-cn-init-db-helm-chart.

   • oldOverrideValues is the name and path to your 12.0 Patch Set 3 override-values.yaml file for oc-cn-init-db-helm-cart.

To configure and deploy BRM 15.0.x on your cloud native system on Site 2:

1. Download and extract the BRM 15.0.x cloud native (oc-cn-helm-chart) and BRM 15.0.x Operator Job Helm Chart (oc-cn-op-job-helm-chart) from Oracle Software Delivery Cloud (https://edelivery.oracle.com).

   See "Downloading Packages for the BRM Cloud Native Helm Charts and Docker Files" for more information.

2. To reuse your 12.0 Patch Set 3 SSL KeyStore with the new release, copy the PDC KeyStore files from the 12.0 Patch Set 3 Helm chart to the 15.0.x oc-cn-op-job-helm-chart/pdc/pdc_keystore/ directory.

3. Create an override-values-15.yaml file.

   You will use this file with the 15.0.x version of oc-cn-helm-chart and oc-cn-op-job-helm-chart.

4. In your override-values-15.yaml file, set the following keys:

   • In the BRM section:

     • ocbrm.is_upgrade: Set this to true.

     • ocbrm.existing_rootkey_wallet: Set this to false.

     • ocbrm.db.*: Set the BRM database schema details to the same values as your 12.0 Patch Set 3 release.

   • In the PDC section:

     • ocpdc.configEnv.pdcSchemaUserName: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocpdc.configEnv.crossRefSchemaUserName: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocpdc.configEnv.rcuPrefix: Set this key to a new prefix to create a new RCU schema.

     • ocpdc.configEnv.transformation.upgrade: Set this to true. (For upgrades to 15.0.0 only)

     • ocpdc.secretValue.walletPassword: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocpdc.configEnv.deployAndUpgradeSite2: Set this to true. (For upgrades to 15.0.1 only)

     • ocpdc.configEnv.upgrade: Set this to true. (For upgrades to 15.0.1 only)

   • In the Business Operations Center section:

     • ocboc.boc.configEnv.bocSchemaUserName: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocboc.boc.configEnv.runUpgrade: Set this to true.

     • ocboc.boc.configEnv.rcuPrefix: Set this to a new prefix to create a new RCU schema.

     • ocboc.boc.secretVal.*: Set the Business Operations Center passwords to the same values as your 12.0 Patch Set 3 release.

   • In the Billing Care section:

     • ocbc.bc.configEnv.rcuPrefix: Set this to a new prefix to create a new RCU schema.

     • ocbc.bc.secretVal.*: Set the Billing Care passwords to the same values as your 12.0 Patch Set 3 release.

   • In the Billing Care REST API section:

     • ocbc.bcws.configEnv.rcuPrefix: Set this to a new prefix to create a new RCU schema.

     • ocbc.bcws.secretVal.*: Set the Billing Care REST API passwords to the same values as your 12.0 Patch Set 3 release.

5. Create WebLogic domains by running the 15.0.x version of oc-cn-op-job-helm-chart from the helmcharts directory:

   helm install oldOpJobReleaseName oc-cn-op-job-helm-chart --namespace oldBrmNameSpace --namespace oldOverrideValuesFile --values override-values-15.yaml

6. Install BRM cloud native services by running the 15.0.x version of oc-cn-helm-chart from the helmcharts directory:

   helm install oldBrmReleaseName oc-cn-helm-chart --namespace oldBrmNameSpace --values oldOverrideValuesFile --values override-values-15.yaml

To drop the ECE persistence database schema from Site 2:

1. Uninstall the ECE 12.0 Patch Set 3 Helm chart:

   helm uninstall oldEceReleaseName

   where oldEceReleaseName is the release name for your 12.0 Patch Set 3 version of oc-cn-ece-helm-chart.

2. Delete the 12.0 Patch Set 3 version of ece-persistence-job from your system by running this command:

   kubectl delete job ece-persistence-job --namespace oldBrmNameSpace

   where oldBrmNameSpace is the namespace for your 12.0 Patch Set 3 version of your BRM Helm release.

To install ECE 15.0.x cloud native on Site 2:

1. Download and extract the ECE 15.0.x cloud native (oc-cn-ece-helm-chart) from Oracle Software Delivery Cloud (https://edelivery.oracle.com).

   See "Downloading Packages for the BRM Cloud Native Helm Charts and Docker Files" for more information.

2. Create an override-values-ece-15.yaml file.

3. In the file, configure your ECE 15.0.x cloud native services by following the instructions in "Configuring ECE Services".

4. Deploy the ECE 15.0.x cloud native services by entering this command from the helmcharts directory:

   helm install oldEceReleaseName oc-cn-ece-helm-chart --namespace oldBrmNameSpace --values oldOverrideValuesFile --values override-values-ece-15.yaml

   where:

   • oldEceReleaseName is the release name for your 12.0 Patch Set 3 version of oc-cn-ece-helm-chart.

   • oldBrmNameSpace is the namespace for your 12.0 Patch Set 3 oc-cn-helm-chart deployment.

   • oldOverrideValuesFile is the file name and path to the 12.0 Patch Set 3 version of your override-values.yaml file for oc-cn-ece-helm-chart.

Site 2 has been upgraded at this stage, but it is not handling any usage requests. You now switch usage request processing from Site 1 to Site 2.

To fail over from Site 1 to Site 2:

1. Start the Coherence federation process from Site 1 to Site 2. This provisions the empty ECE Site 2 cache with the latest data from Site 1.

2. On Site 2, mark Site 1 inactive to ensure no preferred site routing occurs from Site 2 to Site 1.

   Note:

   Site 1 also has Site 2 marked as inactive.

3. Check that the federation process is up to date. After the federation process completes successfully:

   • The ECE pods in Site 2 transition to the Running state.

   • Site 2 transitions to the Usage Processing state and spawns Monitoring Agent pods.

4. Stop all usage requests to Site 1 and then redirect them to Site 2.

5. Switch provisioning to BRM on Site 2. To do so:

   1. On BRM Site 2, remove any connections to EM Gateway on Site 1.

   2. On the client side, switch provisioning to BRM on Site 2.

   3. On ECE Site 1, stop the EM Gateway.

   Note:

   If latency between Site 1 and Site 2 is too high to give you acceptable performance for the provisioning flow, reverse the Oracle Data Guard roles and make the BRM database on Site 2 active. Be aware that this can cause a service interruption of a few minutes for provisioning.

6. Check that the federation process from Site 1 to Site 2 has completed.

   Now that all traffic to Site 1 has stopped, ensure all data from Site 1 has been synchronized with Site 2. The Coherence Federation Metrics should display IDLE instead of YIELDING.

7. Check that all rated events in the ECE Site 1 cache have been extracted by running the query utility:

   ./query.sh
   Coherence Command Line Tool
   CohQl> select value() from AggregateObjectUsage

   If successful, this command returns zero entries.

8. On Site 1, check that all Site 1 rated events have been extracted from the persistence database and are present in BRM by using SQL*Plus:

   sqlplus pin@databaseName
   Enter password: password
   
   SQL> select count(*) from ratedevent_site1Name

   where databaseName is the service name or database alias of the BRM database, and password is the password for the pin user.

   Note:

   The Site 1 persistence database might contain some Site 2 events. After the Site 2 to Site 1 federation process is stopped, these events are not extracted or purged. However, they will be processed on Site 2. You can ignore these events because they get purged when you re-create the Site 1 persistence database later.

9. Stop the federation process from Site 1 to Site 2.

   Site 1 is isolated and ready for the upgrade.

Uninstall the 12.0 Patch Set 3 version of BRM and ECE from Site 1.

To do so, perform the following steps on Site 1:

1. Uninstall BRM 12.0 Patch Set 3 cloud native:

   helm uninstall BrmReleaseName --namespace BrmNameSpace

2. Uninstall ECE 12.0 Patch Set 3 cloud native

   helm uninstall EceReleaseName --namespace BrmNameSpace

where:

• BrmReleaseName is the release name for oc-cn-helm-chart and is used to track this installation instance.

• EceReleaseName is the release name for oc-cn-ece-helm-chart and is used to track this installation instance.

• BrmNameSpace is the namespace in which to create BRM Kubernetes objects for the BRM Helm chart.

These commands delete all of the resources associated with the chart's last release and the release history.

To configure and deploy BRM 15.0.x cloud native on Site 1:

1. Download and extract the BRM 15.0.x cloud native Helm chart (oc-cn-helm-chart) and BRM Operator Job Helm Chart (oc-cn-op-job-helm-chart) from Oracle Software Delivery Cloud (https://edelivery.oracle.com).

   See "Downloading Packages for the BRM Cloud Native Helm Charts and Docker Files" for more information.

2. To reuse your 12.0 Patch Set 3 SSL KeyStore, copy the PDC KeyStore files from the 12.0 Patch Set 3 Helm chart to the 15.0.x oc-cn-op-job-helm-chart/pdc/pdc_keystore/ directory.

3. Create an override-values-15.yaml file.

   You will use this file with the 15.0.x version of oc-cn-helm-chart and oc-cn-op-job-helm-chart.

4. In your override-values-15.yaml file, set the following keys:

   • In the BRM section:

     • ocbrm.is_upgrade: Set this to true.

     • ocbrm.existing_rootkey_wallet: Set this to false.

     • ocbrm.db.*: Set the BRM database schema details to the same values as your 12.0 Patch Set 3 release.

   • In the PDC section:

     • ocpdc.configEnv.pdcSchemaUserName: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocpdc.configEnv.crossRefSchemaUserName: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocpdc.configEnv.rcuPrefix: Set this key to a new prefix to create a new RCU schema. The value must be different from the one used in Site 2.

     • ocpdc.configEnv.transformation.upgrade: Set this to true. (For upgrades to 15.0.0 only)

     • ocpdc.secretValue.walletPassword: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocpdc.configEnv.deployAndUpgradeSite2: Set this to false. (For upgrades to 15.0.1 only)

     • ocpdc.configEnv.upgrade: Set this to true. (For upgrades to 15.0.1 only)

   • In the Business Operations Center section:

     • ocboc.boc.configEnv.bocSchemaUserName: Set this to the same value as your 12.0 Patch Set 3 release.

     • ocboc.boc.configEnv.runUpgrade: Set this to true.

     • ocboc.boc.configEnv.rcuPrefix: Set this to a new prefix for the RCU schema. The value must be different from the one used in Site 2.

     • ocboc.boc.secretVal.*: Set the Business Operations Center passwords to the same values as your 12.0 Patch Set 3 release.

   • In the Billing Care section:

     • ocbc.bc.configEnv.rcuPrefix: Set this to a new prefix for the RCU schema. This value must be different from the one used in Site 2.

     • ocbc.bc.secretVal.*: Set the passwords to the same values as your 12.0 Patch Set 3 release.

   • In the Billing Care REST API section:

     • ocbc.bcws.configEnv.rcuPrefix: Set this to a new prefix for the RCU schema. This value must be different from the one used in Site 2.

     • ocbc.bcws.secretVal.*: Set the passwords to the same values as your 12.0 Patch Set 3 release.

5. Save and close the file.

6. Create WebLogic domains by running the 15.0.x version of oc-cn-op-job-helm-chart from the helmcharts directory:

   helm install oldOpJobReleaseName oc-cn-op-job-helm-chart --namespace oldBrmNameSpace --namespace oldOverrideValuesFile --values override-values-15.yaml

   where:

   • oldOpJobReleaseName is the release name assigned to your 12.0 Patch Set 3 release of the oc-cn-op-job-helm-chart installation.

   • oldBrmNameSpace is the namespace for your 12.0 Patch Set 3 version of the BRM deployment.

   • oldOverrideValuesFile is the file name and path of your 12.0 Patch Set 3 version of the override-values.yaml file for oc-cn-op-job-helm-chart.

7. Install BRM cloud native services by running the 15.0.x version of oc-cn-helm-chart from the helmcharts directory:

   helm install oldBrmReleaseName oc-cn-helm-chart --namespace oldBrmNameSpace --values oldOverrideValuesFile --values override-values-15.yaml

   where oldBrmReleaseName is the release name assigned to your 12.0 Patch Set 3 version of the oc-cn-helm-chart installation.

To drop the ECE 12.0 Patch Set 3 persistence database schema from Site 1:

1. Uninstall the ECE 12.0 Patch Set 3 Helm chart:

   helm uninstall oldEceReleaseName

   oldEceReleaseName is the release name for the 12.0 Patch Set 3 version of oc-cn-ece-helm-chart.

2. Uninstall the 12.0 Patch Set 3 version of ece-persistence-job from your system by running this command:

   kubectl delete job ece-persistence-job --namespace oldBrmNameSpace

   where BrmNameSpace is the namespace for the 12.0 Patch Set 3 version of the BRM Helm chart.

To install ECE 15.0.x cloud native on Site 1:

1. Download and extract the ECE 15.0.x cloud native (oc-cn-ece-helm-chart) from Oracle Software Delivery Cloud (https://edelivery.oracle.com).

   See "Downloading Packages for the BRM Cloud Native Helm Charts and Docker Files" for more information.

2. Create an override-values-ece-15.yaml file.

3. In the file, configure your ECE 15.0.x cloud native services by following the instructions in "Configuring ECE Services".

4. Deploy the ECE 15.0.x cloud native services by entering this command from the helmcharts directory:

   helm install oldEceReleaseName oc-cn-ece-helm-chart --namespace oldBrmNameSpace --values oldOverrideValuesFile --values override-values-ece-15.yaml

   where:

   • oldEceReleaseName is the release name for your 12.0 Patch Set 3 version of oc-cn-ece-helm-chart.

   • oldBrmNameSpace is the release name for your 12.0 Patch Set 3 oc-cn-helm-chart deployment.

   • oldOverrideValuesFile is the file name and path to the 12.0 Patch Set 3 version of your override-values.yaml file for oc-cn-ece-helm-chart.

Site 1 is now upgraded, but it is idle.

Now that Site 1 and Site 2 have been upgraded to release 15.0.x, you can restart the federation process between them.

To start the federation process of ECE cache data between Site 1 and Site 2:

1. Enable the two-way federation process between Site 1 and Site 2.

2. Check that the federation backlog is processed successfully.

   After the federation process completes, ECE in Site 1 transitions to the Usage Processing state and spawns the Monitoring Agent pods.

3. In ECE Site 1, mark ECE in Site 2 as active. Likewise, in ECE Site 2, mark ECE in Site 1 as active.

   This enables usage rating requests to be forwarded to the subscriber's preferred site.

4. Enable usage requests to Site 2.

5. Enable failover connection to the EM Gateway. On both sides, ensure the CM has a failover EM connection to the EM Gateway on the other site.