13 Common Configuration and Management Tasks for an Enterprise Deployment

The following topics describe configuration and management tasks you will likely need to perform on the enterprise deployment environment.

13.1 Verifying Manual Failover of the Administration Server

In case a host computer fails, you can fail over the Administration Server to another host. The following sections provide the steps to verify the failover and failback of the Administration Server from BIHOST1 and BIHOST2.

Assumptions:

  • The Administration Server is configured to listen on ADMINVHN, and not on localhost or ANY address.

    For more information about the ADMINVHN virtual IP address, see Reserving the Required IP Addresses for an Enterprise Deployment.

  • These procedures assume that the Administration Server domain home (ASERVER_HOME) has been mounted on both host computers. This ensures that the Administration Server domain configuration files and the persistent stores are saved on the shared storage device.

  • The Administration Server is failed over from BIHOST1 to BIHOST2, and the two nodes have these IPs:

    • BIHOST1: 100.200.140.165

    • BIHOST2: 100.200.140.205

    • ADMINVHN : 100.200.140.206. This is the Virtual IP where the Administration Server is running, assigned to ethX:Y, available in BIHOST1 and BIHOST2.

  • Oracle WebLogic Server and Oracle Fusion Middleware components have been installed in BIHOST2 as described in the specific configuration chapters in this guide.

    Specifically, both host computers use the exact same path to reference the binary files in the Oracle home.

The following topics provide details on how to perform a test of the Administration failover procedure.

13.1.1 Failing Over the Administration Server to a Different Host

The following procedure shows how to fail over the Administration Server to a different node (BIHOST2). Note that even after failover, the Administration Server will still use the same Oracle WebLogic Server machine (which is a logical machine, not a physical machine).

This procedure assumes you’ve configured a per domain Node Manager for the enterprise topology. For more information, see About the Node Manager Configuration in a Typical Enterprise Deployment

To fail over the Administration Server to a different host:

  1. Stop the Administration Server.

  2. Stop the Node Manager in the Administration Server domain directory (ASERVER_HOME).

  3. Migrate the ADMINVHN virtual IP address to the second host:

    1. Run the following command as root on BIHOST1(where X:Y is the current interface used by ADMINVHN):

      /sbin/ifconfig ethX:Y down
      
    2. Run the following command as root on BIHOST2:

      /sbin/ifconfig <interface:index> ADMINVHN netmask <netmask>
      

      For example:

      /sbin/ifconfig eth0:1 100.200.140.206 netmask 255.255.255.0
      

      Note:

      Ensure that the netmask and interface to be used to match the available network configuration in BIHOST2.

  4. Update the routing tables using arping, for example:

    /sbin/arping -q -U -c 3 -I eth0 100.200.140.206
    
  5. Start the Node Manager in the Administration Server domain home on BIHOST2.

  6. Start the Administration Server on BIHOST2.

  7. Test that you can access the Administration Server on BIHOST2 as follows:

    1. Ensure that you can access the Oracle WebLogic Server Administration Console using the following URL:

      http://ADMINVHN:7001/console
      
    2. Check that you can access and verify the status of components in Fusion Middleware Control using the following URL:

      http://ADMINVHN:7001/em
      

13.1.2 Validating Access to the Administration Server on BIHOST2 Through Oracle HTTP Server

After you perform a manual failover of the Administation Server, it is important to verify that you can access the Administration Server, using the standard administration URLs.

From the load balancer, access the following URLs to ensure that you can access the Administration Server when it is running on BIHOST2:

  • http://admin.example.com/console

    This URL should display the WebLogic Server Administration console.

  • http://admin.example.com/em

    This URL should display Oracle Enterprise Manager Fusion Middleware Control.

13.1.3 Failing the Administration Server Back to BIHOST1

After you have tested a manual Administration Server failover, and after you have validated that you can access the administration URLs after the failover, you can then migrate the Administration Server back to its original host.

  1. Stop the Administration Server.
  2. Stop the Node Manager in the Administration Server domain home onBIHOST2.
  3. Run the following command as root on BIHOST2.
    /sbin/ifconfig ethZ:N down
    
  4. Run the following command as root on BIHOST1:
    /sbin/ifconfig ethX:Y 100.200.140.206 netmask 255.255.255.0
    

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in BIHOST1

  5. Update the routing tables using arping on BIHOST1:
    /sbin/arping -q -U -c 3 -I ethX 100.200.140.206
    
  6. Start the Node Manager in the Administration Server domain home on BIHOST1.
  7. Start the Administration Server on BIHOST1.
  8. Test that you can access the Oracle WebLogic Server Administration Console using the following URL:
    http://ADMINVHN:7001/console
    
  9. Check that you can access and verify the status of components in the Oracle Enterprise Manager using the following URL:
    http://ADMINVHN:7001/em
    

13.2 Enabling SSL Communication Between the Middle Tier and the Hardware Load Balancer

This section describes how to enable SSL communication between the middle tier and the hardware load balancer.

Note:

This steps are applicable if the hardware load balancer is configured with SSL and the front end address of the system has been secured accordingly.

13.2.1 When is SSL Communication Between the Middle Tier and Load Balancer Necessary?

In an enterprise deployment, there are scenarios where the software running on the middle tier must access the front-end SSL address of the hardware load balancer. In these scenarios, an appropriate SSL handshake must take place between the load balancer and the invoking servers. This handshake is not possible unless the Administration Server and Managed Servers on the middle tier are started using the appropriate SSL configuration.

13.2.2 Generating Self-Signed Certificates Using the utils.CertGen Utility

This section describes the procedure for creating self-signed certificates on BIHOST1. Create these certificates using the network name or alias of the host.

The directory where keystores and trust keystores are maintained must be on shared storage that is accessible from all nodes so that when the servers fail over (manually or with server migration), the appropriate certificates can be accessed from the failover node. Oracle recommends using central or shared stores for the certificates used for different purposes (for example, SSL set up for HTTP invocations). In this case, BIHOST2 uses the cert directory created for BIHOST1certificates.

For information on using trust CA certificates instead, see the information about configuring identity and trust in Oracle Fusion Middleware Securing Oracle WebLogic Server.

About Passwords

The passwords used in this guide are used only as examples. Use secure passwords in a production environment. For example, use passwords that include both uppercase and lowercase characters as well as numbers.

To create self-signed certificates:

  1. Set up your environment by running the WL_HOME/server/bin/setWLSEnv.sh script:
  2. Verify that the CLASSPATH environment variable is set:
    echo $CLASSPATH
    
  3. Verify that the shared configuration directory folder has been created and mounted to shared storage correctly as described in Preparing the File System for an Enterprise Deployment.

    For example, use the following command to verify that the shared configuration directory is available to each host:

    df -h | grep -B1 SHARED_CONFIG_DIR
    

    Replace SHARED_CONFIG_DIR with the actual path to your shared configuration directory.

    You can also do a listing of the directory to ensure it is available to the host:

    ls -al SHARED_CONFIG_DIR
    
  4. Create the keystore home folder structure if does not already exist.

    For example:

    cd SHARED_CONFIG_DIR
    mkdir keystores
    chown oracle:oinstall keystores
    chmod 750 keystores 
    
  5. Change directory to the keystore home:
    cd KEYSTORE_HOME
    
  6. Run the utils.CertGen tool to create the certificates for both the physical host names and the virtual host names used by servers in the node.

    Syntax:

    java utils.CertGen key_passphrase cert_file_name key_file_name [export | domestic] [hostname]
    

    Examples:

    java utils.CertGen password ADMINVHN.example.com_cert \
          ADMINVHN.example.com_key domestic ADMINVHN.example.com
    
    java utils.CertGen password BIHOST1.example.com_cert \
          BIHOST1.example.com_key domestic BIHOST1.example.com
    

13.2.3 Creating an Identity Keystore Using the utils.ImportPrivateKey Utility

This section describes how to create an Identity Keystore on BIHOST1.example.com.

In previous sections you have created certificates and keys that reside in a shared storage. In this section, the certificate and private key for both BIHOST1 and ADMINVHN are imported into a new Identity Store. Make sure that you use a different alias for each of the certificate/key pair imported.

Note:

The Identity Store is created (if none exists) when you import a certificate and the corresponding key into the Identity Store using the utils.ImportPrivateKey utility.

  1. Import the certificate and private key for ADMINVHN and BIHOST1 into the Identity Store. Make sure that you use a different alias for each of the certificate/key pair imported.

    Syntax:

    java utils.ImportPrivateKey
          -certfile cert_file
          -keyfile private_key_file
          [-keyfilepass private_key_password]
          -keystore keystore
          -storepass storepass
          [-storetype storetype]
          -alias alias 
          [-keypass keypass]
    

    Note:

    Default keystore_type is jks.

    Examples:

    Java utils.ImportPrivateKey 
         -certfile KEYSTORE_HOME/ADMINVHN.example.com_cert.pem
         -keyfile KEYSTORE_HOME/ADMINVHN.example.com_key.pem
         -keyfilepass password
         -keystore appIdentityKeyStore.jks 
         -storepass password
         -alias ADMINVHN
         -keypass password
    
    java utils.ImportPrivateKey 
         -certfile KEYSTORE_HOME/BIHOST1.example.com_cert.pem
         -keyfile KEYSTORE_HOME/BIHOST1.example.com_key.pem
         -keyfilepass password
         -keystore appIdentityKeyStore.jks
         -storepass password 
         -alias BIHOST1
         -keypass password
    
  2. Repeat the above step for all the remaining hosts used in the system (for example, BIHOST2).

13.2.4 Creating a Trust Keystore Using the Keytool Utility

To create the Trust Keystore on BIHOST1.example.com.

  1. Copy the standard java keystore to create the new trust keystore since it already contains most of the root CA certificates needed.

    Oracle does not recommend modifying the standard Java trust key store directly. Copy the standard Java keystore CA certificates located under the WL_HOME/server/lib directory to the same directory as the certificates. For example:

    cp WL_HOME/server/lib/cacerts KEYSTORE_HOME/appTrustKeyStore.jks
    
  2. Use the keytool utility to change the default password.

    The default password for the standard Java keystore is changeit. Oracle recommends always changing the default password, as follows:

    keytool -storepasswd -new NewPassword -keystore TrustKeyStore -storepass Original_Password
    

    For example:

    keytool -storepasswd -new password -keystore appTrustKeyStore.jks -storepass changeit
    
  3. Import the CA certificate into the appTrustKeyStore using the keytool utility.

    The CA certificate CertGenCA.der is used to sign all certificates generated by the utils.CertGen tool and is located at WL_HOME/server/lib directory.

    Use the following syntax to import the certificate:

    keytool -import -v -noprompt -trustcacerts -alias AliasName -file CAFileLocation -keystore KeyStoreLocation -storepass KeyStore_Password
    

    For example:

    keytool -import -v -noprompt -trustcacerts -alias clientCACert -file WL_HOME/server/lib/CertGenCA.der -keystore appTrustKeyStore.jks -storepass password
    

13.2.5 Importing the Load Balancer Certificate into the Trust Store

For the SSL handshake to behave properly, the load balancer's certificate needs to be added to the WLS servers trust store. For adding it, follow these steps:

  1. Access the site on SSL with a browser (this add the server's certificate to the browser's repository).
  2. From the browser's certificate management tool, export the certificate to a file that is on the server's file system (with a file name like bi.example.com).
  3. Use the keytool to import the load balancer's certificate into the truststore:
    keytool -import -file bi.example.com -v -keystore appTrustKeyStore.jks
    

13.2.6 Adding the Updated Trust Store to the Oracle WebLogic Server Start Scripts

To ensure each server can access the updated trust store, edit the setDomainEnv.sh script in each of the domain home directories in the enterprise deployment.
  1. Log in to BIHOST1 and open the following file with a text editor:
    ASERVER_HOME/bin/setDomainEnv.sh
    
  2. Replace reference to the existing DemoTrustStore entry with the location of the appTrustKeyStore.jks file.

    Note that all the values for EXTRA_JAVA_PROPERTIES must be on one line in the file, followed by the export command on a new line. For example:

    EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
             -Djavax.net.ssl.trustStore=KEYSTORE_HOME/appTrustKeyStore.jks..."
    export EXTRA_JAVA_PROPERTIES
    
  3. Make the same change to the setDomainEnv.sh file in the MSERVER_HOME/bin directory on BIHOST1 and BIHOST2.

Alternatively, you can use the setUserOverrides.sh file to place these options, this way they will not get overwritten when the domain's scripts are extended with the configuration wizard or updated with unpack operations. For more information, see "Customizing Domain Wide Server Parameters" in Administering Server Startup and Shutdown for Oracle WebLogic Server.

13.2.7 Configuring Node Manager to Use the Custom Keystores

To configure the Node Manager to use the custom keystores, add the following lines to the end of the nodemanager.properties files located both in ASERVER_HOME/nodemanager and MSERVER_HOME/nodemanager directories in all nodes:

KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=Identity KeyStore
CustomIdentityKeyStorePassPhrase=Identity KeyStore Passwd
CustomIdentityAlias=Identity Key Store Alias
CustomIdentityPrivateKeyPassPhrase=Private Key used when creating Certificate

Make sure to use the correct value for CustomIdentityAlias for Node Manager's listen address. For example on BIHOST1, use appIdentity2 according to the steps in Creating a Trust Keystore Using the Keytool Utility

(appIdentity2 mapped to the BIHOST1 listen address).
Example for Node 1:
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=KEYSTORE_HOME/appIdentityKeyStore.jks
CustomIdentityKeyStorePassPhrase=password
CustomIdentityAlias=appIdentity1
CustomIdentityPrivateKeyPassPhrase=password

The passphrase entries in the nodemanager.properties file are encrypted when you start Node Manager as described in "Starting the Node Manager on BIHOST1." For security reasons, minimize the time the entries in the nodemanager.properties file are left unencrypted. After you edit the file, start Node Manager as soon as possible so that the entries are encrypted.

13.2.8 Configuring WebLogic Servers to Use the Custom Keystores

Configure the WebLogic Servers to use the custom keystores using the Oracle WebLogic Server Administration Console. Complete this procedure for the Administration Server and the Managed Servers that require access to the front end LBR on SSL.

To configure the identity and trust keystores:

  1. Log in to the Administration Console, and click Lock & Edit.
  2. In the left pane, expand Environment, and select Servers.
  3. Click the name of the server for which you want to configure the identity and trust keystores.
  4. Select Configuration, and then Keystores.
  5. In the Keystores field, click Change, and select Custom Identity and Custom Trust method for storing and managing private keys/digital certificate pairs and trusted CA certificates, and click Save.
  6. In the Identity section, define attributes for the identity keystore.
    • Custom Identity Keystore: Enter the fully qualified path to the identity keystore:

      KEYSTORE_HOME/appIdentityKeyStore.jks 
      
    • Custom Identity Keystore Type: Leave this field blank, it defaults to JKS.

    • Custom Identity Keystore Passphrase: Enter the password Keystore_Password you provided in Creating an Identity Keystore Using the utils.ImportPrivateKey Utility

      This attribute may be optional or required depending on the type of keystore. All keystores require the passphrase in order to write to the keystore. However, some keystores do not require the passphrase to read from the keystore. WebLogic Server reads only from the keystore, so whether or not you define this property depends on the requirements of the keystore.

  7. In the Trust section, define properties for the trust keystore:
    • Custom Trust Keystore: Enter the fully qualified path to the trust keystore:

      KEYSTORE_HOME/appTrustKeyStore.jks 
      
    • Custom Trust Keystore Type: Leave this field blank, it defaults to JKS.

    • Custom Trust Keystore Passphrase: The password you provided in as New_Password in "Creating a Trust Keystore Using the Keytool Utility."

      As mentioned in the previous step, this attribute may be optional or required depending on the type of keystore.

  8. Click Save.
  9. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
  10. Click Lock & Edit.
  11. Select Configuration, then SSL.
  12. In the Private Key Alias field, enter the alias you used for the host name the managed server listens on.

    In the Private Key Passphrase and the Confirm Private Key Passphrase fields, enter the password for the keystore that you created in Creating an Identity Keystore Using the utils.ImportPrivateKey Utility

  13. Click Save.
  14. Click Activate Changes in the Administration Console's Change Center to make the changes take effect.
  15. Restart the server for which the changes have been applied. The fact that servers can be restarted using the Administration Console/Node Manager is a good verification that the communication between Node Manager, Administration Server and the managed servers is correct.

13.3 Performing Backups and Recoveries for an Enterprise Deployment

This section provides guidelines for making sure you back up the necessary directories and configuration data for an Oracle Business Intelligence enterprise deployment.

Note:

Some of the static and run-time artifacts listed in this section are hosted from Network Attached Storage (NAS). If possible, backup and recover these volumes from the NAS filer directly rather than from the application servers.

For general information about backing up and recovering Oracle Fusion Middleware products, see the following sections in Administering Oracle Fusion Middleware:

Table 13-1 lists the static artifacts to back up in a typical Oracle Business Intelligence enterprise deployment.

Table 13-1 Static Artifacts to Back Up in the Oracle Business Intelligence Enterprise Deployment

Type Host Tier

Database Oracle home

DBHOST1 and DBHOST2

Data Tier

Oracle Fusion Middleware Oracle home

WEBHOST1 and WEBHOST2

Web Tier

Oracle Fusion Middleware Oracle home

BIHOST1 and BIHOST2

Application Tier

Installation-related files

WEBHOST1, WEHOST2, and shared storage

N/A

Table 13-2 lists the runtime artifacts to back up in a typical Oracle Business Intelligence enterprise deployment.

Table 13-2 Run-Time Artifacts to Back Up in the Oracle Business Intelligence Enterprise Deployment

Type Host Tier

Administration Server domain home (ASERVER_HOME)

BIHOST1 and BIHOST2

Application Tier

Application home (APPLICATION_HOME)

BIHOST1 and BIHOST2

Application Tier

Oracle RAC databases

DBHOST1 and DBHOST2

Data Tier

Scripts and Customizations

BIHOST1 and BIHOST2

Application Tier

Deployment Plan home (DEPLOY_PLAN_HOME)

BIHOST1 and BIHOST2

Application Tier

Singleton Data Directory (SDD)

BIHOST1 and BIHOST2

Application Tier