The following procedure shows how to fail over the Administration Server to a different node (BIHOST2). Note that even after failover, the Administration Server will still use the same Oracle WebLogic Server machine (which is a logical machine, not a physical machine).

If you have configured the web tier to access AdminServer, it is important to verify that you can access the Administration Server after you perform a manual failover of the Administration Server, by using the standard administration URLs.

In order to manage each product effectively within a single enterprise deployment domain, you must understand which products require specific administration roles or groups, and how to add a product-specific administration role to the Enterprise Deployment Administration group.

• Adding the Enterprise Deployment Administration User to a Product-Specific Administration Group
  

After you have tested a manual Administration Server failover, and after you have validated that you can access the administration URLs after the failover, you can then migrate the Administration Server back to its original host.

1. Stop the Administration Server.
2. Stop the Node Manager in the Administration Server domain directory (ASERVER_HOME).
3. Migrate the ADMINVHN virtual IP address to the second host:

   1. Run the following command as root on BIHOST2 to check the virtual IP address at its CIDR:

      ip addr show dev ethX

      Where, X is the current interface used by ADMINVHN.

      For example:

      ip addr show dev eth0

   2. Run the following command as root on BIHOST2 (where X:Y is the current interface used by ADMINVHN):

      ip addr del ADMINVHN/CIDR dev ethX:Y

      Where, X:Y is the current interface used by ADMINVHN.

      For example:

      ip addr del 100.200.140.206/24 dev eth0:1

   3. Run the following command as root on BIHOST1:

      ip addr add ADMINVHN/CIDR dev ethX label ethX:Y

      Where, X:Y is the current interface used by ADMINVHN.

      For example:

      ip addr add 100.200.140.206/24 dev eth0 label eth0:1

      Note:

      Ensure that the CIDR representing the netmask and interface to be used to match the available network configuration in BIHOST1.

4. Update the routing tables using arping, for example:

   arping -b -A -c 3 -I eht0 100.200.140.206

5. Start the Node Manager in the Administration Server domain home on BIHOST1.
6. Start the Administration Server on BIHOST1.
7. Test that you can access the Administration Server on BIHOST1 as follows:

   1. Ensure that you can access the Oracle WebLogic Server Administration Console using the following URL:

      http://ADMINVHN:7001/console
      

   2. Check that you can access and verify the status of components in Fusion Middleware Control using the following URL:

      http://ADMINVHN:7001/em

In an enterprise deployment, there are scenarios where the software running on the middle tier must access the front-end SSL address of the hardware load balancer. In these scenarios, an appropriate SSL handshake must take place between the load balancer and the invoking servers. This handshake is not possible unless the Administration Server and Managed Servers on the middle tier are started by using the appropriate SSL configuration.

1. Temporarily, set up your environment by running the following script:

   . WL_HOME/server/bin/setWLSEnv.sh

   Note that there is a dot(.) and space( ) preceding the script name in order to source the shell script in the current shell.

2. Verify that the CLASSPATH environment variable is set:

   echo $CLASSPATH
   

3. Verify that the shared configuration directory folder has been created and mounted to shared storage correctly, as described in Preparing the File System for an Enterprise Deployment.

   For example, use the following command to verify that the shared configuration directory is available to each host:

   df -h | grep -B1 SHARED_CONFIG_DIR

   Replace SHARED_CONFIG_DIR with the actual path to your shared configuration directory.

   You can also do a listing of the directory to ensure that it is available to the host:

   ls -al SHARED_CONFIG_DIR

4. Create the keystore home folder structure if does not already exist.

   For example:

   cd SHARED_CONFIG_DIR
   mkdir keystores
   chown oracle:oinstall keystores
   chmod 750 keystores
   export KEYSTORE_HOME=SHARED_CONFIG_DIR/keystores
5. Change directory to the keystore home:

   cd KEYSTORE_HOME

6. Run the utils.CertGen tool to create the certificates for hostnames or aliases used by the managed servers and node managers, one per host.

   Note:

   You must run the utils.CertGen tool to create certificates for all the other hosts that run the Manager Servers.

   Syntax:

   java utils.CertGen key_passphrase cert_file_name key_file_name [export | domestic] [hostname]

   Examples:

   java utils.CertGen password ADMINVHN.example.com_cert \
         ADMINVHN.example.com_key domestic ADMINVHN.example.com
   
   java utils.CertGen password BIHOST1.example.com_cert \
         BIHOST1.example.com_key domestic BIHOST1.example.com
   

   java utils.CertGen password BIHOST1VHN.example.com_cert \
          BIHOST1VHN.example.com_key domestic BIHOST1VHN.example.com

7. Repeat the above step for all the remaining hosts used in the system (for example, BIHOST2 and BIHOST2VHN).
8. For Dynamic clusters, in addition to ADMINVHN and one certificate for each host, a certificate matching a wildcard URL should also be generated.

   For example:

   java utils.CertGen password WILDCARD.example.com_cert \ 
   WILDCARD.example.com_key domestic \*.example.com 
   

1. Import the certificate and private key for ADMINVHN and BIHOST1 into the Identity Store. Make sure that you use a different alias for each of the certificate and key pair imported.

   Syntax:

   java utils.ImportPrivateKey
         -certfile cert_file
         -keyfile private_key_file
         [-keyfilepass private_key_password]
         -keystore keystore
         -storepass storepass
         [-storetype storetype]
         -alias alias 
         [-keypass keypass]

   Note:

   The default keystore_type is jks.

   Examples:

   java utils.ImportPrivateKey\ 
        -certfile KEYSTORE_HOME/ADMINVHN.example.com_cert.pem\
        -keyfile KEYSTORE_HOME/ADMINVHN.example.com_key.pem\
        -keyfilepass password\
        -keystore appIdentityKeyStore.jks\ 
        -storepass password\
        -alias ADMINVHN\
        -keypass password
   
   java utils.ImportPrivateKey\ 
        -certfile KEYSTORE_HOME/BIHOST1.example.com_cert.pem\
        -keyfile KEYSTORE_HOME/BIHOST1.example.com_key.pem\
        -keyfilepass password\
        -keystore appIdentityKeyStore.jks\
        -storepass password\ 
        -alias BIHOST1\
        -keypass password

2. Repeat the java importPrivateKey command for each of the remaining host-specific certificate and key pairs. (for example, for BIHOST1, BIHOST2).

   Note:

   Make sure to use a unique alias for each certificate and key pair imported.

3. For Dynamic clusters, import the wildcard certificate and private key pair by using the custom id alias of WILDCARD.

   Example:

   ${JAVA_HOME}/bin/java utils.ImportPrivateKey \ 
   -certfile ${KEYSTORE_HOME}/WILDCARD.example.com_cert.pem \ 
   -keyfile ${KEYSTORE_HOME}/WILDCARD.example.com_key.pem \ 
   -keyfilepass password \ 
   -keystore ${KEYSTORE_HOME}/appIdentityKeyStore.jks \ 
   -storepass password \
   -alias WILDCARD \ 
   -keypass password

1. Copy the standard java keystore to create the new trust keystore since it already contains most of the root CA certificates needed.

   Oracle does not recommend modifying the standard Java trust key store directly. Copy the standard Java keystore CA certificates located under the WL_HOME/server/lib directory to the same directory as the certificates. For example:

   cp WL_HOME/server/lib/cacerts KEYSTORE_HOME/appTrustKeyStore.jks
   

2. Use the keytool utility to change the default password.

   The default password for the standard Java keystore is changeit. Oracle recommends that you always change the default password, as follows:

   keytool -storepasswd -new NewPassword -keystore TrustKeyStore -storepass Original_Password
   

   For example:

   keytool -storepasswd -new password -keystore appTrustKeyStore.jks -storepass changeit
   

3. Import the CA certificate into the appTrustKeyStore by using the keytool utility.

   The CA certificate CertGenCA.der is used to sign all certificates generated by the utils.CertGen tool and is located at WL_HOME/server/lib directory.

   Use the following syntax to import the certificate:

   keytool -import -v -noprompt -trustcacerts -alias AliasName -file CAFileLocation -keystore KeyStoreLocation -storepass KeyStore_Password
   

   For example:

   keytool -import -v -noprompt -trustcacerts -alias clientCACert -file WL_HOME/server/lib/CertGenCA.der -keystore appTrustKeyStore.jks -storepass password
   

1. Access the site on SSL with a browser (this adds the server's certificate to the browser's repository).
2. Obtain the certificate from the load balancer. You can obtain the load balancer certificate using a browser such as Firefox. However, the easiest way to obtain the certificate is to use the openssl command. The syntax of the command is as follows:
   openssl s_client -connect LOADBALANCER -showcerts </dev/null 2>/dev/null|openssl x509 -outform PEM > SHARED_CONFIG_DIR/keystores/LOADBALANCER.pem

   For example:

   openssl s_client -connect login.example.com:443 -showcerts </dev/null 2>/dev/null|openssl x509 -outform PEM > SHARED_CONFIG_DIR/keystores/login.example.com.pem

3. Use the keytool to import the load balancer's certificate into the truststore:

   For example:

   keytool -import -file /oracle/certificates/bi.example.com.crt -v -keystore appTrustKeyStore.jks
   

1. Log in to BIHOST1 and open the following file with a text editor:

   ASERVER_HOME/bin/setDomainEnv.sh
   

2. Replace reference to the existing DemoTrust.jks entry with the following entry:

   Note:

   All the values for EXTRA_JAVA_PROPERTIES must be on one line in the file, followed by the export command on a new line.

   EXTRA_JAVA_PROPERTIES="-Djavax.net.ssl.trustStore=/u01/oracle/config/keystores/appTrustKeyStore.jks ${EXTRA_JAVA_PROPERTIES} ......." 
   export EXTRA_JAVA_PROPERTIES

3. Make the same change to the setDomainEnv.sh file in the MSERVER_HOME/bin directory BIHOST1, BIHOST2, WEBHOST1, and WEBHOST2.

   Note:

   The setDomainEnv.sh file cannot be copied between ASERVER_HOME/bin and MSERVER_HOME/bin as there are differences in the files for these two domain home locations. TheMSERVER_HOME/bin/setDomainEnv.sh file can be copied between hosts.

   WebLogic Serverautomatically overwrites the setDomainEnv.sh file after each domain extension. Some patches may also replace this file. Verify your customizations to setDomainEnv.sh after each of these types of maintenance operations.

1. Log in to the Administration Console, and click Lock & Edit.
2. Navigate based on the Managed Server type:
   For configured Managed Servers:
   1. In the Domain Structure pane, expand Environment and select Servers.
   2. Click the name of the server for which you want to configure the identity and trust keystores.
   For dynamic Managed Servers:
   1. In the Domain Structure pane, expand Environment, then Clusters, and then select Server Templates.
   2. Click the name of the appropriate server template for which you want to configure the identity and trust keystores.
3. Select Configuration, and then Keystores.
4. In the Keystores field, click Change, and select Custom Identity and Custom Trust method for storing and managing private keys and digital certificate pairs and trusted CA certificates, and click Save.
5. In the Identity section, define attributes for the identity keystore.

   • Custom Identity Keystore: Enter the fully qualified path to the identity keystore:

     KEYSTORE_HOME/appIdentityKeyStore.jks 
     

   • Custom Identity Keystore Type: Leave this field blank, it defaults to JKS.

   • Custom Identity Keystore Passphrase: Enter the password Keystore_Password you provided in Creating an Identity Keystore Using the utils.ImportPrivateKey Utility

     This attribute may be optional or required depending on the type of keystore. All keystores require the passphrase in order to write to the keystore. However, some keystores do not require the passphrase to read from the keystore. WebLogic Server reads only from the keystore, so whether or not you define this property depends on the requirements of the keystore.

6. In the Trust section, define properties for the trust keystore:

   • Custom Trust Keystore: Enter the fully qualified path to the trust keystore:

     KEYSTORE_HOME/appTrustKeyStore.jks 
     

   • Custom Trust Keystore Type: Leave this field blank, it defaults to JKS.

   • Custom Trust Keystore Passphrase: The password you provided as the New_Password value in Creating a Trust Keystore Using the Keytool Utility.

     As mentioned in the previous step, this attribute may be optional or required depending on the type of keystore.

7. Click Save.
8. To activate these changes, in the Change Center of the Administration Console, click Activate Changes.
9. Click Lock & Edit.
10. Select Configuration, then SSL.
11. Update the SSL Identity details as follows:
    1. In the Private Key Alias field, enter the alias value for the appropriate private key.

       • With a Static Cluster: Enter the alias that corresponds to the host the managed server listens on.

       • With a Dynamic Cluster: Enter the wildcard alias so any dynamic managed server can match any server.

    2. In the Private Key Passphrase and the Confirm Private Key Passphrase fields, enter the password for the keystore that you created in Creating an Identity Keystore Using the utils.ImportPrivateKey Utility.
12. Click Save.
13. If you are updating a server template SSL configuration for a dynamic cluster, perform these additional tasks:
    1. Click the Advanced link at the bottom of the SSL view.
    2. Select the Custom Hostname Verifier option from the HostName Verification menu.
    3. Set the Custom Hostname Verifier value to: weblogic.security.utils.SSLWLSWildcardHostnameVerifier.
    4. Click Save.
14. Click Activate Changes in the Administration Console's Change Center to make the changes take effect.
15. Restart the Administration Server.
16. Restart the Managed Servers where the keystore has been updated.

    Note:

    The fact that servers can be restarted by using the Administration Console and Node Manager is a good verification that the communication between Node Manager, Administration Server, and the managed servers is correct.