1. Contact My Oracle Support to request hardware replacement.

1. Run syscheck in Verbose mode to verify a fan failure using the following command:

   [admusr@hostname1351690497 ~]$ sudo syscheck -v hardware fan
   Running modules in class hardware...
            fan: Checking Status of Server Fans.
   *         fan: FAILURE:: MAJOR::3000000000000001 -- Server Fan Failure. This test uses the leaky bucket algorithm.
   *         fan: FAILURE:: Fan RPM is too low, fana: 0, CHIP: FAN
   One or more module in class "hardware" FAILED
   
   LOG LOCATION: /var/TKLC/log/syscheck/fail_log
   

2. Refer to the procedure for determining the location of the fan assembly that contains the failed fan and replacing a fan assembly in the appropriate hardware manual. After you have opened the front lid to access the fan assemblies, determine whether any objects are interfering with the fan rotation. If some object is interfering with fan rotation, remove the object.
3. Run "syscheck -v hardware fan" (see Running syscheck Through the EPAP GUI)
   • If the alarm has been cleared (as shown below), the problem is resolved

   [admusr@hostname1351691862 ~]$ sudo syscheck -v hardware fan
   Running modules in class hardware...
   Discarding cache...
            fan: Checking Status of Server Fans.
            fan: Fan is OK. fana: 1, CHIP: FAN
            fan: Server Fan Status OK.
                                    OK
   

   • If the alarm has not been cleared (as shown below) continue with the next step

   [admusr@hostname1351690497 ~]$ sudo syscheck -v hardware fan
   Running modules in class hardware...
            fan: Checking Status of Server Fans.
   *         fan: FAILURE:: MAJOR::3000000000000001 -- Server Fan Failure. This test uses the leaky bucket algorithm.
   *         fan: FAILURE:: Fan RPM is too low, fana: 0, CHIP: FAN
   One or more module in class "hardware" FAILED
   
   LOG LOCATION: /var/TKLC/log/syscheck/fail_log
   

4. Contact My Oracle Support.

1. Run syscheck in Verbose mode (see procedure Running the System Health Check.).
2. Contact the My Oracle Support and provide the system health check output.

   Note:

   Refer to Hardware and Installation Guide for E5-APP-B, Field Replaceable Units (FRUs) section for information about installing a hard disk drive.
3. Contact My Oracle Support.

1. Run syscheck in Verbose mode (see procedure Run Syscheck Manually).).
2. Contact My Oracle Support and provide the system health check output.

1. Contact My Oracle Support.

1. Rerun syscheck in verbose mode (see procedure Running the System Health Check).
   • If the alarm has been cleared, the problem is solved.
   • If the alarm has not been cleared, contact My Oracle Support.
2. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Verify that a customer-supplied cable labeled TO CUSTOMER NETWORK is securely connected to the upper right port on the rear of the server on the EAGLE backplane.to the appropriate server. Follow the cable to its connection point on the local network and verify this connection is also secure.
2. Test the customer-supplied cable labeled TO CUSTOMER NETWORK with an Ethernet Line Tester. If the cable does not test positive, replace it.
3. Have your network administrator verify that the network is functioning properly.
4. If no other nodes on the local network are experiencing problems and the fault has been isolated to the server or the network administrator is unable to determine the exact origin of the problem, contact My Oracle Support.

1. Refer to MPS-specific documentation for information regarding this alarm.
2. Contact My Oracle Support.
3. Perform the following:
   1. Verify that both servers are powered on by confirming that the POWER LEDs on both servers are illuminated green.
   2. Verify that the switch is powered on.
   3. Verify that the switch does not have any fault lights illuminated.
   4. Verify that the eth01 cable is securely connected to the top port on the server that is reporting the error.
   5. Trace the eth01 cable to the switch. Verify that the eth01 cable is securely connected at correct point of the customer uplink.
   6. Verify that the cable connecting the switches is securely connected at both switches.
4. Run syscheck (see Running syscheck Through the EPAP GUI).
   1. If the alarm is cleared, the problem is resolved.
   2. If the alarm is not cleared, continue with the next step.
5. Verify that the cable from eth01 to the switch tests positive with an Ethernet Line Tester. Replace any faulty cables.
6. If the problem persists, call My Oracle Support.
7. Perform general IP troubleshooting.
   The syscheck utility reports this error when it tries to ping hosts dsmm-a and dsmm-b a set number of times and fails. This failure could mean any number of things are at fault on the network, but general IP troubleshooting will usually resolve the issue. The platcfg utility can be used to help isolate the problem. To access the platcfg utility:
   1. Log in as platcfg to the server that is generating the alarm.

      
      Login:  platcfg
      Password: <Enter platcfg password>
      

   2. To display various network information and statistics, select menu options:Diagnostics->Network Diagnostics->Netstat
   3. To ping the dsmb-a and/or dsmb-b select menu options:Diagnostics->Network Diagnostics->Ping
   4. To verify no routing issues exist, select menu options:Diagnostics->Network Diagnostics->Traceroute
8. Run savelogs to gather all application logs, (see Saving Logs Using the EPAP GUI).
9. Run savelogs_plat to gather system information for further troubleshooting, (see Saving Logs Using the EPAP GUI ), and contact My Oracle Support.

1. Refer to MPS-specific documentation for information regarding this alarm.
2. Contact My Oracle Support.
3. Perform the following:
   1. Verify that both servers are powered on by confirming that the POWER LEDs on both servers are illuminated green.
   2. Verify that the switch is powered on.
   3. Verify that the switch does not have any fault lights illuminated.
   4. Verify that the eth01 cable is securely connected to the top port of the server that is reporting the error.
   5. Trace the eth01 cable to the switch. Verify that the eth01 cable is securely connected to the correct point of the customer uplink.
   6. Verify that the cable connecting the switches is securely connected at both switches.
4. Run syscheck (see Running syscheck Through the EPAP GUI).
   1. If the alarm is cleared, the problem is resolved.
   2. If the alarm is not cleared, continue with the next step.
5. Verify that the cable from eth01 to the hub tests positive with an Ethernet Line Tester. Replace any faulty cables.
6. If the problem persists, call My Oracle Support for assistance.
7. Perform general IP troubleshooting.
   The syscheck utility reports this error when it tries to ping hosts dsmb-a and dsmb-b a set number of times and fails. This failure could mean any number of things are at fault on the network, but general IP troubleshooting will usually resolve the issue. The platcfg utility can be used to help isolate the problem. To access the platcfg utility:
   1. Log in as platcfg to the server that is generating the alarm.

      
      Login:  platcfg
      Password: <Enter  platcfg
       password>
      

   2. To display various network information and statistics, select menu options:Diagnostics->Network Diagnostics->Netstat
   3. To ping the dsmm-a and/or dsmm-b select menu options:Diagnostics->Network Diagnostics->Ping
   4. To verify no routing issues exist, select menu options:Diagnostics->Network Diagnostics->Traceroute
8. Run savelogs to gather all application logs, (see Saving Logs Using the EPAP GUI).
9. Run savelogs_plat to gather system information for further troubleshooting, (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.

1. Refer to MPS-specific documentation for information regarding this alarm.
2. Contact My Oracle Support.
3. Verify that both servers are powered on by confirming that the POWER LEDs on both servers are illuminated green.
4. Verify that the eth03 cable is securely connected to the 2nd from bottom ethernet port on both Server A and Server B.
5. Test the eth03 cable with an Ethernet Line Tester that is set to test a straight-through cable.
6. If the cable does not test positive, replace it.
7. If the problem persists, call My Oracle Support for assistance. Switch B may have failed.
8. Perform general IP troubleshooting.
   The syscheck utility reports this error when it tries to ping hosts sync-a and sync-b a set number of times and fails. This failure could mean any number of things are at fault on the network, but general IP troubleshooting will usually resolve the issue. The platcfg utility can be used to help isolate the problem. To access the platcfg utility:
   1. Log in as platcfg to the server that is generating the alarm.

      
      Login:  platcfg
      Password: <Enter platcfg password>
      

   2. To display various network information and statistics, select menu options:Diagnostics->Network Diagnostics->Netstat
   3. To ping the sync-a and/or sync-b select menu options:Diagnostics->Network Diagnostics->Ping
   4. To verify no routing issues exist, select menu options:Diagnostics->Network Diagnostics->Traceroute
9. Run savelogs to gather all application logs (see Saving Logs Using the EPAP GUI).
10. Run savelogs_plat to gather system information for further troubleshooting, (see Saving Logs Using the EPAP GUI), and contact Platform Engineering.

1. Run syscheck.
2. Examine the syscheck output to determine if the file system /var/TKLC/epap/free/var/TKLC/elap/free is low on space. If it is, continue to the next step. Otherwise, go to 4
3. If possible, recover space on the free partition by deleting unnecessary files:
   1. Log in to the EPAP GUI.
   2. Select Debug>Manage Logs & Backups.
      A screen similar to Figure 4-1 is displayed. This screen displays the information about the total amount of space allocated for and currently used by logs and backups. The display includes logs and backup files which might be selected for deletion to recover additional disk space.

      Figure 4-1 Manage Logs and Backups

      
   3. Click the checkbox of each file that you want to delete and then click Delete Selected File(s).
4. If the file system mounted on /var/TKLC/epap/logs/var/TKLC/elap/logs is the file system that syscheck is reporting to be low on space, execute the following steps:
   1. Log into the server generating the alarm as the admusr:

      
      Login:  admusr
      Password:<Enter admusr password>
      

   2. Change to the /var/TKLC/epap/logs directory:

      $ cd /var/TKLC/epap/logs

   3. Confirm that you are in the /var/TKLC/epap/logs directory:

      $ pwd
      /var/TKLC/epap/logs
      

   4. When the pwd command is executed, if /var/TKLC/epap/logs is not output, go back to sub-step b.
   5. Look for files that you want to delete and execute an rm command for each:

      $ sudo rm <filename>

      where <filename> is replaced by the name of the file to be deleted.
   6. Re-run syscheck.

      • If the alarm is cleared, the problem is solved.

      • If the alarm is not cleared, go to the next step.

5. If syscheck has determined inodes have been depleted or a file system has a different number of blocks, skip to 11.
6. Execute the following steps to collect and remove any core files from the server.
   Core files can occupy a large amount of disk space and may be the cause of this alarm:
   1. Log into the server generating the alarm as the admusr:

      
      Login:  admusr
      Password:<Enter admusr password>
      

   2. To list core files on the server, execute the following command, where <mountpoint> is the file system’s mount point:

      $ sudo find <mountpoint> -name core.[0-9]\* -print -exec gzip -9 {} \;

      Note:

      The find command shown above will list any core files found and then compress and rename the file adding a “.gz” extension.

      If any core files are found, transfer them off of the system and save them aside for examination by Oracle. Once a copy of a compressed file has been saved it is safe to delete it from the server.
   3. Re-run syscheck.

      • If the alarm has been cleared, the problem is resolved.

      • If the alarm has not been cleared, proceed to 7.

7. Execute the following steps if the file system reported by syscheck is /tmp, otherwise skip to 11.
   1. Log into the server generating the alarm as the admusr:

      
      Login:  admusr
      Password:<Enter admusr password>
      

   2. Change to the /tmp directory:

      $ cd /tmp

   3. Confirm that you are in the /tmp directory:

      $ pwd
      /tmp
      

   4. When the pwd command is executed, if /tmp is not output, go back to 5.
   5. Look for possible candidates for deletion:

      $ ls *.iso *.bz2 *.gz *.tar *.tgz *.zip

   6. If any files that can be deleted exist, the output of the ls will show them. For each of the files listed, execute the rm command to delete the file:

      $ sudo rm <filename>

   7. Run syscheck.

      • If the alarm is cleared, the problem is solved.

      • If the alarm is not cleared, go to the next step.

   8. Upon a reboot the system will clean the /tmp directory.
      To reboot the system, issue the following command:

      $ sudo shutdown -r now

   9. Re-run syscheck.

      • If the alarm has been cleared, the problem is resolved.

      • If the alarm has not been cleared, proceed to the next step.

8. Execute the following steps if the file system reported by syscheck is /var, otherwise skip to 11.
   1. Log into the server generating the alarm as the admusr:

      
      Login:  admusr
      Password:<Enter admusr password>
      

   2. Change to the /var/tmp directory:

      $ cd /var/tmp

   3. Confirm that you are in the /var/tmp directory:

      $ pwd
      /var/tmp
      

   4. When the pwd command is executed, if /var/tmp is not output, go back to 6.
   5. Since all files in this directory can be safely deleted, execute the rm * command to delete all files from the directory:
      $ sudo rm -i *
   6. Re-run syscheck.

      • If the alarm is cleared, the problem is solved.

      • If the alarm is not cleared, go to 11.

9. Execute the following steps if the file system reported by syscheck is /var/TKLC, otherwise skip to 11.
   1. Log into the server generating the alarm as the admusr:

      
      Login:  admusr
      Password:<Enter admusr password>
      

   2. Change to the /var/TKLC/upgrade directory:

      $ cd /var/TKLC/upgrade

   3. Confirm that you are in the /var/TKLC/upgrade directory:

      $ pwd
      /var/TKLC/upgrade
      

   4. When the pwd command is executed, if /var/TKLC/upgrade is not output, go back to 6.
   5. Since all files in this directory can be safely deleted, execute the rm * command to delete all files from the directory:

      $ sudo rm -i *

   6. Run syscheck.

      • If the alarm is cleared, the problem is solved.

      • If the alarm is not cleared, go to 11.

10. For any other file system, execute the following command, where <mountpoint> is the file system’s mount point:

    $ sudo find <mountpoint> -type f -exec du -k {} \; | sort -nr > /tmp/file_sizes.txt

    This will produce a list of files in the given file system sorted by file size in the file /tmp/file_sizes.txt.

    Note:

    The find command noted above could possibly take a few minutes to complete if the given mountpoint contains many files.

    Do not delete any file unless you know for certain that it is not needed. Continue to 11 .
11. Run savelogs to gather all application logs (see Saving Logs Using the EPAP GUI).
12. Run savelogs_plat to gather system information for further troubleshooting, (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.
13. Run syscheck in Verbose mode.
14. Contact My Oracle Support.

1. Run syscheck in Verbose mode.
   The output should indicate one of the following errors:

   • The default router at <IP_address> cannot be pinged.

     This error indicates that the router may not be operating or is unreachable. If the syscheck Verbose output returns this error, go to 4.

   • The default route is not on the provisioning network.

     This error indicates that the default route has been defined in the wrong network. If the syscheck Verbose output returns this error, go to 4.

   • An active route cannot be found for a configured default route.

     This error indicates that a mismatch exists between the active configuration and the stored configuration. If the syscheck Verbose output returns this error, go to 5.

2. Run syscheck in Verbose mode.
   If the output should indicates:

   
   The default router at <IP_address> cannot be pinged
   

   Go to 3, otherwise go to 4.

3. Perform the these substeps:
   1. Verify the network cables are firmly attached to the server, network switch, router, hub, and any other connection points.
   2. Verify that the configured router is functioning properly.
      Request that the network administrator verify the router is powered on and routing traffic as required.
   3. Request that the router administrator verify that the router is configured to reply to pings on that interface.
   4. If the alarm is cleared, the problem is resolved.
4. Perform the following substeps when syscheck Verbose output indicates:

   
   The default route is not on the provisioning network
   

   1. Obtain the proper Provisioning Network netmask and the IP address of the appropriate Default Route on the provisioning network.
      This information is maintained by the customer network administrators.
   2. Log in to the server with username epapconfig.
      The server designation at this site is displayed as well as hostname, hostid, Platform Version, Software Version, and date. Verify that the side displayed is the MPS that is reporting the problem. In this example, MPS A is reporting the problem. Enter option 2, Configure Network Interfaces Menu, from the EPAP Configuration Menu.

      
      MPS Side A:  hostname: mpsa-d1a8f8  hostid: 80d1a8f8
                   Platform Version: x.x.x-x.x.x
                   Software Version: EPAP x.x.x-x.x.x
                   Wed Jul 17 09:51:47 EST 2002
       /-------EPAP Configuration Menu--------\
      /----------------------------------------\
      |  1 | Display Configuration             |
      |----|-----------------------------------|
      |  2 | Configure Network Interfaces Menu |
      |----|-----------------------------------|
      |  3 | Set Time Zone                     |
      |----|-----------------------------------|
      |  4 | Exchange Secure Shell Keys        |
      |----|-----------------------------------|
      |  5 | Change Password                   |
      |----|-----------------------------------|
      |  6 | Platform Menu                     |
      |----|-----------------------------------|
      |  7 | Configure NTP Server              |
      |----|-----------------------------------|
      |  8 | PDB Configuration Menu            |
      |----|-----------------------------------|
      |  9 | Security                          |
      |----|-----------------------------------|
      | 10 | SNMP Configuration                |
      |----|-----------------------------------|
      | 11 | Configure Alarm Feed              |
      |----|-----------------------------------|
      | 12 | Configure Query Server            |
      |----|-----------------------------------|
      | 13 | Configure Query Server Alarm Feed |
      |----|-----------------------------------|
      | 14 | Configure SNMP Agent Community    |
      |----|-----------------------------------|
      |  e | Exit                              |
      \----------------------------------------/
      Enter Choice:  2
      

   3. Enter option 1, Configure Provisioning Network, from the Configure Network Interfaces Menu.
      The submenu for configuring communications networks and other information is displayed.

      
       /-----Configure Network Interfaces Menu----\
      /--------------------------------------------\
      |  1 | Configure Provisioning Network        |
      |----|---------------------------------------|
      |  2 | Configure Sync Network                |
      |----|---------------------------------------|
      |  3 | Configure DSM Network                 |
      |----|---------------------------------------|
      |  4 | Configure Backup Provisioning Network |
      |----|---------------------------------------|
      |  5 | Configure Forwarded Ports             |
      |----|---------------------------------------|
      |  6 | Configure Static NAT Addresses        |
      |----|---------------------------------------|
      |  7 | Configure Provisioning VIP Addresses  |
      |----|---------------------------------------|
      |  e | Exit                                  |
      \--------------------------------------------/
      Enter choice:  1
      

   4. Enter option 1, IPv4 Configuration (or 2 for IPv6 Configuration), from the Configure Provisioning Network Menu.

      MPS Side A:  hostname: EPAP17  hostid: f80a110f
                  Platform Version: 6.0.2-7.0.3.0.0_86.45.0
                  Software Version: EPAP 161.0.28-16.1.0.0.0_161.28.0
                  Wed Jun 15 01:33:56 EDT 2016
      
      /-----Configure Provisioning Network Menu-\
      /-------------------------------------------\
      |  1 | IPv4 Configuration                   |
      |----|--------------------------------------|
      |  2 | IPv6 Configuration                   |
      |----|--------------------------------------|
      |  e | Exit                                 |
      \-------------------------------------------/
      
      Enter Choice:  1
      

   5. The following warning is displayed. Type Y and press Enter.

      
      EPAP software and PDBA are running. Stop them? [N]  Y
      

   6. The EPAP A provisioning network IP address is displayed.

      
      Verifying connectivity with mate ...
      Enter the EPAP A provisioning network IP Address [192.168.61.90]:
      

   7. Press Enter after each address is displayed until the Default Route address is displayed.

      
      Verifying connectivity with mate ...
      Enter the EPAP A provisioning network IP Address [192.168.61.90]: 
      Enter the EPAP B provisioning network IP Address [192.168.61.91]: 
      Enter the EPAP provisioning network netmask [255.255.255.0]: 
      Enter the EPAP provisioning network default router IP Address: 192.168.61.250
      

   8. If the default router IP address is incorrect, type the correct address and press Enter.
   9. After you have verified or corrected the Provisioning Network configuration information, enter e to return to the Configure Network Interfaces Menu.
   10. Enter e again to return to the EPAP Configuration Menu.
   11. Go to 6.
5. Perform the following substeps to reboot the server if the syscheck output indicates the following error. Otherwise, go to 6:

   
   An active route cannot be found for a configured default route
   

   1. Log in as epapconfig on the server console.
      Enter option 6, Platform Menu, from the EPAP Configuration Menu.

      
       /-------EPAP Configuration Menu--------\
      /----------------------------------------\
      |  1 | Display Configuration             |
      |----|-----------------------------------|
      |  2 | Configure Network Interfaces Menu |
      |----|-----------------------------------|
      |  3 | Set Time Zone                     |
      |----|-----------------------------------|
      |  4 | Exchange Secure Shell Keys        |
      |----|-----------------------------------|
      |  5 | Change Password                   |
      |----|-----------------------------------|
      |  6 | Platform Menu                     |
      |----|-----------------------------------|
      |  7 | Configure NTP Server              |
      |----|-----------------------------------|
      |  8 | PDB Configuration Menu            |
      |----|-----------------------------------|
      |  9 | Security                          |
      |----|-----------------------------------|
      | 10 | SNMP Configuration                |
      |----|-----------------------------------|
      | 11 | Configure Alarm Feed              |
      |----|-----------------------------------|
      | 12 | Configure Query Server            |
      |----|-----------------------------------|
      | 13 | Configure Query Server Alarm Feed |
      |----|-----------------------------------|
      | 14 | Configure SNMP Agent Community    |
      |----|-----------------------------------|
      |  e | Exit                              |
      \----------------------------------------/
      Enter Choice:  6
      

   2. Enter option 2, Reboot MPS, from the EPAP Platform Menu.
      At the prompt, enter the identifier of the server to which you are logged in (A or B). In this example, A is used.

      MPS Side A:  hostname: EPAP17  hostid: f80a110f
                   Platform Version: 6.0.2-7.0.3.0.0_86.45.0
                   Software Version: EPAP 161.0.28-16.1.0.0.0_161.28.0
                   Wed Jun 15 01:34:39 EDT 2016
      
      /-----EPAP Platform Menu-\
      /--------------------------\
      |  1 | Initiate Upgrade    |
      |----|---------------------|
      |  2 | Reboot MPS          |
      |----|---------------------|
      |  3 | MySQL Backup        |
      |----|---------------------|
      |  4 | RTDB Backup         |
      |----|---------------------|
      |  5 | PDB Backup          |
      |----|---------------------|
      |  e | Exit                |
      \--------------------------/
      
      
      Enter Choice:  2
      Reboot MPS A, MPS B or BOTH? [BOTH]:  A
      Reboot local MPS...
      

   3. Wait for the reboot to complete.
   4. Go to 6.
6. Run syscheck.
   • If the alarm is cleared, the problem is resolved.
   • If the alarm is not cleared, go to the next step.
7. Contact My Oracle Support with the syscheck output collected in the previous steps.
8. Run savelogs to gather all application logs (see Saving Logs Using the EPAP GUI).
9. Run savelogs_plat to gather system information for further troubleshooting (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.
10. Run syscheck in Verbose mode.
    The output should indicate one of the following errors:

    • The default router at <IP_address> cannot be pinged.

      This error indicates that the router may not be operating or is unreachable. If the syscheck Verbose output returns this error, go to 4.

    • The default route is not on the provisioning network.

      This error indicates that the default route has been defined in the wrong network. If the syscheck Verbose output returns this error, contact My Oracle Support.

    • An active route cannot be found for a configured default route.

      This error indicates that a mismatch exists between the active configuration and the stored configuration. If the syscheck Verbose output returns this error, contact My Oracle Support.

11. Perform the following substeps when syscheck Verbose output indicates:

    
    The default router at <IP_address> cannot be pinged
    

    1. Verify the network cables are firmly attached to the server, network switch, router, hub, and any other connection points.
    2. Verify that the configured router is functioning properly.
       Request that the network administrator verify the router is powered on and routing traffic as required.
    3. Request that the router administrator verify that the router is configured to reply to pings on that interface.
    4. Rerun syscheck:
       • If the alarm has been cleared, the problem is solved.
       • If the alarm has not been cleared, contact My Oracle Support.
12. Contact My Oracle Support.

1. Ensure that nothing is blocking the fan's intake. Remove any blockage.
2. Verify that the temperature in the room is normal (see the following table). If it is too hot, lower the temperature in the room to an acceptable level.

   Table 4-2 Server Environmental Conditions

   Ambient Temperature	Operating: 5 degrees C to 40 degrees C Exceptional Operating Limit: 0 degrees C to 50 degrees C Storage: -20 degrees C to 60 degrees C
   Ambient Temperature	Operating: 5° C to 35° C Storage: -20° C to 60° C
   Relative Humidity	Operating: 5% to 85% non-condensing Storage: 5% to 950% non-condensing
   Elevation	Operating: -300m to +300m Storage: -300m to +1200m
   Heating, Ventilation, and Air Conditioning	Capacity must compensate for up to 5100 BTUs/hr for each installed frame. Calculate HVAC capacity as follows: Determine the wattage of the installed equipment. Use the formula: watts x 3.143 = BTUs/hr

   Note:

   Be prepared to wait the appropriate period of time before continuing with the next step. Conditions need to be below alarm thresholds consistently for the alarm to clear. The alarm may take up to five minutes to clear after conditions improve. It may take about ten minutes after the room returns to an acceptable temperature before syscheck shows the alarm cleared.
3. Verify that the temperature in the room is normal. If it is too hot, lower the temperature in the room to an acceptable level.

   Note:

   Be prepared to wait the appropriate period of time before continuing with the next step. Conditions need to be below alarm thresholds consistently for the alarm to clear. It may take about ten minutes after the room returns to an acceptable temperature before the alarm cleared.
4. Run syscheck Check to see if the alarm has cleared
   • If the alarm has been cleared, the problem is resolved.
   • If the alarm has not been cleared, continue with the next step.
5. Run syscheck Check to see if the alarm has cleared
   • If the alarm has been cleared, the problem is resolved.
   • If the alarm has not been cleared, continue with the next step.
6. Replace the filter (refer to the appropriate hardware manual).

   Note:

   Be prepared to wait the appropriate period of time before continuing with the next step. Conditions need to be below alarm thresholds consistently for the alarm to clear. The alarm may take up to five minutes to clear after conditions improve. It may take about ten minutes after the filter is replaced before syscheck shows the alarm cleared.
7. Run syscheck (see Running the System Health Check).
   • If the alarm has been cleared, the problem is resolved.
   • If the alarm has not been cleared, continue with the next step.
8. If the problem has not been resolved, contact My Oracle Support.

1. Contact My Oracle Support.

1. Immediately contact My Oracle Support for assistance with a disk replacement.
2. Perform the recovery procedures for the other alarms that accompany this alarm.
3. If the problem has not been resolved, contact My Oracle Support.

1. Contact My Oracle Support.

1. No recovery necessary. If the condition persists, contact My Oracle Support to request hardware replacement.

1. My Oracle Support should do the following:
   1. Check in the MSA administration console (web-application) that correct "volumes" on MSA exist, and read/write access is granted to the blade server.
   2. Check if multipath daemon/service is running on the blade server: service multipathd status. Resolution:
      1. start multipathd: service multipathd start
   3. Check output of "multipath -ll": it shows all multipath devices existing in the system and their access paths; check that particular /dev/sdX devices exist. This may be due to SCSI bus and/or FC HBAs haven't been rescanned to see if new devices exist. Resolution:
      1. run "/opt/hp/hp_fibreutils/hp_rescan -a",
      2. "echo 1 > /sys/class/fc_host/host*/issue_lip",
      3. "echo '- - -' > /sys/class/scsi_host/host*/scan"
   4. Check if syscheck::disk::multipath test is configured to monitor right multipath devices and its access paths: see output of "multipath -ll" and compare them to "syscheckAdm disk multipath - -get - -var=MPATH_LINKS" output. Resolution:
      1. configure disk::multipath check correctly.
2. Contact My Oracle Support.

1. Verify cabling between the offending port and remote side.
2. Verify networking on the remote end.
3. If problem persists, contact My Oracle Support to verify port settings on both the server and the switch.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Log in to the EPAPGUI (see Accessing the EPAP GUI).
2. View the EPAP status on the banner.
   • If the mate EPAP status is DOWN, go to 3 .
   • If the mate EPAP status is ACTIVE or STANDBY, go to 4.
3. Select the Select Mate menu item to change to the mate EPAP.
4. Select Process Control > Start Software to start the mate EPAP software.
5. View the EPAP status on the banner.
   • If the mate EPAP status is ACTIVE or STANDBY, the problem is resolved.
   • If the mate EPAP status is still DOWN, continue with 6.
6. Select the Select Mate menu item to change back to the side that reported the alarm.
7. Stop and start the software on the side that is reporting the alarm (see Restarting the EPAP Software).
8. If the problem persists, run savelogs to gather system information for further troubleshooting (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.

1. Log in to the EPAPGUI (see Accessing the EPAP GUI).
2. View the EPAP status on the banner.
   • If the mate EPAP status is DOWN, go to 3.
   • If the mate EPAP status is ACTIVE or STANDBY, go to 4.
3. Select Process Control > Start Software to start the mate EPAP software.
4. Select the Select Mate menu item to change to the mate EPAP.
5. Determine whether the alarm has cleared by verifying whether it is still being displayed in the banner or in the Alarm View window.
   • If the alarm has cleared, the problem is resolved.
   • If the alarm has not yet cleared, continue with 6.
6. Make sure that you are logged into the side opposite from the side reporting the alarm.

   If it is necessary to change sides, select the Select Mate menu item to change to the side opposite the side that reported the alarm.

7. Stop and start the software on the side that is reporting the alarm (see Restarting the EPAP Software).
8. Select RTDB>View RTDB Status to verify that the RTDB status on both sides is coherent, as shown in Figure 4-2.

   Figure 4-2 Coherent RTDB Status

   
9. If the problem persists, run savelogs to gather system information for further troubleshooting (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.

1. At the EAGLE input terminal, enter the rept-stat-mps command to verify the status.

   Refer to Commands User's Guide to interpret the output.

2. If the problem does not clear within 2 hours with an "EPAP Available" notice, capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. Call My Oracle Support for assistance.

1. Call My Oracle Support for assistance.

1. Check the physical connections between the local server and the Service Module cards on the EAGLE.

   Make sure the connectors are firmly seated.

2. Stop and restart the software on the side that is reporting the alarm (see Restarting the EPAP Software).
3. Capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. Restart EPAP software. See Restarting the EPAP Software
2. Capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. Capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. Log in to the User Interface screen of EPAP A (see Accessing the EPAP GUI)
2. Check the banner information above the menu to verify that you are logged into the EPAP A that is reporting the problem.
   If it is necessary to switch to EPAP B, click the Select Mate menu item.
3. From the menu, select RTDB>View RTDB Status to display status information about the RTDBs.
   Figure 4-3 shows an example of two Inconsistent RTDBs.

   Figure 4-3 Inconsistent RTDB Status

   
   If one RTDB is inconsistent and the other is coherent in a mated pair setup, proceed to 4. If both RTDBs on an EPAP paired setup are inconsistent, reload from the nearest EPAP site with a coherent RTDB. If all RTDBs are inconsistent, additional steps may be required to reload one RTDB from PDB and backup the new RTDB, then restore the remaining RTDBs.
4. Verify the PDB information on the RTDB Status view is correct before continuing.
5. Before attempting to copy the RTDB, the EPAP A software must be stopped by doing the following:

   Caution:

   If the software is not stopped as directed in 5.a through 5.c, the RTDB will become corrupted.
   1. Select Process Control>Stop Software to stop the software.
      The following warning appears:

      
      CAUTION: This action will stop all EPAP software processes, and will prevent the selected EPAP from updating the RTDB until the EPAP software is re-started (by executing the Start Software menu item).
      

   2. On the Stop EPAP Software screen, make sure the following item on the screen is checked: Check if you want the software to automatically start on reboot.
   3. Select the Stop EPAP Software button to stop the software.
   4. Select Select Mate from the menu to return to the EPAP that is reporting the problem.
6. Select RTDB>Maintenance>Reload from Remote.
   The screen shown in Figure 4-6 shows this function.

   Figure 4-4 Reload RTDB from Mate EPAP

   
7. Make sure that the Mate radio button is filled in, as shown in Figure 4-6 and click the Begin RTDB Reload from Remote button.
8. Click the Reload button as shown in Figure 4-6.
9. When the reload has completed, start the software on EPAP A by doing the following:
   1. Select Process Control > Start Software to start the software again.

      Make sure the following item on the screen is checked:

      Check if you want to start the PDBA software along with the EPAP software

   2. Select the Start EPAP Software button to start the software.
10. If the problem persists, capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

0452 - RTDB reload complete

1. If this alarm displays while an RTDB download is in progress, no further action is necessary.
2. If this alarm displays when an RTDB download is not in progress, capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. On the EAGLE, turn ON the optional EPAP Data Split feature to allow more room for the provisioned data.
2. On the EAGLE, turn ON the epap240m STP option (E5-SM8G-B card required) to allow more room for the provisioned data.
3. Turn ON the optional 120M DN and 120M IMSIs via Split database feature on the EPAP and Eagle to allow more room for the provisioned data.
4. Contact My Oracle Support for assistance.

1. No further action is necessary.

1. Log in to the User Interface screen of the EPAP (see 4000000000004000 - RTDB Reload Is Required)
2. Check the banner information above the menu to verify that you are logged into the EPAP that is reporting the problem.
   If it is necessary to switch to the problem EPAP, click the Select Mate menu item.
3. From the menu, select RTDB>View RTDB Status to display status information about the RTDBs.
   Figure 4-5 shows an example.

   Figure 4-5 RTDB Status

   
4. If the RTDB birthdays for both the local RTDB and the mate RTDB are the same, you can copy the mate’s RTDB to the local RTDB.
   If the RTDB birthdays are not the same, go to step 5.
5. Before attempting to copy the RTDB, you must stop the software on both sides by doing the following:

   Caution:

   If you do not stop the software on both sides, as directed in substeps 5a through 5c, the RTDBs will become corrupted.
   1. Select Process Control > Stop Software to stop the software.
      The following warning appears:

      
      CAUTION: This action will stop all EPAP software processes, and will prevent the selected EPAP from updating the RTDB until the EPAP software is re-started (by executing the Start Software menu item).
      

   2. On the Stop EPAP Software screen, make sure the following item on the screen is checked:
      Check if you want the software to automatically start on reboot.
   3. Select the Stop EPAP Software button to stop the software.
   4. Select Select Mate from the menu.
   5. Repeat substeps 5.a through 5.c on the other side.
   6. Select Select Mate from the menu to return to the EPAP that is reporting the problem.
6. Verify that you are logged in to the side that is reporting the problem.
7. Select RTDB>Maintenance>Reload from Remote.
   The screen shown in Figure 4-6 shows this function.

   Figure 4-6 Reload RTDB from Mate EPAP

   
8. Make sure that the Mate radio button is filled in, as shown in Figure 4-6 and click the Begin RTDB Reload from Remote button.
9. When the reload has completed, start the software on both sides by doing the following:
   1. Select Process Control > Start Software to start the software again.

      Make sure the following item on the screen is checked (this item applies only if performing this procedure on Side A):

      Check if you want to start the PDBA software along with the EPAP software.

   2. Select the Start EPAP Software button to start the software.
   3. Select Select Mate from the menu.
   4. Repeat substeps 9.a and 9.b on the other side.
10. If you wish to increase the size of the transaction logs, select PDBA> Maintenance > Transaction Log Params > Change Params as shown in Figure 4-7.

    Figure 4-7 Changing Transaction Log Parameters

    
11. If the problem persists, contact My Oracle Support.

1. Log in to the User Interface screen of the EPAP GUI (see Accessing the EPAP GUI).
2. Check the banner information above the menu for the PDBA status.
   1. If neither PDBA status is DOWN, go to 3.
   2. If status of one of the PDBAs is DOWN, continue with 4.
   Figure 4-8 shows an example in which the PDBA on EPAP B is DOWN.

   Figure 4-8 PDBA Down

   
   

   

   
   
3. View Tool Tips to verify the alarm and to verify that you are logged into the EPAP whose PDBA is DOWN.

   Figure 4-9 Alarms Details

   
   

   

   
   
   If it necessary to switch to the other PDBA, select PDBA>Select Other PDBA.
4. Attempt to start the PDBA by selecting PDBA> Process Control > Start PDBA Software.
   The window shown in Figure 4-10 is displayed.

   Figure 4-10 Start PDBA

   
5. Click the Start PDBA Software button.
6. When the PDBA software has been started, the window shown in Figure 4-11 displays, and within moments the banner will show the PDBA status as ACTIVE or STANDBY.
   If the status does not change to ACTIVE or STANDBY, continue to 7.

   Figure 4-11 PDBA Started

   
7. Check the status of the provisioning network.
   If problems exist in the provisioning network, fix them.
8. If the problem persists, run savelogs (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.

1. Log in to the User Interface screen of the EPAP (see Accessing the EPAP GUI).
2. Check the banner information above the menu to verify that you are logged into the problem EPAP indicated in the UAM.
   Select Select Mate if necessary to switch to the problem EPAP.
3. Perform Restarting the EPAP and PDBA.
4. Select RTDB>View RTDB Status and determine the homing policy for the PDBA.
   In the example shown in Figure 4-12, the Homing Policy shows that the Standby PDB is preferred for homing.

   Figure 4-12 Determining the Homing Policy

   
5. At the EPAP indicated by the Homing Policy, repeat 3 and 5 to restart the PDBA.
6. If the problem persists, run savelogs (see Saving Logs Using the EPAP GUI), and contact My Oracle Support.

1. Run savelogs (see Saving Logs Using the EPAP GUI).
2. Contact My Oracle Support.

1. Install Service Module cards in the attached EAGLE with sufficient memory to accommodate the expected size of the RTDB.
2. Contact My Oracle Support for assistance.

1. Contact My Oracle Support.

1. Log in to the User Interface screen of EPAP A (see Accessing the EPAP GUI)
2. Refer to the LNP Database Synchronization Manual for the correct procedures.
3. Select PDBA>View PDBA Status to verify that the PDBA proxy feature is enabled.
   The Local Proxy Status items only appear if the PDBA Proxy feature is enabled (See Figure 4-13).

   Figure 4-13 View PDBA Status Screen

   
4. Refer to Restarting the EPAP and PDBA to restart the PDBA.
5. If the problem persists, capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. Monitor this situation.
   If it does not improve, contact My Oracle Support for guidance.

1. Restart the EPAP software.
   See Restarting the EPAP Software.
2. 
   

   If the standby EPAP state is not changed back to STANDBY, capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. On the EAGLE, turn ON the optional EPAP Data Split feature to allow more room for the provisioned data.
2. On the EAGLE, turn ON the epap240m STP option (E5-SM8G-B card required) to allow more room for the provisioned data.
3. Turn ON the optional 120M DN and 120M IMSIs via Split database feature on the EPAP and Eagle to allow more room for the provisioned data.
4. Contact My Oracle Support for assistance.

1. Examine the syscheck output to determine if the file system /var/TKLC/epap/free/var/TKLC/elap/free is low on space. If so, continue to step 2a; otherwise skip to step 3.
2. Delete unnecessary files, as follows, to free up space on the free partition:
   1. Log in to the EPAP GUI (see Accessing the EPAP GUI)
   2. Select Debug>Manage Logs & Backups.

      A screen similar to Figure 4-14 displays. This screen displays the information about the total amount of space allocated for, and the amount of space currently used by logs and backups, and it lists logs and backup files that you might choose to delete, freeing up additional disk space.

      Figure 4-14 Manage Logs and Backups

      
   3. Click the checkbox of each file that you want to delete and then click Delete Selected File(s).
3. Contact My Oracle Support, and provide the system health check output.

1. Run syscheck in verbose mode (see procedure Run Syscheck Manually)
2. Contact My Oracle Support, and provide the system health check output.
3. Contact My Oracle Support.
4. If a 32305 3000000000000020 - Server Platform Process Error alarm is also present, execute the recovery procedure associated with that alarm before proceeding.
5. Log in to the User Interface screen of the EPAPGUI (see Accessing the EPAP GUI)
6. Check the banner information above the menu to verify that you are logged into the problem EPAP indicated in the UAM.

   If it is necessary to switch to the other side, select Select Mate.

7. Open the Process Control folder, and select the Stop Software menu item.
8. Open the Process Control folder, and select the Start Software menu item.
9. Capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI) and contact My Oracle Support.

1. Run syscheck in verbose mode.
2. Call My Oracle Support for assistance.

1. Run syscheck in verbose mode (see procedure Running the System Health Check).
2. Contact My Oracle Support, and provide the system health check output.
3. To define the default router:
   1. Obtain the proper Provisioning Network netmask and the IP address of the appropriate Default Route on the provisioning network.
      These are maintained by the customer network administrators.
   2. Log in to the server with username epapconfig (see Accessing the EPAP GUI).

      The server designation at this site is displayed, as well as hostname, hostid, Platform Version, Software Version, and the date. Ensure that the side displayed is the server that is reporting the problem. In the following example, it is server A.

   3. Enter option 2, Configure Network Interfaces Menu, from the EPAP Configuration Menu.

      
      MPS Side A:  hostname: mpsa-d1a8f8  hostid: 80d1a8f8
                   Platform Version: x.x.x-x.x.x
                   Software Version: EPAP x.x.x-x.x.x
                   Wed Jul 17 09:51:47 EST 2002
       /-------EPAP Configuration Menu--------\
      /----------------------------------------\
      |  1 | Display Configuration             |
      |----|-----------------------------------|
      |  2 | Configure Network Interfaces Menu |
      |----|-----------------------------------|
      |  3 | Set Time Zone                     |
      |----|-----------------------------------|
      |  4 | Exchange Secure Shell Keys        |
      |----|-----------------------------------|
      |  5 | Change Password                   |
      |----|-----------------------------------|
      |  6 | Platform Menu                     |
      |----|-----------------------------------|
      |  7 | Configure NTP Server              |
      |----|-----------------------------------|
      |  8 | PDB Configuration Menu            |
      |----|-----------------------------------|
      |  9 | Security                          |
      |----|-----------------------------------|
      | 10 | SNMP Configuration                |
      |----|-----------------------------------|
      | 11 | Configure Alarm Feed              |
      |----|-----------------------------------|
      | 12 | Configure Query Server            |
      |----|-----------------------------------|
      | 13 | Configure Query Server Alarm Feed |
      |----|-----------------------------------|
      | 14 | Configure SNMP Agent Community    |
      |----|-----------------------------------|
      |  e | Exit                              |
      \----------------------------------------/
      Enter Choice:  2
      

   4. Enter option 1, Configure Provisioning Network from the Configure Network Interfaces Menu.
      This displays the following submenu for configuring communications networks and other information.

      MPS Side A:  hostname: EPAP17  hostid: f80a110f
                   Platform Version: 6.0.2-7.0.3.0.0_86.45.0
                   Software Version: EPAP 161.0.28-16.1.0.0.0_161.28.0
                   Wed Jun 15 01:33:55 EDT 2016
      
      /-----Configure Network Interfaces Menu----\
      /--------------------------------------------\
      |  1 | Configure Provisioning Network        |
      |----|---------------------------------------|
      |  2 | Configure Sync Network                |
      |----|---------------------------------------|
      |  3 | Configure DSM Network                 |
      |----|---------------------------------------|
      |  4 | Configure Backup Provisioning Network |
      |----|---------------------------------------|
      |  5 | Configure Static NAT Addresses        |
      |----|---------------------------------------|
      |  6 | Configure Provisioning VIP Addresses  |
      |----|---------------------------------------|
      |  e | Exit                                  |
      \--------------------------------------------/
      
      Enter choice:  1
      

   5. Enter option 1, IPv4 Configuration (or option 2, IPv6 Configuration), from the Configure Network Interfaces Menu.

      MPS Side A:  hostname: EPAP17  hostid: f80a110f
                  Platform Version: 6.0.2-7.0.3.0.0_86.45.0
                  Software Version: EPAP 161.0.28-16.1.0.0.0_161.28.0
                  Wed Jun 15 01:33:56 EDT 2016
      
      /-----Configure Provisioning Network Menu-\
      /-------------------------------------------\
      |  1 | IPv4 Configuration                   |
      |----|--------------------------------------|
      |  2 | IPv6 Configuration                   |
      |----|--------------------------------------|
      |  e | Exit                                 |
      \-------------------------------------------/
      
      Enter Choice:  1
      

      The following warning appears:

      
      EPAP software and PDBA are running. Stop them? [N] 
      

   6. Type Y and press Enter.
      The EPAP A provisioning network IP address displays:

      
      Verifying connectivity with mate ...
      Enter the EPAP A provisioning network IP Address [192.168.61.90]:
      

   7. Press Enter after each address is displayed until the Default Route address displays:

      
      Verifying connectivity with mate ...
      Enter the EPAP A provisioning network IP Address [192.168.61.90]: 
      Enter the EPAP B provisioning network IP Address [192.168.61.91]: 
      Enter the EPAP provisioning network netmask [255.255.255.0]: 
      Enter the EPAP provisioning network default router IP Address: 192.168.61.250
      

   8. If the default router IP address is incorrect, correct it, and press Enter.
   9. After you have verifying or correcting the Provisioning Network configuration information, enter e to return to the Configure Network Interfaces Menu.
   10. Enter e again to return to the EPAP Configuration Menu.
4. Run syscheck again. If the alarm has not been cleared, go to 6
5. Run savelogs to gather all application logs, (see Saving Logs Using the EPAP GUI).
6. Contact My Oracle Support.

1. Ensure that nothing is blocking the fan's intake. Remove any blockage.
2. Verify that the temperature in the room is normal. If it is too hot, lower the temperature in the room to an acceptable level.

   Table 4-3 Server Environmental Conditions

   Ambient Temperature	Operating: 5 degrees C to 40 degrees C Exceptional Operating Limit: 0 degrees C to 50 degrees C Storage: -20 degrees C to 60 degrees C
   Relative Humidity	Operating: 5% to 85% non-condensing Storage: 5% to 950% non-condensing
   Elevation	Operating: -300m to +300m Storage: -300m to +1200m
   Heating, Ventilation, and Air Conditioning	Capacity must compensate for up to 5100 BTUs/hr for each installed frame. Calculate HVAC capacity as follows: Determine the wattage of the installed equipment. Use the formula: watts x 3.143 = BTUs/hr

   Note:

   Be prepared to wait the appropriate period of time before continuing with the next step. Conditions need to be below alarm thresholds consistently for the alarm to clear. The alarm may take up to five minutes to clear after conditions improve. It may take about ten minutes after the room returns to an acceptable temperature before syscheck shows the alarm cleared.
3. Verify that the temperature in the room is normal. If it is too hot, lower the temperature in the room to an acceptable level.

   Note:

   Be prepared to wait the appropriate period of time before continuing with the next step. Conditions need to be below alarm thresholds consistently for the alarm to clear. It may take about ten minutes after the room returns to an acceptable temperature before the alarm cleared.
4. Run syscheck Check to see if the alarm has cleared
   • If the alarm has been cleared, the problem is resolved.
   • If the alarm has not been cleared, continue with the next step.
5. Replace the filter (refer to the appropriate hardware manual).

   Note:

   Be prepared to wait the appropriate period of time before continuing with the next step. Conditions need to be below alarm thresholds consistently for the alarm to clear. It may take about ten minutes after the filter is replaced before the alarm cleared.
6. Run syscheck Check to see if the alarm has cleared
   • If the alarm has been cleared, the problem is resolved.
   • If the alarm has not been cleared, continue with the next step.
7. If the problem has not been resolved, contact My Oracle Support and provide the system health check output.
8. If the problem has not been resolved, contact My Oracle Support.

1. Run syscheck in verbose mode.
2. Run savelogs to gather system information (see Saving Logs Using the EPAP GUI)
3. Contact Customer Care Center.

   Note:

   There is a special case of heartbeat process aborting and producing core file not as a result of a bug, but as an expected and intentional response of the process to unexpected activity on the network connecting the cluster nodes. Example of such activity could be switch configuration being performed during the time cluster nodes are trying to, or already are coupled together. To recognize such a case, the investigator first needs to find out if the core file was produced by the heartbeat process:

   1. Inspect syscheck verbose output, and look for "core" module. The output would be similar to following:

           core: Checking for core files.     core: There are core files on the system:     core:     CORE DIR: /var/TKLC/core     core:         CORE: core.heartbeat.<pid>     core:         CORE: core.heartbeat.<pid>.bt *     core: FAILURE:: MINOR::5000000000000100 -- Server Core File Detected

      There, investigator finds out there is a core file named core.heartbeat.<pid>, where <pid> is the process ID of the failed heartbeat process.
   2. If heartbeat core file was found, the investigator must get the backtrace of the process from the core file by running command:

      gdb /usr/lib/hearbeat/heartbeat /var/TKLC/core/core.heartbeat.<pid>

      Once in gdb shell, entering bt. The output would be similar to the following:

      (gdb) bt #0 0x00002b872c2c0215 in raise () from /lib64/libc.so.6 #1 0x00002b872c2c1cc0 in abort () from /lib64/libc.so.6 #2 0x000000000040b20c in update_ackseq () #3 0x000000000040d225 in send_cluster_msg () #4 0x000000000040d8d7 in send_local_status () #5 0x000000000040da63 in hb_send_local_status () #6 0x00002b872b2733d7 in Gmain_timeout_dispatch (src=0x13b66bc8, func=0x40da40 , user_data=0x0) at GSource.c:1570 #7 0x00002b872b8bbdb4 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 #8 0x00002b872b8bec0d in ?? () from /lib64/libglib-2.0.so.0 #9 0x00002b872b8bef1a in g_main_loop_run () from /lib64/libglib-2.0.so.0 #10 0x000000000040e8de in initialize_heartbeat () #11 0x000000000040f235 in main ()

      The investigator is concerned in lines beginning with #0 through #5, where, in the fourth column, after the word "in", are listed function names called within the heartbeat process. If the order of called functions is the same as in the example above (i.e., raise on line #0) then abort, update_ackseq, send_cluster_msg, send_local_status, and hb_send_local_status on line #5, it is likely that the special case occurred. If such a case was recognized, the investigator can safely delete files /var/TKLC/core/core.heartbeat.<pid> and /var/TKLC/core/core.heartbeat.<pid>.bt and then clear the alarm itself by calling alarmMgr - -clear TKSPLATMI9.

   They will examine the files in /var/TKLC/core and remove them after all information has been extracted.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Run syscheck in Verbose mode (see Running the System Health Check).
2. Contact My Oracle Support.

1. Run the following command to clear the alarm.
   /usr/TKLC/plat/bin/alarmMgr –clear TKSPLATMI31
2. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Check the physical connections between the local EPAPs, and the EPAPs and the Service Module cards on the EAGLE. Make sure that the connectors are firmly seated.
2. Run syscheck (see Running the System Health Check)

   If you cannot log in, go to 3.

3. Perform Restarting the EPAP Software.
4. Capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI”)
5. Contact My Oracle Support.

1. Check the physical connections between the local EPAPs, and the EPAPs and the Service Module cards on the EAGLE.

   Make sure the connectors are firmly seated.

2. Run syscheck (see Running the System Health Check).

   If you cannot log in, go to 4.

3. Perform Restarting the EPAP Software.
4. Capture the log files on both EPAPs (see Saving Logs Using the EPAP GUI).
5. Contact My Oracle Support.

1. On the EAGLE, turn ON the optional EPAP Data Split feature to allow more room for the provisioned data.
2. On the EAGLE, turn ON the epap240m STP option (E5-SM8G-B card required) to allow more room for the provisioned data.
3. Turn ON the optional 120M DN and 120M IMSIs via Split database feature on the EPAP and Eagle to allow more room for the provisioned data.
4. Contact My Oracle Support for assistance.

1. Run syscheck.
2. Contact My Oracle Support.

   Have the system health check data available.

1. Periodically, verify that the level of the standby PDBA is catching up by selecting PDBA>View PDBA Status and comparing the Level of the Standby PDBA (on EPAP A in the example shown in Figure 4-15) to the Level of the Active PDBA (on EPAP B in the example).

   Figure 4-15 View PDBA Status

   
2. If the problem persists for more than two hours, run savelogs (see Saving Logs Using the EPAP GUI), and contact My Oracle Support for assistance.
3. Login to the User Interface screen of the EPAP GUI as any user who has permission to use the Set Log Levels menu item.
4. Select PDBA> Maintenance> Logs> Set Log Levels.
   The Set PDBA Log Info Levels screen displays, as shown in Figure 4-16.

   Figure 4-16 Set PDBA Log Info Levels

   
5. Verify that the Log Levels match the Log Levels of the MPS on the mated EAGLE STP.
   Figure 4-16 shows the usual settings. Correct log levels if necessary.
6. If adjustments are necessary, it is recommended that the Command Log debug level and the Debug Log debug level are lowered before adjusting the Error Log debug level.

1. Contact My Oracle Support.

1. To clear this alarm immediately, perform one of the following:

   • Cancel the Automatic PDB / RTDB backup via the EPAP GUI as follows:

     Note:

     Automatic PDB / RTDB Backup will have to be rescheduled if it is cancelled.

     1. Log in to the User Interface screen of the EPAP GUI (see Accessing the EPAP GUI).

     2. From the menu, select Maintenance>Automatic PDB/RTDB Backup to display the automatic backup screen.

     3. From the Automatic PDB/RTDB Backup screen, select None as the Backup Type.

     4. Select the Schedule Backup button to complete the cancellation.

        Automatic PDB/RTDB Backup will have to be rescheduled. Refer to Administration Guide to reschedule the Automatic PDB / RTDB Backup.

   • Perform a manual backup via the EPAPGUI (see Backing Up the PDB).

1. Log in to the User Interface screen of the EPAP GUI (see Accessing the EPAP GUI).
2. From the menu, select Maintenance>Automatic PDB/RTDB Backup to display the Automatic PDB/RTDB Backup screen.
3. From the Automatic PDB/RTDB Backup screen, select None as the Backup Type.
4. Select the Schedule Backup button to complete the cancellation.

   Note:

   Automatic PDB/RTDB Backup will have to be rescheduled. Refer to Administration Guide to reschedule the Automatic PDB/RTDB Backup.

1. To clear this alarm immediately, perform one of the following:

   • Cancel the Automatic PDB/RTDB backup in the EPAP GUI.

     Note:

     Automatic PDB/RTDB Backup will have to be rescheduled if it is cancelled.

     1. Log in to the User Interface screen of the EPAP GUI (see Accessing the EPAP GUI).

     2. From the menu, select Maintenance>Automatic PDB/RTDB Backup to display the Automatic PDB/RTDB Backup screen.

     3. From the Automatic PDB/RTDB Backup screen, select None as the Backup Type.

     4. Select the Schedule Backup button to complete the cancellation. Automatic PDB/RTDB Backup will have to be rescheduled. Refer to Administration Guide to reschedule the Automatic PDB/RTDB Backup.

   • Perform a manual backup via the EPAP GUI as described in Backing Up the RTDB.

1. Log in to the User Interface screen of the EPAP GUI (see Accessing the EPAP GUI).
2. From the menu, select Maintenance>Automatic PDB/RTDB Backup to display the Automatic PDB/RTDB Backup screen.
3. From the Automatic PDB/RTDB Backup screen, select None as the Backup Type.
4. Select the Schedule Backup button to complete the cancellation. Automatic PDB/RTDB Backup will have to be rescheduled. Refer to Administration Guide to reschedule the Automatic PDB/RTDB Backup.

• Verify that the Customer Provisioning Application (CPA) machine is up and running.
  • If the CPA machine is not running, restart it and wait for the alarm to clear.
  • If the CPA machine is running, or if the alarm does not clear, contact My Oracle Support.
• If the alarm text is "SSH tunnel down for <IP>", verify that the port specified for SSH tunneling is not in use on the remote machine.

1. On the EAGLE, turn ON the optional EPAP Data Split feature to allow more room for the provisioned data.
2. On the EAGLE, turn ON the epap240m STP option (E5-SM8G-B card required) to allow more room for the provisioned data.
3. Turn ON the optional 120M DN and 120M IMSIs via Split database feature on the EPAP and Eagle to allow more room for the provisioned data.
4. Contact My Oracle Support for assistance.

1. Log in to the EPAP CLI as epapdev.
2. Use the manageLicenseInfo utility to check the value of purchased capacity (see the "Current license capacity"):

   $ manageLicenseInfo –l

3. Purchase additional provisioning database capacity licenses as needed.
4. Use the manageLicenseInfo utility to specify the additional amount of desired PDB capacity, where "License Capacity value" is the number of additional licenses required when one license supports 0.5M data (DN, IMSI, and IMEI):

   $ manageLicenseInfo –a <License Capacity value>

1. Log in to the EPAP CLI as epapdev.
2. Use the manageLicenseInfo utility to check the value of purchased capacity (see the "Current license capacity"):

   $ manageLicenseInfo –l

3. Purchase additional provisioning database capacity licenses as needed.
4. Use the manageLicenseInfo utility to specify the additional amount of desired PDB capacity, where "License Capacity value" is the number of additional licenses required when one license supports 0.5M data (DN, IMSI, and IMEI):

   $ manageLicenseInfo –a <License Capacity value>

1. Purchase additional provisioning database capacity licenses.
2. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Contact My Oracle Support.

1. Log in to the EPAP CLI as epapdev.
2. Use the manageLicenseInfo utility to check the value of purchased capacity (see the "Current license capacity"):

   $ manageLicenseInfo –l

3. Use the manageLicenseInfo utility to specify the additional amount of desired PDB capacity, where <License Capacity value> is the number of additional licenses required when one license supports 0.5M data (DN, IMSI, and IMEI):

   $ manageLicenseInfo –a <License Capacity value>

   For example, if 120M is currently provisioned and an additional 80M is desired for a new capacity of 200M, 160 should be specified for the <License Capacity value>:

   manageLicenseInfo -a 160

Customer should schedule a cron job in “/etc/cron.d/TS.EXAP” for “/usr/TKLC/appl/bin/checkNEsanity.pl” in order to raise the alarm, when there is count mismatch between PDB and RTDB for Network Entity (NE). This cron will be scheduled only on the server having RTDB, hence don’t schedule the cron on pdbonly server.

For example: To schedule daily once at 05:00 , Sched="daily,1,05:00"

00 05 * * * epapdev /usr/TKLC/appl/bin/checkNEsanity.pl

Contact support for any query.