Messages, Alarms, and Status Processing

4 Messages, Alarms, and Status Processing

This chapter provides a description of EPAP messages, alarms, and status processing.

4.1 EPAP Messages

This section includes Table 4-1. For alarm-related banner messages that appear on the UI browser screen in the Message Box described in EPAP GUI Main Screen. Refer to Alarms and Maintenance Guide for alarm recovery procedures.

EPAP Error Messages

Table 4-1 lists the error codes and associated text that are generated by the EPAP user interface. The <> fields indicate values that are different for each error; the fields are filled at run time.

Table 4-1 EPAP Error Messages

E0047	Cmd Rej: RTDB returned error code RTDB_RET_DB_ENTRY_NOT_FOUND
E1000	Unknown error <error number>. No error text is available.
E1001	Invalid menu selection: <menu selection>
E1002	Invalid syntax: <input>
E1003	Mate EPAPs may not have the same designation.
E1004	EPAP software is running. You must stop the EPAP software before performing this operation.
E1005	EPAP software is not running. You must start the EPAP software before performing this operation.
E1006	Mate EPAP not available
E1007	Could not eject media: <device>
E1008	Could not read file: <file name>
E1009	Active PDBA is not available.
E1010	This host is not an EPAP.
E1011	Cannot find EPAP <A\|B> (host name <host name>)
E1012	Subscription (SP= <subscription number>) does not exist.
E1013	Subscription (SP= <subscription number>) already exists.
E1014	SP <identifier> does not exist.
E1015	IMSI <identifier> does not exist.
E1016	DN <identifier> does not exist.
E1017	PDBI error: <error text>
E1018	DN <identifier> already associated with IMSI <identifier>.
E1019	Device <device> unavailable.
E1020	Remote PDB unavailable.
E1021	IP address <address> is not authorized for PDB access.
E1022	RN <identifier> does not exist.
E1023	Invalid value for <prompt >: <value>: Valid values are <range>. Hit the Escape key to abort the command.
E1024	MTSU error: <error text>
E1025	File lock failed: <file name>
E1026	Environment variable <variable name> not defined.
E1027	ssh error: <error text>
E1028	IP address <IP address> is already authorized for PDBI access.
E1029	IP address <IP address> is not authorized for PDBI access.
E1030	Operation timed out waiting for a response from the PDBA.
E1031	Operation timed out waiting for a response from the RTDB.
E1032	Operation aborted by user.
E1033	Unexpected response received from the PDBA: id=<txid>, return code=<return code>
E1034	Unexpected response received from the RTDB: id=<txid>, return code=<return code>
E1035	Script <script name> failed: status=<status>
E1036	EPAP configuration failed. Please use the configuration menu to manually configure the EPAP sync and DSM networks.
E1037	One or more EPAP software processes did not start
E1038	One or more EPAP software processes did not stop
E1039	Transaction log query failed: <error text>
E1040	Transaction log response file was not created.
E1041	Transaction log response file could not be parsed.
E1042	Transaction log export failed: <error text>
E1043	The specified EPAP was not available.
E1044	Remote EPAP software is running. You must stop the remote EPAP software before performing this operation.
E1045	RTDB copy operation failed.
E1046	Improperly encoded alarm string. Re-check source.
E1047	RTDB did not respond to query.
E1048	Invalid response received from RTDB.
E1049	Could not connect to <device or process>: <error text>
E1050	Secure shell daemon is not running on mate EPAP. Config files could not be synchronized !!!
E1051	No feature(s) specified.
E1052	Both ELAP and EPAP features specified.
E1053	This action may only be performed on the local EPAP.
E1054	Another user is currently performing this same action.
E1055	Missing mandatory parameter: <parameter>
E1056	Unexpected parameter was provided:<parameter>
E1057	The EPAP must be in Forced Standby mode for this operation.
E1058	An internal error in the <parameter> occurred: <error text>
E1059	The passwords did not match.
E1060	The provisioning addresses for MPS A and B must be different.
E1061	The provisioning addresses for MPS A and B must be on the same network.
E1062	The default router must be on the same network as MPS A and MPS B.
E1063	The local and remote PDB addresses must be different.
E1064	This action may only be performed on EPAP A.
E1065	<device or process> must be configured.
E1066	The requested user <user> was not found.
E1067	The requested group <group> was not found.
E1068	The password entered was not correct.
E1069	The new password has been used too recently.
E1070	The provided password does not meet the security requirements. Reason: <reason text>
E1071	The specified group already exists.
E1072	This action may only be performed on EPAP B.
E1073	The file you have attempted to upload is larger than the <number> bytes of allocated storage space.
E1074	LPU batch failure: <error text>
E1075	This action must be done on the Active PDBA.
E1077	File system violation: <error text>
E1078	File '<file name>' was empty.
E1079	There are no PDBI connections available. Try again later.
E1080	The provisioning addresses for the main and backup networks must be different.
E1081	The specified IP already exists.
E1082	The specified IP does not exist.
E1083	The maximum number of authorized UI IPs has been reached.
E1084	This action may be performed only on a provisionable MPS.
E1085	The specified address is local to this MPS.
E1088	Attempt to access the PDBA was not successful.
E1092	The range would overlap the existing range 232323 to 232324 (applicable for Both IMSI range and Blocklist Range).
E1093	The <Provisioning Blocklist or IMSI range> does not exist.
E1097	The maximum capacity for this provisioning has been reached.
E1098	Unable to establish SSH tunnel to <machine Identifier> machine.
E1099	Invalid IP Address/Username/Password combination
E1100	Task Scheduler error: <error text>
E1101	IP address <IP Address> is not authorized to perform this action: <action text>
E1102	This backup is incompatible: <backup identifier> Reason: <reason text>
E1103	This action cannot be performed as <action text> is in progress.
E1104	Both the entered IPs are same.
E1105	Invalid EMS Server name: <Value>. EMS Server name can contain only alphanumeric characters, hyphen and underscore! EMS Server name can contain a minimum of 5 and a maximum of 20 characters.
E1106	Invalid Community String: <Value>. Only alphanumeric characters, hyphen and underscore are allowed. Community string length cannot exceed 127 characters.
E1107	Invalid value for Alarm feed variable: <Value>. Valid values are 'ON' or 'OFF'.
E1108	Maximum 5 EMS can be added.
E1109	The default router must be on the same network as MPS A.
E1110	The addresses for provisioning and GUI access must be different.
E1111	The addresses for provisioning and operations and maintenance must be different.
E1112	The addresses for GUI access and operations and maintenance must be different.
E1113	Invalid IP Address.
E1114	The specified IP already used for Local Network Configuration.
E1115	Remote PDB cannot be configured with %s as Local and Remote PDBA should be in the same IP format.

4.2 MPS and EPAP Status and Alarm Reporting

The System Health Check (syscheck) utility runs automatically at least every five minutes, and can be run manually to test for error conditions in each MPS Server and in each EPAP. See Run Health Check and refer to Alarms and Maintenance Guide for more information about executing and viewing results from the System Health Check.

Alarms of minor, major, and critical levels of severity are reported for error conditions detected for the MPS hardware platform and for the EPAP application.

On the MPS front panel, three LEDs correspond directly to alarm severities: critical, major, and minor. If more than one alarm level is active, all applicable LED lights are illuminated, not only the most severe alarm, until all alarms in that level are cleared.

4.2.1 Raising Alarms

This section details the steps to raise alarms for Long Wait on Write for PDBI Update and for NE count mismatch between PDB and RTDB.

4.2.1.1 Raising Alarm for Long Wait on Write for PDBI Update

Customers should complete the following steps in order to raise the " Long Wait on Write for PDBI Update" alarm. This will ensure the user is alerted to a PDBI write connection holding for too long:

Issue the uiEdit command "PDBI_LONG_WAIT_ALARM_TIME" <time in seconds> where <time in seconds> is the time value that a PDBI connection is allowed to hold a write connection before triggering the alarm.
Investigate the alarm banner on the EPAP GUI for the alarm text "Long wait on write for PDBI update"; or, identify the alarm bit 6000000000800000 from the connected EAGLE; or, find the alarm number 45121 from the SNMP NM server.
If the alarm is triggered, find the PDBI connection information by issuing the grep "Throw alarm for connection" pdba.err.* command in the /usr/TKLC/epap/logs directory.
Clear the alarm to release the PDBI write connection in question.

4.2.1.2 Raising Alarm for NE count mismatch between PDB and RTDB

Customer should schedule a cron job in “/etc/cron.d/TS.EXAP” for “/usr/TKLC/appl/bin/checkNEsanity.pl” in order to raise the alarm, when there is count mismatch between PDB and RTDB for Network Entity (NE).This will ensure that the user is alerted when there is any mismatch in NE count between PDB and RTDB. This cron will be scheduled only on the server having RTDB, hence don’t schedule the cron on pdbonly server.

Customer can schedule the cron considering the following conditions:

Cron should be scheduled once in a day.
Server should not be involved in any other activity at the time when this cron is scheduled, to neglect any impact.
It will be good to schedule the cron when the provisioning rate is very low or negligible.
Cron should be scheduled at some specific time of the day when scheduling for once in a day.

For example: To schedule daily once at 05:00 , Sched="daily,1,05:00"

00 05 * * * epapdev /usr/TKLC/appl/bin/checkNEsanity.pl

Logs will be formed at /usr/TKLC/epap/logs directory.

On compact architecture, RTDB server:

rtdbEpap17NEPrintCompact.log

rtdbEpap17NEPrintInorderCompact.log

checkNEsanity.log

On extreme architecture RTDB server:

rtdbEpap17NEPrintExtreme.log

rtdbEpap17NEPrintInorderExtreme.log

checkNEsanity.log

On PDB server (compact/extreme):

retrieveNEfromPDB.log (to retrieve NE count from PDB)

If the alarm is triggered, follow the below mentioned recovery steps:

When the alarm is observed, savelogs (application logs) is automatically taken for the first time but the customer will have to take the platform logs manually from the platcfg menu. Also, customer can retake the application logs, if more recent logs are needed.
If we have other RTDBs connected to the same PDB and the DB on them is good, then the customer can restore the backup from one of the other connected RTDBs on this RTDB.
If all RTDBs connected to the same PDB are show NE mismatch then restore both PDB and RTDB from backup taken from another site having the similar database.
If the above two options are not feasible, then do a reload from PDB on this RTDB. This will reload the database from scratch on RTDB, from the connected PDB. This process will take time depending upon the total size of the database.

Alarms will be cleared once the mismatch is resolved.

4.2.2 Maintenance Blocks

MPS and EPAP have no direct means of accepting user input from or displaying output messages on EAGLE terminals. Maintenance, measurements, error, and status information are routed to EAGLE through the primary Service Module card.

The Active EPAP generates and sends Maintenance Blocks to the primary Service Module card. One Maintenance Block is sent as soon as the IP link is established between the Active EPAP and the primary Service Module card. Additional Maintenance Blocks are sent whenever the EPAP needs to report any change in status or error conditions. The information returned in Maintenance Blocks is also included in the output of the rept-stat-mps command.

It is possible for the EPAP to be at a provisioning congestion threshold, and to be entering and exiting congested mode at a very high rate of speed. To minimize this “thrashing” effect, the EPAP is restricted to sending no more than one EPAP Maintenance Block per second.

EPAP Maintenance Block Contents

The EPAP sends Maintenance Blocks that contain (at a minimum) the following information. The actual states are defined in the description of the rept-stat-mps command in Commands Manual.

MPS major, minor, and dot software versions
MPS Status (down/up)
MPS Status (Active/Standby)

If the EPAP needs to report one or more alarm conditions, it inserts the appropriate alarm data string for the indicated alarm category into the Maintenance Block.

EAGLE Alarm Reporting

The System Health Check (syscheck) is responsible for forwarding platform errors to the application. The application combines the platform alarms with the application alarms and forwards all of this information to the EAGLE. The information that is transferred is described in Alarms and Maintenance Guide.

4.2.3 Alarm Priorities

The EPAP sends the maintenance information, including the alarm data strings, to the EAGLE for interpretation. Alarm priorities determine which alarm category is displayed at the EAGLE terminal when multiple alarm levels exist simultaneously. EAGLE prioritizes the data and displays only the alarm category with the highest severity level and priority for each MPS.

If an alarm category of lower priority is sent from the MPS, the lower priority alarm category is not displayed on the EAGLE terminal until any higher priority alarms are cleared.

4.2.4 Multiple Alarm Conditions

Critical, major and minor alarms a ppear repeatedly in each alarm delivery to the EAGLE until the alarm condition clears.

If multiple alarms exist, the highest priority alarm category is the Active Alarm. The Active Alarm is shown in the output from the rept-stat-trbl command and the rept-stat-mps command, and the alarm count associated with this alarm is included in the rept-stat-alm command output.

Though only the highest priority alarm is displayed at the EAGLE terminal when multiple alarms are reported, you can use the EAGLE rept-stat-mps command to list the alarm data strings for all of the alarm categories with existing alarms. Then you can use the EPAP user interface maintenance menu item Decode EAGLE Output of MPS Alarms to convert the hexadecimal alarm data string to text. The output text shows the alarm category represented by the string and the alarm text for each alarm encoded in the string.

4.2.5 Service Module Card Status Requests

When the EPAP needs to know the status of a Service Module card, it can send a Service Module Status Request to that Service Module card. Because status messages are sent over UDP, the EPAP broadcasts the Service Module Status Request and all Service Module cards return their status.

Service Module Card Status Reporting to the EPAP

The EPAP needs to know the current status of various aspects of the Service Module cards. Accordingly, the Service Module card sends a Service Module status message to the EPAP when the following events occur:

When the Service Module card is booted
When the Service Module card receives a Service Module Status Request message from the EPAP
When the Service Module card determines that it needs to download the entire database

For example, the database could become totally corrupted, or a user could initialize the card.
When the Service Module card starts receiving DB downloads or DB updates.

When a Service Module card starts downloading the RTDB, or if the Service Module card starts accepting database updates, it needs to send a status message informing the EPAP of the first record received. This helps the EPAP keep track of downloads in progress.

Service Module Card Status Message Fields

The Service Module card status message provides the following information to the EPAP:

Service Module card Memory Size

When the Service Module card is initialized, it determines the amount of applique memory present. The EPAP uses this value to determine if the Service Module card has enough memory to hold the RTDB.
Load Mode Status

This flag indicates whether or not 80% of the IS-NR LIMs have access to SCCP services.
Database Level Number

The EPAP maintains a level number for the RTDB. Each time the database is updated, the level number will be incremented. When the database is sent to the Service Module card, the Service Module card keeps track of the database level number. The database level number will be included in all Status messages sent from the Service Module card. A level number of 0 signifies that no database has been loaded into the Service Module card (this can be done any time the Service Module card wants to request a full database download).
Database Download Starting Record Number

When the Service Module card starts downloading either the entire RTDB or updates to the database, it will identify the starting record number. This allows the EPAP to know when to wrap around the end of the file, and when the Service Module card has finished receiving the file or updates.

4.3 System Hardware Verification

Service Module card loading verifies the validity of the hardware configuration for the Service Module cards. The verification of the hardware includes:

Validity of the Service Module card motherboard
Verification of installed memory size

4.3.1 Service Module Card Motherboard Verification

An SM8G-B card is required to support the G-Flex/ G-Port/INP/EIR VSCCP application on the Service Module card. EAGLE maintenance stores the validity status of the Service Module card motherboard configuration. The system does not allow the G-Flex, G-Port, INP/AINPQ, EIR, A-Port, V-Flex, and Migration feature to be enabled if the hardware configuration is invalid.

When the VSCCP application is initializing, it determines the motherboard type, as well as verifies the amount of memory in the SM card. The SCCP Maintenance Block is the mechanism that relays the motherboard information to OAM. This requires the application software to be loaded to the Service Module card and then verification of the motherboard information received in the SCCP Maintenance Block. If the motherboard is determined to be invalid for the G-Flex/G-Port/INP/AINPQ/EIR/A-Port/V-Flex/Migration application, loading of the Service Module card is automatically inhibited and the card is booted via PMTC. Booting the card in this manner suppresses any obituary.

4.3.2 Service Module Card Installed Memory Verification

The VSCCP application performs two types of memory validation to determine whether or not a Service Module card has sufficient memory to run G-Flex/G-Port/INP/AINPQ/EIR/A-Port/Migration/V-Flex: Local Memory validation and Continual Memory validation.

The report from the rept-stat-sccp command includes the installed memory both allocated and physically present on each Service Module card. See Commands User's Guide for a description of the rept-stat-sccp command output.

Local Memory Validation

When the G-Flex or INP feature bit is first enabled (a Feature Access Key is used for the EIR, AINPQ, G-Port, Migration, and V-Flex features), or any time the G-Flex, G-Port, INP/AINPQ, EIR, A-Port, V-Flex or Migration feature is enabled and the Service Module card is initializing, VSCCP checks to see if the Service Module card has at least one D1G installed memory. The G-Flex, G-Port or INP feature bit cannot be enabled if any of the Service Module cards have less than 1 GB of memory installed.

Real-Time Memory Validation

When communication between the Service Module card and EPAP is established and the Service Module card joins the RMTP Tree, the EPAP starts downloading the RTDB to the Service Module card. After the Service Module card has downloaded the RTDB, it continues to receive database updates as necessary. The EPAP includes the size of the current RTDB in all records sent to the Service Module card. The Service Module card compares the size required to the amount of memory installed, and issues a minor alarm whenever the database exceeds 80% of the Service Module card memory. If the database completely fills the Service Module card memory, a major alarm is issued and the Service Module card status changes to IS-ANR/Restricted.

4.3.3 Actions Taken When Hardware Determined to be Invalid

When the hardware configuration for a Service Module card is determined to be invalid for the G-Flex, G-Port, INP/AINPQ, EIR, A-Port, V-Flex, and Migrationapplication, SCM automatically inhibits loading for that specific Service Module card. A major alarm is generated indicating that card loading for that Service Module card failed and was automatically inhibited (that is, prevented from reloading again). Refer to Maintenance Guidefor the specific alarm that is generated. When card loading is inhibited, the primary state of the card is set to OOS-MT-DSBLD and the secondary state of the card is set to MEA (Mismatch of Equipment and Attributes).

The following actions apply to a Service Module card determined to be invalid:

The Service Module card will not download the EAGLE databases.
The Service Module card will not download the RTDB from the EPAP.
The Service Module card will not accept RTDB updates (additions, changes, and deletes) from the EPAP.

The rept-stat-sccp command supports the Service Module cards running the VSCCP application and reports G-Flex, G-Port, INP/AINPQ, EIR, A-Port, V-Flex, and Migration statistics. See “Commands” for more details on the rept-stat-sccp command.

4.4 Unstable Loading Mode

At some point, having a number of invalid Service Module cards results in some of the LIMs being denied SCCP services. There is a threshold that needs to be monitored: if the number of valid Service Module cards is insufficient to provide service to at least 80% of the IS-NR LIMs, the system is said to be in an unstable Loading Mode.

The system interrupts and aborts card loading upon execution of an STP database change command. Loading Mode support denies the execution of STP database change commands when the system is in an unstable loading mode.

An unstable loading mode exists when any of the following conditions are true:

The system’s maintenance baseline has not been established.
Less than 80% of the number of LIMs provisioned are IS-NR or OOS-MT-DSBLD.

The conditions that an insufficient number of Service Module cards are IS-NR or OOS-MT-DSBLD relative to 80% of the number of provisioned LIMs is called a failure to provide adequate SCCP capacity.
The number of IS-NR and OOS-MT-DSBLD Service Module cards is insufficient to service at least 80% of all provisioned LIMs.

Loading Mode is based on the ability of the system to provide SCCP service to at least 80% of the LIMs. Refer to Maintenance Guide for EAGLE for additional information about LIM and Service Module cards.
There is insufficient SCCP service, which occurs if an insufficient number of IS-NR Service Module cards are available to service at least 80% of the number of IS-NR LIMs.

It is possible for LIMs or Service Module cards to be inhibited or to have problems that prevent them from operating normally. If enough Service Module cards are out of service, it may not be possible for the remaining IS-NR Service Module cards to service at least 80% of the number of IS-NR LIMs. This is called “insufficient SCCP service.” When this occurs, some of the LIMs are denied SCCP service. It is possible to use the inh-card command to inhibit LIMs to improve the ratio (see Actions Taken When the System is in an Unstable Loading Mode).
If LIM cards are being denied SCCP service and any Service Module cards are in an abnormal state (OOS-MT, IS-ANR)

Actions Taken When the System is in an Unstable Loading Mode

Unstable loading mode has no impact on RTDB downloads or the stream of RTDB updates.
When the loading mode is unstable, the rept-stat-sys command reports the existence of the unstable loading mode and the specific trigger that caused it.
When in an unstable Loading Mode, the EAGLE 5 does not accept database updates. When updates are rejected, the reason is given as:
```
E3112 Cmd Rej: Loading Mode unstable due to SCCP service is deficient.
```
The inh-card and alw-card commands can be used to alter SCCP service levels to achieve the 80% threshold. This can be repeated for each card until the system is able to supply SCCP services to at least 80% of the IS-NR LIMs. The remaining 20% LIM or supporting Service Module cards may remain out of service until the stream of database updates ceases. This stream of updates can be temporarily interrupted to allow the remaining 20% of the system to come in service.

After an EAGLE database has been loaded, that database can be updated, if the system is not in an unstable Loading Mode. However, if an EAGLE update comes in during EAGLE database loading, the Service Module card aborts the current loading, issues a class 01D7 obit message (Figure 4-1), and reboots.

Figure 4-1 Obit Message for Abort of Card Loading
If executing the ent-card or inh-card command would cause the system to enter an unstable Loading Mode, it is necessary to use the Force parameter on the command.

4.5 System Status Reporting

The following status reporting is described in this section:

System status
G-Flex status
G-Port status
INP/AINPQ status
EIR status
A-Port status
Migration status
V-Flex status
Service Module card memory capacity status
Loading mode support status

System Status Reporting

The rept-stat-sccp command supports the Service Module cards running the VSCCP application, and reports G-Flex, G-Port, INP/AINPQ, EIR, A-Port, V-Flex, and Migration statistics. See rept-stat-sccp for details on the rept-stat-sccp command.

G-Flex/G-Port/INP/AINPQ/EIR/A-Port/V-Flex/Migration Status Reporting

The rept-stat-mps command reports the status of the G-Flex/G-Port/INP/AINPQ/EIR/A-Port/V-Flex/Migration provisioning system. The rept-stat-sccp command reports separately the statistics for G-Flex, G-Port, INP/AINPQ, EIR, A-Port, V-Flex, and Migration. See Commands for details on the rept-stat-mps and rept-stat-sccp commands.

Service Module Card Memory Capacity Status Reporting

The Service Module card sends a message to the EPAP containing the amount of memory on the Service Module card. The EPAP determines whether the Service Module card has enough memory to store the RTDB and send an ACK or NAK back to the Service Module card indicating whether or not the Service Module card has an adequate amount of memory.

When the EPAP sends database updates to the Service Module cards, the update messages includes a field that contains the new database memory requirements. Each Service Module card monitors the DB size requirements, and issue a minor alarm if the size of the DB exceeds 80% of its memory. If a database increases to the point that it occupies 100% of the Service Module card memory, a major alarm is issued.

The rept-stat-mps:loc=xxxx command shows the amount of memory used by the RTDB as a percent of available Service Module card memory (see rept-stat-mps ).

Loading Mode Support Status Reporting

The OAM application can determine whether or not the system is in an unstable Loading Mode because it knows the state of all LIM, SCCP, and Service Module cards in the system. When the loading mode is unstable, the rept-stat-syscommand reports the existence of the unstable Loading Mode and the specific conditions which caused it. See Unstable Loading Mode for more details on Loading Mode support.

4.6 Commands

The commands described in this section report status information for the provisioning system.

rept-stat-sccp

The command handling and scroll area output for the rept-stat-sccp command includes the Service Module card. You can add the loc parameter to display detailed card traffic statistics.

Samples of the reports produced by these commands are shown in Figure 4-2:

Figure 4-2 rept-stat-sccp Command Report Examples

rept-stat-db

The rept-stat-db command report includes the RTDB birthdate, level, and status. This information is used to help determine the need for and method to use for an RTDB resynchronization, audit and reconcile, reload from another RTDB, or reload from the PDB.

Figure 4-3 rept-stat-db Command Report Example

rept-stat-mps

The rept-stat-mps command reports the status of the provisioning system, including EPAP information.

There are two possible variants of this new command:

rept-stat-mps - This produces a summary report showing the overall status of the G-Flex/G-Port/INP/EIR provisioning system and a moderate level of information for each Service Module card.
rept-stat-mps:loc=xxxx – This produces a more detailed report showing the G-Flex/G-Port/INP/EIR status of a specific Service Module card.

When the EPAP sends database updates to the Service Module cards, the update messages include a field that contains the new database memory requirements. This version of the rept-stat-mps command displays the amount of memory used by the RTDB as a percent of available Service Module card memory.

Each Service Module card monitors the DB size requirements, and issue a minor alarm if the size of the DB exceeds 80% of its memory. If a database increases to the point that it occupies 100% of the Service Module card memory, a major alarm is issued.

Samples of the reports produced by these commands are shown in Figure 4-4:

Figure 4-4 rept-stat-mps Command Report Examples

rept-stat-trbl

This command includes the G-Flex/G-Port/A-Port /Migration Subsystem, INP/AINQP Subsystem, EIR Subsystem, V-Flex Subsystem, and Service Module card/EPAP IP link alarms.

Figure 4-5 rept-stat-trbl Command Output Example

rept-stat-alm

This command includes the alarm totals for the G-Flex/G-Port Subsystem, INP Subsystem, EIR Subsystem, and Service Module card/EPAP IP links.

Figure 4-6 rept-stat-alm Command Report Example

pass: cmd=”Ping”

The ‘ping’ command allows for troubleshooting of the private EPAP-Service Module card IP network.

Figure 4-7 pass: cmd=”Ping” Command Output Example

pass: cmd=”netstat”

The ‘pass: cmd=”netstat” command allows troubleshooting of network interface and routing configuration problems within the private EPAP-Service Module card IP network.

Figure 4-8 pass: cmd=”netstat” Command Output Example

4.7 Hourly Maintenance Report

The hourly maintenance report includes the alarm totals for the G-Flex/G-Port/A-Port/Migration Subsystem, INP/AINQP Subsystem, V-Flex Subsystem, and DSM/EPAP IP links.

Figure 4-9 Hourly Maintenance Report Output Example

4.8 Unsolicited Alarm and Information Messages

This section describes EPAP Unsolicited Alarm Messages (UAMs) and Unsolicited Information Messages (UIMs).

The EAGLE outputs two types of unsolicited messages:

Unsolicited Alarm Messages (UAMs): Denotes persistent problems with a device or object that needs the attention of a craftsperson

Unsolicited Informational Messages (UIMs): Indicates transient events that have occurred

Unsolicited Alarm Messages are generated by the maintenance system as trouble notification for the OS. The maintenance system is able to determine the status of the system through polling and periodic audits. Troubles are detected through analysis of system status and notifications from various subsystems in the EAGLE. The EAGLE controls and generates the alarm number, associated text, and formatting for alarms sent to EAGLE through the Maintenance Block mechanism.

Alarms and Maintenance Guide describes all EAGLE UAMs and the appropriate recovery actions.

MPS Platform and EPAP Application Alarms

MPS platform errors are detected by the system health check utility. The system health check output contains a 16-digit hexadecimal alarm data string for each detected platform or application error. The 16-character hexadecimal alarm data string reports any errors found during the last System Health Check and the level of severity for each error. The first character (four bits) uniquely identifies the alarm severity for the alarm data. The remaining 15 characters (60 bits) uniquely identify up to 60 individual failure cases for the alarm category. The system health check utility, the alarm data strings, and the corrective procedures are described in detail in Alarms and Maintenance Guide.

MPS platform and EPAP application alarms are reported in five categories of alarms. The categories are:

Critical Platform Alarm: This is a 16-character hexadecimal string in which each bit represents a unique critical platform failure and alarm. An alarm in this category results in the associated MPS state being set to OOS-MT// Fault.

Major Platform Alarm: This is a 16-character hexadecimal string in which each bit represents a unique major platform failure and alarm. An alarm in this category results in the associated MPS state being set to OOS-MT// Fault.

Minor Platform Alarm: This is a 16-character hexadecimal string in which each bit represents a unique minor platform failure and alarm. An alarm in this category results in the associated MPS state being set to IS-ANR// Restricted.

Major Application Alarm: This is a 16-character hexadecimal string in which each bit represents a unique major application failure/alarm. An alarm in this category results in the associated MPS state being set to OOS-MT// Fault.

Minor Application Alarm: This is a 16-character hexadecimal string in which each bit represents a unique minor application failure and alarm. An alarm in this category results in the associated MPS state being set to IS-ANR/Restricted.

Table 4-2 defines the application and platform alarms that are forwarded to EAGLE when MPS and EPAP failures or errors are detected. Each alarm category is sent with a hexadecimal alarm data string that recovered from the MPS/EPAP (see MPS and EPAP Status and Alarm Reporting ). The clearing alarm for all of the MPS Platform and Application alarms is UAM 0250, MPS Available.

Note:

The recovery actions for the platform and application alarms are defined in Alarms and Maintenance Guide.

Table 4-2 EAGLE MPS Platform and Application Alarms

UAM #	Severity	Message Text
370	Critical	Critical Platform Failure(s)
372	Major	Major Platform Failure(s)
373	Major	Major Application Failure(s)
374	Minor	Minor Platform Failure(s)
375	Minor	Minor Application Failure(s)
250	Clearing	MPS Available

Figure 4-10 Alarm Output Example

Figure 4-11 MPS Available Alarm

The clearing alarm is generated after existing alarms have been cleared. The clearing alarm sets the MPS primary status to IS-NR.

4.8.1 EPAP-to-Service Module Card Connection Status

The EPAP and the Service Module cards are connected over two Ethernet networks and use TCP/IP. If connection is inoperative, the Service Module card is responsible for generating an appropriate UAM. Loss of connectivity or inability of the EPAP to communicate (from hardware or software failure, for example) will be detected and reported within 30 seconds.

EPAP-Service Module Card UAMs

Maintenance Blocks EPAP have a field to identify error message requests. (See Maintenance Blocks ). The Service Module card processes incoming Maintenance Blocks and generates the requested UAM. The Service Module card acts only as a delivery agent. The recovery actions for the EPAP-Service Module card UAMs are defined in Unsolicited Alarm and Information Messages.

Service Module Card-EPAP Link Status Alarms

Two alarms indicate the Service Module card-to-MPS link status:

0084 “IP Connection Unavailable” (Major)
0085 “IP Connection Available” (Normal/Clearing)

Figure 4-12 Service Module Card-EPAP Link Alarm Example

RTDB Audit Alarms

During an audit of the Service Module cards and the EPAPs, the status of each real-time database (RTDB) is examined and the following alarms can be raised. The recovery actions for the RTDB Audit Alarms are defined in Unsolicited Alarm and Information Messages.

When an RTDB has become corrupted, the following minor alarm is raised.


         1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 00-04-30 16:28:08 EST EAGLE 35.0.0
*   0012.0443 * CARD 1108 VSCCP        RTDB Database is corrupted

When a card’s RTDB is inconsistent (its contents are not identical to the current RTDB on the Active EPAP fixed disks), the following minor alarm is raised.


1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 00-04-30 16:28:08 EST EAGLE 35.0.0
*   0012.0444 * CARD 1108 VSCCP        RTDB Database is inconsistent

When an inconsistent, incoherent, or corrupted RTDB has been fixed and the card or EPAP is in an IS-NR condition, the following alarm is raised.


1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 00-04-30 16:28:08 EST EAGLE 35.0.0
    0012.0445 CARD 1108 VSCCP          RTDB Database has been corrected

While the RTDB is being downloaded or an update has failed, it is in an incoherent state. The following minor alarm is raised.


1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 99-09-30 16:28:08 EST EAGLE 35.0.0
*   0012.0448 * CARD 1108 VSCCP        RTDB Database is incoherent

When a Service Module card detects that its RTDB needs to be resynchronized and has started the resync operation, the following major alarm is raised.


         1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 99-09-30 16:28:08 EST EAGLE 35.0.0
**  0012.0449** CARD 1108 VSCCP        RTDB resynchronization in progress

After a Service Module card completes its RTDB resync operation, the following clearing alarm is raised.


         1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 99-09-30 16:28:08 EST EAGLE 35.0.0
    0012.0450   CARD 1108 VSCCP        RTDB resynchronization complete

When a Service Module card detects that its RTDB needs to be reloaded because the resync log does not contain all of the required updates, the following major alarm is raised.


         1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 99-09-30 16:28:08 EST EAGLE 35.0.0
**  0012.0451** CARD 1108 VSCCP        RTDB reload required

After a Service Module card completes its RTDB reload operation, the following clearing alarm is raised.


         1         2         3         4         5         6         7         8
12345678901234567890123456789012345678901234567890123456789012345678901234567890
    station1234 99-09-30 16:28:08 EST EAGLE 35.0.0
    0012.0452   CARD 1108 VSCCP        RTDB reload complete