6 Using Notifications
The notification system allows you to notify Enterprise Manager administrators when specific incidents, events, or problems arise.
Note:
This chapter assumes that you are familiar with incident management. For information about monitoring and managing your IT infrastructure via incident management, see Using Incident Management.
As an integral part of the management framework, notifications can also perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when specific incidents, events, or problems occur. This capability allows you to automate IT practices. For example, if an incident (such as monitoring of the operational (up/down) status of a database) arises, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.
By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView for events published in Enterprise Manager. In Enterprise Manager 13.5 RU15, the Enterprise Manager notification system also supports sending incident or events to third-party applications through Webhooks. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.
This chapter covers the following:
Setting Up Notifications
All Enterprise Manager administrators can set up email notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.
Setting Up a Mail Server for Notifications
Before Enterprise Manager can send email notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define email notifications for yourself or, if you have Super Administrator privileges, you can also define notifications for other Enterprise Manager administrators.
You specify the Outgoing Mail (SMTP) server on the Notification Methods page. To display the Notification Methods page, from the Setup menu, select Notifications, then select Mail Servers.
Note:
You must have Super Administrator privileges in order to configure the Enterprise Manager notifications system. This includes:
-
Setting up the SMTP server
-
Defining notification methods
-
Customizing notification email formats
Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the email address you want to use to send your email notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this email address if any problem is encountered during the sending of an email notification. Example 6-1 shows sample notification method entries.
Note:
The email address you specify on this page is not the email address to which the notification is sent. You will have to specify the email address (where notifications will be sent) from the Password and Email page. From the Setup menu, choose MyPreferences and then Enterprise Manager Password & Email.
As standard practice, each user should have their own email address.
After configuring the email server, click Test Mail Servers to verify your email setup. You should verify that an email message was received by the email account specified in the Sender's Email Address field.
Defining multiple mail servers will improve the reliability of email notification delivery. Email notifications will be delivered if at least one email server is up. The notification load is balanced across multiple email servers by the OMS, which switches through them (servers are allocated according to availability) after 20 emails have been sent. Switching is controlled by the oracle.sysman.core.notification.emails_per_connection emoms property.
Setting the Cloud Control Console URL when Using an SLB
If you have a multi-OMS environment with a Server Load Balancer (SLB) configured for the OMS instances, you should update the console URL to ensure that any emails from Enterprise Manager direct you to the Enterprise Manager console through the SLB URL and not the specific OMS URL from which the email may have originated.
To change the console URL:
- On the Management Services and Repository page, in the Overview section, click Add/Edit against the Console URL label.
The Console URL page displays.
Example 6-1 Mail Server Settings
-
Outgoing Mail (SMTP) Server - smtp01.example.com:587, smtp02.example.com
-
User Name - myadmin
-
Password - ******
-
Confirm Password - ******
-
Identify Sender As - Enterprise Manager
-
Sender's Email Address - mgmt_rep@example.com
-
Use Secure Connection - No: Email is not encrypted. SSL: Email is encrypted using the Secure Sockets Layer protocol. TLS, if available: Email is encrypted using the Transport Layer Security protocol if the mail server supports TLS. If the server does not support TLS, the email is automatically sent as plain text.
Setting Up Email for Yourself
If you want to receive notifications by email, you will need to specify your email address(s) in the Password & Email page (from the Setup menu, select MyPreferences, then select Enterprise Manager Password & Email). In addition to defining notification email addresses, you associate the notification message format (long, short, pager) to be used for your email address.
Setting up email involves three steps:
Step 1: Define an email addresses.
Step 2: Set up a Notification Schedule.
Step 3: Subscribe to incident rules in order to receive emails.
Defining Email Addresses
An email address can have up to 128 characters. There is no upper limit with the number of email addresses.
To add an email address:
Example 6-2 Long Email Notification for Metric Alerts
Target type=Host Target name=machine6140830.example.com Message=Filesystem / has 54.39% available space, fallen below warning (60) or critical (30) threshold. Severity=Warning Event reported time=Apr 28, 2011 2:33:55 PM PDT Event Type=Metric Alert Event name=Filesystems:Filesystem Space Available (%) Metric Group=FilesystemsMetric=Filesystem Space Available (%)Metric value=54.39Key Value=/Key Column 1=Mount PointRule Name=NotifRuleSet1,Event rule1 Rule Owner=SYSMAN
Example 6-3 Short Email Notification for Alerts
Subject is : EM:Unreachable Start:myhost Body is : Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused) but the host is UP
More about Email(Short) and Pager(Short) Formats
Enterprise Manager does not directly support message services such as paging or SMS, but instead relies on external gateways to, for example, perform the conversion from email to page. Beginning with Enterprise Manager 12c, the notification system allows you to tag email addresses explicitly as 'page' or 'email'. Explicit system differentiation between these two notification methods allows you to take advantage of the multiple action capability of incident rules. For example, the email versus page distinction is required in order to send you an email if an event severity is 'warning' or page you if the severity is 'critical'. To support this capability, a Pager format has been made available that sends an abbreviated version of the short format email.
Note:
To receive a Page, an administrator should be added to the Page Notification option in the Incident Rule.
Setting Up a Notification Schedule
Once you have defined your email notification addresses, you will need to define a notification schedule. For example, if your email addresses are user1@myco.com, user2@myco.com, user3@myco.com, you can choose to use one or more of these email addresses for each time period in your notification schedule. Only email addresses that have been specified with your user preferences (Enterprise Manager Password and Email page) can be used in the notification schedule.
Note:
When you enter email addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.
A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and email addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the email address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.
If you are a regular Enterprise Manager administrator and are defining your own notification schedule:
- From Setup menu, select Notifications, then select My Notification Schedule.
- Follow the directions on the Notification Schedule page to specify when you want to receive emails.
Subscribe to Receive Email for Incident Rules
An incident rule is a user-defined rule that specifies the criteria by which notifications should be sent for specific events that make up the incident. An incident rule set, as the name implies, consists of one or more rules associated with the same incident.
When creating an incident rule, you specify criteria such as the targets you are interested in, the types of events to which you want the rule to apply. Specifically, for a given rule, you can specify the criteria you are interested in and the notification methods (such as email) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, email should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. Notification flexibility is further enhanced by the fact that with a single rule, you can perform multiple actions based on specific conditions. Example: When monitoring a condition such as machine memory utilization, for an incident severity of 'warning' (memory utilization at 80%), send the administrator an email, if the severity is 'critical' (memory utilization at 99%), page the administrator immediately.
You can subscribe to a rule you have already created.
-
From the Setup menu, select Incidents, then select Incident Rules.
-
On the Incident Rules - All Enterprise Rules page, click on the rule set containing incident escalation rule in question and click Edit... Rules are created in the context of a rule set.
Note: In the case where there is no existing rule set, create a rule set by clicking Create Rule Set... You then create the rule as part of creating the rule set.
-
In the Rules section of the Edit Rule Set page, highlight the escalation rule and click Edit....
-
Navigate to the Add Actions page.
-
Select the action that escalates the incident and click Edit...
-
In the Notifications section, add the DBA to the Email cc list.
-
Click Continue and then navigate back to the Edit Rule Set page and click Save.
Out-of-Box Incident Rules
Enterprise Manager comes with two incident rule sets that cover the most common monitoring conditions, they are:
-
Incident Management Ruleset for All Targets
-
Event Management Ruleset for Self Update
If the conditions defined in the out-of-box incident rules meet your requirements, you can simply subscribe to receive email notifications for the conditions defined in the rule using the subscribe procedure shown in the previous section.
The out-of-box incident rule set for all targets does not generate emails for warning alerts by default.
Creating Your Own Incident Rules
You can define your own custom rules. The following procedure documents the process of incident rule creation for non-Super Administrators.
To create your own incident rule:
Setting Up Email for Other Administrators
If you have Super Administrator privileges, you can set up email notifications for other Enterprise Manager administrators. To set up email notifications for other Enterprise Manager administrators, you need to:
Step 1: Ensure Each Administrator Account has an Associated Email Address
Each administrator to which you want to send email notifications must have a valid email address.
-
From the Setup menu, select Security and then Administrators.
-
For each administrator, define an email address. This sets up a 24x7 notification schedule for this user that uses all the email addresses specified. By default, this adds the Email ID with type set to Email Long. It is not possible to specify the Email Type option here.
Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.
Step 2: Define Administrators' Notification Schedules
Once you have defined email notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specify an email address for the first time, you should review and edit the notification schedule as needed.
-
From the Setup menu, select Notifications, then select Notification Schedule.
From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.
-
Specify the administrator who's notification schedule you wish to edit and click Change.
-
Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.
-
Click Continue. The Edit Schedule Definition: Email Addresses page appears.
-
Follow the directions on the Edit Schedule Definition: Email Addresses page to modify the notification schedule.
-
Click Finish when you are done.
-
Repeat steps three through seven for each administrator.
Step 3: Assign Incident Rules to Administrators
With the notification schedules set, you now need to assign the appropriate incident rules for each designated administrator.
- From the Setup menu, select Incidents, then select Incident Rules.
- Select the desired Ruleset and click Edit.
- Click on the Rules tab, select the desired rule and click Edit.
- Click Add Actions, select desire action and click Edit.
- Enter the Administrator name on either Email To or Email Cc field in the Basic Notification region.
- Click Continue, click Next, click Next, click Continue, and finally click Save.
Email Customization
Enterprise Manager allows Super Administrators to customize global email notifications for the following types: All events, incidents, problems, and specific event types installed. You can alter the default behavior for all events by customizing Default Event Email Template. In addition, you can further customize the behavior for a specific event type by customizing the template for the event type. For instance, you can customize the Metric Alert Events template for the metric alert event type. Using predefined building blocks (called attributes and labels) contained within a simple script, Super Administrators can customize alert emails by selecting from a wide variety of information content.
To customize an email:
- From the Setup menu, select Notifications, then select Customize Email Formats.
- Choose the Type and Format.
- Click Customize. The Customize Email Template page displays.
From the Customize Email Template page, you can modify the content of the email template Enterprise Manager uses to generate email notifications. Extensive information on script formatting, syntax, and options is available from the Edit Email Template page via imbedded assistance and online help.
Email Customization Reference
The following reference summarizes the semantics and component syntax of the pseudo-language used to define emails. The pseudo-language provides you with a simple, yet flexible way to customize email notifications. The following is a summary of pseudo-language conventions/limitations:
-
You can add comments (or any free-form text) using separate lines beginning with "--" or at end of lines.
-
You can use attributes.
-
You can use IF & ELSE & ENDIF control structures. You can also use multiple conditions using "AND" or "OR". Nested IF statements are not supported.
-
You can insert spaces for formatting purposes. Spaces at the beginning of a line will be ignored in the actual email. To insert spaces at the beginning of a line, use the [SP] attribute.
-
Use "/" to escape and "[" or "]" if you want to add attribute names, operators, or IF clauses to the actual email.
-
HTML is not supported.
Reserved Words and Operators
The following table lists all reserved words and operators used when modifying email scripts.
Table 6-1 Reserved Words and Operators
| Reserved Word/Operator | Description |
|---|---|
|
IF, ELSIF, ENDIF, ELSE |
Used in IF-ELSE constructs. |
|
AND, OR |
Boolean operators – used in IF-ELSE constructs only. |
|
NULL |
To check NULL value for attributes - used in IF-ELSE constructs only. |
|
| |
Pipe operator – used to show the first non-NULL value in a list of attributes. For example:
|
|
EQ, NEQ |
Equal and Not-Equal operators – applicable to NULL, STRING and NUMERIC values. |
|
/ |
Escape character – used to escape reserved words and operators. Escape characters signify that what follows the escape character takes an alternative interpretation. |
|
[ , ] |
Delimiters used to demarcate attribute names and IF clauses. |
Syntax Elements
Literal Text
You can specify any text as part of the email content. The text will be displayed in the email and will not be translated if the Oracle Management Services (OMS) language setting is changed. For example, ‘my Oracle Home' appears as ‘my Oracle Home' in the generated email.
Predefined Attributes
Predefined attributes/labels will be substituted with actual values in a specific context. To specify a predefined attribute/label, use the following syntax:
[PREDEFINED_ATTR]
Attribute names can be in either UPPER or LOWER case. The parsing process is case-insensitive.
A pair of square brackets is used to distinguish predefined attributes from literal text. For example, for a job email notification, the actual job name will be substituted for [EXECUTION_STATUS]. For a metric alert notification, the actual metric column name will be substituted for [METIRC_COLUMN].
You can use the escape character “/" to specify words and not have them interpreted as predefined labels/attributes. For example, "/[NEW/]" will not be considered as the predefined attribute [NEW] when parsed.
Operators
EQ, NEQ – for text and numeric values
NULL- for text and numeric values
GT, LT, GE, LE – for numeric values
Control Structures
The following table lists acceptable script control structures.
Table 6-2 Control Structures
| Control Structure | Description |
|---|---|
|
Pipe "|" |
Two or more attributes can be separated by ‘|' character. For example, In this example, only the applicable attribute within the current alert context will be used (replaced by the actual value) in the email. If more than one attribute is applicable, only the left-most attribute is used. |
|
IF |
Allows you to make a block of text conditional. Only one level of IF and ELSIF is supported. Nested IF constructs are not supported. All attributes can be used in IF or ELSIF evaluation using EQ/NEQ operators on NULL values. Other operators are allowed for “SEVERITY" and “REPEAT_COUNT" only. Inside the IF block, the values need to be contained within quotation marks “ ". Enterprise Manager will extract the attribute name and its value based on the position of “EQ" and other key words such as “and", “or". For example, [IF REPEAT_COUNT EQ “1" AND SEVERITY EQ “CRITICAL" THEN] The statement above will be true when the attributes of the alert match the following condition:
Example IF Block: [IF EXECUTION_STATUS NEQ NULL]
[JOB_NAME_LABEL]=[EXECUTION_STATUS]
[JOB_OWNER_LABEL]=[JOB_OWNER]
[ENDIF]
[IF SEVERITY_CODE EQ CRITICAL ]
[MTRIC_NAME_LABEL]=[METRIC_GROUP]
[METRIC_VALUE_LABEL]=[METRIC_VALUE]
[TARGET_NAME_LABEL]=[TARGET_NAME]
[KEY_VALUES]
[ENDIF]
Example IF and ELSEIF Block: [IF SEVERITY_CODE EQ CRITICAL] statement1[ELSIF SEVERITY_CODE EQ WARNING] statement2[ELSIF SEVERITY_CODE EQ CLEAR] statement3[ELSE] statement4[ENDIF] |
Comments
You can add comments to your script by prefacing a single line of text with two hyphens "--". For example,
-- Code added on 8/3/2009 [IF REPEAT_COUNT NEQ NULL] . . .
Comments may also be placed at the end of a line of text.
[IF SEVERITY_SHORT EQ W] -- for Warning alert
HTML Tags in Customization Content
Use of HTML tags is not supported.
When Enterprise Manager parses the email script, it will convert the “<" and “>" characters of HTML tags into encoded format (< and >). This ensures that the HTML tag is not treated as HTML by the destination system.
Examples
Email customization template scripts support three main operators.
-
Comparison operators: EQ/NEQ/GT/LT/GE/LELogic operators: AND/ORPipeline operator: |
Setting Up Repeat Notifications
Repeat notifications allow administrators to be notified repeatedly until an incident is either acknowledged or the number of Maximum Repeat Notifications has been reached. Enterprise Manager supports repeat notification for all notification methods (email, OS command, PL/SQL procedure, and SNMP trap).
Configuring Repeat Notifications Globally
To enable repeat notifications for a notification method (globally), select the Send Repeat Notifications option on the Notification Methods page . In addition to setting the maximum number of repeat notifications, you can also set the time interval at which the notifications are sent.
Note:
For Oracle database versions 10 and higher, it is recommend that no modification be made to aq_tm_processes init.ora parameter. If, however, this parameter must be modified, its value should be at least one for repeat notification functionality. If the Enterprise Manager Repository database version is 9.2, the aq_tm_processes init.ora parameter must be set to at least one to enable repeat notification functionality.
Configuring Repeat Notifications Via Incident Rules
Setting repeat notifications globally at the notification method level may not provide sufficient flexibility. For example, you may want to have different repeat notification settings based on event type. Enterprise Manager accomplishes this by allowing you to set repeat notifications for individual incident rule sets or individual rules within a rule set. Repeat notifications set at the rule level take precedence over those defined at the notification method level.
Note:
Repeat notifications will only be sent if the Send Repeat Notifications option is enabled in the Notification Methods page.
Non-Email Repeat Notifications
For non-email repeat notifications (PL/SQL, OS command, and SNMP trap notification methods), you must enable each method to support repeat notifications. You can select Supports Repeat Notifications option when adding a new notification method or by editing an existing method.
Extending Notification Beyond Email
Notification Methods are the mechanisms by which notifications are sent. Enterprise Manager Super Administrators can set up email notifications by configuring the 'email' notification method. Most likely this would already have been set up as part of the Oracle Management Service installation.
Enterprise Manager Super Administrators can also define other custom notification methods. For example, event notifications may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once the custom method is defined, whenever an administrator needs to send alerts to the trouble-ticketing system, he simply needs to invoke the now globally available notification method called "Log trouble ticket".
Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps. A fourth type of notification method (Java Callback) exists to support Oracle internal functionality and cannot be created or edited by Enterprise Manager administrators.
Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) as actions to their incident rules.
Through the Notification Methods page, you can:
Sending Notifications Using OS Commands and Scripts
Notification system can invoke a custom script when an incident rule matches the OS Command advanced notification action. A custom script receives notifications for matching events, incidents and problem through environment variables.
The length of any environment variable's value is limited to 512 characters by default. Configure emoms property named oracle.sysman.core.notification.oscmd.max_env_var_length for changing the default limit.
Note:
Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.
Note:
Running an OS command such as "sudo" for receiving notifications will fail because the command does not have read permission of the OMS account. The OMS account must have read permission over the OS command in order to send notifications.
To overcome the permissions problem, embed the command in a wrapper script that is readable by the OMS administrator account. Once the command is contained within the wrapper script, you then specify this script in place of the OS command.
Registering a Custom Script
In order to use a custom script, you must first register the script with the notification system. This is performed in four steps:
Example 6-4 Changing the oracle.sysman.core.notification.os_cmd_timeout emoms Property
emctl set property -name oracle.sysman.core.notification.os_cmd_timeout value 30
Example 6-5 OS Command Notification Method
Name Trouble Ticketing Description Notification method to log trouble ticket for a severity occurrence OS Command /private/mozart/bin/logTicket.sh
Step 1: Define your OS command or script.
You can specify an OS command or script that will be called by the notification system when an incident rule matches the OS Command advanced notification action. You can use incident, event, or problem context information, corrective action execution status and job execution status within the body of the script. Passing this contextual information to OS commands/scripts allows you to customize automated responses specific event conditions. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see:
-
"Passing Event, Incident, Problem Information to an OS Command or Script"
-
"Passing Corrective Action Execution Status to an OS Command or Script"
-
"Passing Job Execution Status to an OS Command or Script"
Step 2: Deploy the script on each Management Service host.
You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service. The OS Command or Script should be deployed on the same location on each Management Service host machine.
Note:
Both scripts and OS Commands should be specified using absolute paths. For example, /u1/bin/logSeverity.sh.
The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes, eventually progressing to 30 minutes. From here, the procedure is retried every 30 minutes until the notification is a 24 hours old. The notification will be then be purged.
Example 6-4 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This prevents OS Commands from running for an inordinate length of time and blocks the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed. The oracle.sysman.core.notification.os_cmd_timeout emoms property can be configured to change the default timeout value.
Step 3: Register your OS Command or Script as a new Notification Method.
Add this OS command as a notification method that can be called in incident rules. Log in as a Super Administrator. From the Setup menu, select Notifications, then select Notification Methods. From this page, you can define a new notification based on the 'OS Command' type. See "Sending Notifications Using OS Commands and Scripts".
The following information is required for each OS command notification method:
-
Name
-
Description
Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.
-
OS Command
You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).
Example 6-5 shows information required for the notification method.
Note:
There can be more than one OS Command configured per system.
Step 4: Assign the notification method to an incident rule.
You can edit an existing rule (or create a new instance rule), then go to the Methods page. From the Setup menu, choose Incidents and then Incident Rules. The Incident Rules page provides access to all available rule sets.
For detailed reference information on passing event, incident, and problem information to an OS Command or script, see "Passing Event, Incident, Problem Information to an OS Command or Script".
Script Examples
The sample OS script shown in Example 6-6 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification
Example 6-6 Sample OS Command Script
#!/bin/ksh LOG_FILE=/net/myhost/logs/event.log if test -f $LOG_FILE then echo $TARGET_NAME $MESSAGE $EVENT_REPORTED_TIME >> $LOG_FILE else exit 100 fi
Example 6-7 shows an OS script that logs alert information for both incidents and events to the file 'oscmdNotify.log'. The file is saved to the /net/myhost/logs directory.
Example 6-7 Alert Logging Scripts
#!/bin/sh# LOG_FILE=/net/myhost/logs/oscmdNotify.log echo '-------------' >> $LOG_FILE echo 'issue_type=' $ISSUE_TYPE >> $LOG_FILE echo 'notif_type=' $NOTIF_TYPE >> $LOG_FILE echo 'message=' $MESSAGE >> $LOG_FILE echo 'message_url' = $MESSAGE_URL >>$LOG_FILE echo 'severity=' $SEVERITY >> $LOG_FILE echo 'severity_code' = $SEVERITY_CODE >>$LOG_FILE echo 'ruleset_name=' $RULESET_NAME >> $LOG_FILE echo 'rule_name=' $RULE_NAME >> $LOG_FILE echo 'rule_owner=' $RULE_OWNER >> $LOG_FILE echo 'repeat_count=' $REPEAT_COUNT >> $LOG_FILE echo 'categories_count' = $CATEGORIES_COUNT >>$LOG_FILE echo 'category_1' = $CATEGORY_1 >>$LOG_FILE echo 'category_2' = $CATEGORY_2 >>$LOG_FILE echo 'category_code_1' = $CATEGORY_CODE_1 >>$LOG_FILE echo 'category_code_2' = $CATEGORY_CODE_2 >>$LOG_FILE echo 'category_codes_count' = $CATEGORY_CODES_COUNT >>$LOG_FILE # event if [ $ISSUE_TYPE -eq 1 ] then echo 'host_name=' $HOST_NAME >> $LOG_FILE echo 'event_type=' $EVENT_TYPE >> $LOG_FILE echo 'event_name=' $EVENT_NAME >> $LOG_FILE echo 'event_occurrence_time=' $EVENT_OCCURRENCE_TIME >> $LOG_FILE echo 'event_reported_time=' $EVENT_REPORTED_TIME >> $LOG_FILE echo 'sequence_id=' $SEQUENCE_ID >> $LOG_FILE echo 'event_type_attrs=' $EVENT_TYPE_ATTRS >> $LOG_FILE echo 'source_obj_name=' $SOURCE_OBJ_NAME >> $LOG_FILE echo 'source_obj_type=' $SOURCE_OBJ_TYPE >> $LOG_FILE echo 'source_obj_owner=' $SOURCE_OBJ_OWNER >> $LOG_FILE echo 'target_name' = $TARGET_NAME >>$LOG_FILE echo 'target_url' = $TARGET_URL >>$LOG_FILE echo 'target_owner=' $TARGET_OWNER >> $LOG_FILE echo 'target_type=' $TARGET_TYPE >> $LOG_FILE echo 'target_version=' $TARGET_VERSION >> $LOG_FILE echo 'lifecycle_status=' $TARGET_LIFECYCLE_STATUS >> $LOG_FILE echo 'assoc_incident_escalation_level' = $ASSOC_INCIDENT_ESCALATION_LEVEL >>$LOG_FILE echo 'assoc_incident_id' = $ASSOC_INCIDENT_ID >>$LOG_FILE echo 'assoc_incident_owner' = $ASSOC_INCIDENT_OWNER >>$LOG_FILE echo 'assoc_incident_acknowledged_by_owner' = $ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER >>$LOG_FILE echo 'assoc_incident_acknowledged_details' = $ASSOC_INCIDENT_ACKNOWLEDGED_DETAILS >>$LOG_FILE echo 'assoc_incident_priority' = $ASSOC_INCIDENT_PRIORITY >>$LOG_FILE echo 'assoc_incident_status' = $ASSOC_INCIDENT_STATUS >>$LOG_FILE echo 'ca_job_status' = $CA_JOB_STATUS >>$LOG_FILE echo 'event_context_attrs' = $EVENT_CONTEXT_ATTRS >>$LOG_FILE echo 'last_updated_time' = $LAST_UPDATED_TIME >>$LOG_FILE echo 'sequence_id' = $SEQUENCE_ID >>$LOG_FILE echo 'test_date_attr_noref' = $TEST_DATE_ATTR_NOREF >>$LOG_FILE echo 'test_raw_attr_noref' = $TEST_RAW_ATTR_NOREF >>$LOG_FILE echo 'test_str_attr1' = $TEST_STR_ATTR1 >>$LOG_FILE echo 'test_str_attr2' = $TEST_STR_ATTR2 >>$LOG_FILE echo 'test_str_attr3' = $TEST_STR_ATTR3 >>$LOG_FILE echo 'test_str_attr4' = $TEST_STR_ATTR4 >>$LOG_FILE echo 'test_str_attr5' = $TEST_STR_ATTR5 >>$LOG_FILE echo 'test_str_attr_ref' = $TEST_STR_ATTR_REF >>$LOG_FILE echo 'total_occurrence_count' = $TOTAL_OCCURRENCE_COUNT >>$LOG_FILE fi # incident if [ $ISSUE_TYPE -eq 2 ] then echo 'action_msg=' $ACTION_MSG >> $LOG_FILE echo 'incident_id=' $INCIDENT_ID >> $LOG_FILE echo 'incident_creation_time=' $INCIDENT_CREATION_TIME >> $LOG_FILE echo 'incident_owner=' $INCIDENT_OWNER >> $LOG_FILE echo 'incident_acknowledged_by_owner' = $INCIDENT_ACKNOWLEDGED_BY_OWNER >>$LOG_FILE echo 'incident_status' = $INCIDENT_STATUS >>$LOG_FILE echo 'last_modified_by=' $LAST_MODIFIED_BY >> $LOG_FILE echo 'last_updated_time=' $LAST_UPDATED_TIME >> $LOG_FILE echo 'assoc_event_count=' $ASSOC_EVENT_COUNT >> $LOG_FILE echo 'adr_incident_id=' $ADR_INCIDENT_ID >> $LOG_FILE echo 'occurrence_count=' $OCCURRENCE_COUNT >> $LOG_FILE echo 'escalated=' $ESCALATED >> $LOG_FILE echo 'escalated_level=' $ESCALATED_LEVEL >> $LOG_FILE echo 'priority=' $PRIORITY >> $LOG_FILE echo 'priority_code' = $PRIORITY_CODE >>$LOG_FILE echo 'ticket_id=' $TICKET_ID >> $LOG_FILE echo 'ticket_status=' $TICKET_STATUS >> $LOG_FILE echo 'ticket_url=' $TICKET_ID_URL >> $LOG_FILE echo 'total_duplicate_count=' $TOTAL_DUPLICATE_COUNT >> $LOG_FILE echo 'source_count=' $EVENT_SOURCE_COUNT >> $LOG_FILE echo 'event_source_1_host_name' = $EVENT_SOURCE_1_HOST_NAME >>$LOG_FILE echo 'event_source_1_target_guid' = $EVENT_SOURCE_1_TARGET_GUID >>$LOG_FILE echo 'event_source_1_target_name' = $EVENT_SOURCE_1_TARGET_NAME >>$LOG_FILE echo 'event_source_1_target_owner' = $EVENT_SOURCE_1_TARGET_OWNER >>$LOG_FILE echo 'event_source_1_target_type' = $EVENT_SOURCE_1_TARGET_TYPE >>$LOG_FILE echo 'event_source_1_target_url' = $EVENT_SOURCE_1_TARGET_URL >>$LOG_FILE echo 'event_source_1_target_lifecycle_status' = $EVENT_SOURCE_1_TARGET_LIFECYCLE_STATUS >>$LOG_FILE echo 'event_source_1_target_version' = $EVENT_SOURCE_1_TARGET_VERSION >>$LOG_FILE fi exit 0
Example 6-8 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Cloud Control. When a metric alert is triggered, the Enterprise Manager Cloud Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.
Example 6-8 HP OpenView Script
/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_text="$MESSAGE" object="$TARGET_NAME"
Migrating pre-12c OS Command Scripts
This section describes how to map pre-12c OS Command notification shell environment variables to 13c OS Command shell environment variables.
Note:
Pre-12c notification rules only map to event level rules. Mapping to incident level rules is not permitted.
Note:
Policy Violations are no longer supported beginning with the Enterprise Manager 12c release.
Migrating Metric Alert Event Types
Following table is the mapping for the OS Command shell environment variables when the event_type is metric_alert.
Table 6-3 Pre-12c/13c metric_alert Environment Variable Mapping
| Pre-12c Environment Variable | Corresponding 13c Environment Variables |
|---|---|
|
ACKNOWLEDGED |
ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER |
|
ACKNOWLEDGED_BY |
ASSOC_INCIDENT_OWNER |
|
CYCLE_GUID |
CYCLE_GUID |
|
HOST |
HOST_NAME |
|
KEY_VALUE |
Note: See detail description below. |
|
KEY_VALUE_NAME |
Note: See detail description below |
|
MESSAGE |
MESSAGE |
|
METRIC |
METRIC_COLUMN |
|
NOTIF_TYPE |
NOTIF_TYPE; use the map in section 2.3.5 |
|
REPEAT_COUNT |
REPEAT_COUNT |
|
RULE_NAME |
RULESET_NAME |
|
RULE_OWNER |
RULE_OWNER |
|
SEVERITY |
SEVERITY |
|
TARGET_NAME |
TARGET_NAME |
|
TARGET_TYPE |
TARGET_TYPE |
|
TIMESTAMP |
EVENT_REPORTED_TIME |
|
METRIC_VALUE |
VALUE |
|
VIOLATION_CONTEXT |
EVENT_CONTEXT_ATTRS |
|
VIOLATION_GUID |
SEVERITY_GUID |
|
POLICY_RULE |
No mapping, Obsolete as of Enterprise Manager 12c release. |
To obtain KEY_VALUE_NAME and KEY_VALUE, perform the following steps.
-
If $NUM_KEYS variable is null, then $KEY_VALUE_NAME and $KEY_VALUE are null.
-
If $NUM_KEYS equals 1
KEY_VALUE_NAME=$KEY_COLUMN_1
KEY_COLUMN_1_VALUE
-
If $NUM_KEYS is greater than 1
KEY_VALUE_NAME="$KEY_COLUMN_1;$KEY_COLUMN_2;..;KEY_COLUMN_x"
KEY_VALUE="$KEY_COLUMN_1_VALUE;$KEY_COLUMN_2_VALUE;..;KEY_COLUMN_x_VALUE "
Where x is the value of $NUM_KEYS and ";" is the separator.
Migrating Target Availability Event Types
Following table is the mapping for the OS Command shell environment variables when the event_type is 'target_availability'.
Table 6-4 pre-12c/12c target_availability Environment Variable Mappings
| Pre-12c Environment Variable | Corresponding 13c Environment Variables |
|---|---|
|
TARGET_NAME |
TARGET_NAME |
|
TARGET_TYPE |
TARGET_TYPE |
|
METRIC |
Status |
|
CYCLE_GUID |
CYCLE_GUID |
|
VIOLATION_CONTEXT |
EVENT_CONTEXT_ATTRS |
|
SEVERITY |
TARGET_STATUS |
|
HOST |
HOST_NAME |
|
MESSAGE |
MESSAGE |
|
NOTIF_TYPE |
NOTIF_TYPE; use the map in section 2.3.5 |
|
TIMESTAMP |
EVENT_REPORTED_TIME |
|
RULE_NAME |
RULESET_NAME |
|
RULE_OWNER |
RULE_OWNER |
|
REPEAT_COUNT |
REPEAT_COUNT |
|
KEY_VALUE |
"" |
|
KEY_VALUE_NAME |
"" |
Migrating Job Status Change Event Types
Following table is the mapping for the OS Command shell environment variables when the event_type is 'job_status_change'.
Table 6-5 pre-12c/13c job_status_change Environment Variable Mappings
| Pre-12c Environment Variable | Corresponding 13c Environment Variables |
|---|---|
|
JOB_NAME |
SOURCE_OBJ_NAME |
|
JOB_OWNER |
SOURCE_OBJ_OWNER |
|
JOB_TYPE |
SOURCE_OBJ_SUB_TYPE |
|
JOB_STATUS |
EXECUTION_STATUS |
|
NUM_TARGETS |
1 if $ TARGET_NAME is not null, 0 otherwise |
|
TARGET_NAME1 |
TARGET_NAME |
|
TARGET_TYPE1 |
TARGET_TYPE |
|
TIMESTAMP |
EVENT_REPORTED_TIME |
|
RULE_NAME |
RULESET_NAME |
|
RULE_OWNER |
RULE_OWNER |
Migrating Corrective Action-Related OS Scripts
Refer to section "Migrating Metric Alert Event Types" for mapping the following environment variables while receiving notifications for corrective actions.
KEY_VALUE
KEY_VALUE_NAME
METRIC
METRIC_VALUE
RULE_NAME
RULE_OWNER
SEVERITY
TIMESTAMP
VIOLATION_CONTEXT
Use the map below for mapping other environment variables.
Table 6-6 pre-12c/13c Corrective Action Environment Variable Mappings
| Pre-12c Environment Variable | Corresponding 13c Environment Variables |
|---|---|
|
NUM_TARGETS |
1 |
|
TARGET_NAME1 |
TARGET_NAME |
|
TARGET_TYPE1 |
TARGET_TYPE |
|
JOB_NAME |
CA_JOB_NAME |
|
JOB_OWNER |
CA_JOB_OWNER |
|
JOB_STATUS |
CA_JOB_STATUS |
|
JOB_TYPE |
CA_JOB_TYPE |
Sending Notifications Using PL/SQL Procedures
A user-defined PL/SQL procedure can receive notifications for matching events, incidents and problems.
Note:
When upgrading from pre-12c to 13c versions of Enterprise Manager, existing pre-12c PL/SQL advanced notification methods will continue to work without modification. You should, however, update the procedures to use new signatures.
New PL/SQL advanced notification methods created with Enterprise Manager 13c must use the new signatures documented in the following sections.
Complete the following four steps to define a notification method based on a PL/SQL procedure.
Defining a PL/SQL-based Notification Method
Creating a PL/SQL-based notification method consists of four steps:
- Define the PL/SQL procedure.
- Create the PL/SQL procedure on the Management Repository.
- Register your PL/SQL procedure as a new notification method.
- Assign the notification method to an incident rule.
Example 6-9 PL/SQL Procedure Required Information
Name Open trouble ticket Description Notification method to open a trouble ticket in the event PLSQL Procedure ticket_sys.ticket_ops.open_ticket
Example 6-10 PL/SQL Script
-- Assume log_table is created by following DDL
-- CREATE TABLE log_table (message VARCHAR2(4000)) ;
-- Define PL/SQL notification method for Events
CREATE OR REPLACE PROCEDURE log_table_notif_proc(s IN GC$NOTIF_EVENT_MSG)
IS
l_categories gc$category_string_array;
l_category_codes gc$category_string_array;
l_attrs gc$notif_event_attr_array;
l_ca_obj gc$notif_corrective_action_job;
BEGIN
INSERT INTO log_table VALUES ('notification_type: ' || s.msg_info.notification_type);
INSERT INTO log_table VALUES ('repeat_count: ' || s.msg_info.repeat_count);
INSERT INTO log_table VALUES ('ruleset_name: ' || s.msg_info.ruleset_name);
INSERT INTO log_table VALUES ('rule_name: ' || s.msg_info.rule_name);
INSERT INTO log_table VALUES ('rule_owner: ' || s.msg_info.rule_owner);
INSERT INTO log_table VALUES ('message: ' || s.msg_info.message);
INSERT INTO log_table VALUES ('message_url: ' || s.msg_info.message_url);
INSERT INTO log_table VALUES ('event_instance_guid: ' || s.event_payload.event_instance_guid);
INSERT INTO log_table VALUES ('event_type: ' || s.event_payload.event_type);
INSERT INTO log_table VALUES ('event_name: ' || s.event_payload.event_name);
INSERT INTO log_table VALUES ('event_msg: ' || s.event_payload.event_msg);
INSERT INTO log_table VALUES ('source_obj_type: ' || s.event_payload.source.source_type);
INSERT INTO log_table VALUES ('source_obj_name: ' || s.event_payload.source.source_name);
INSERT INTO log_table VALUES ('source_obj_url: ' || s.event_payload.source.source_url);
INSERT INTO log_table VALUES ('target_name: ' || s.event_payload.target.target_name);
INSERT INTO log_table VALUES ('target_url: ' || s.event_payload.target.target_url);
INSERT INTO log_table VALUES ('severity: ' || s.event_payload.severity); INSERT INTO log_table VALUES ('severity_code: ' || s.event_payload.severity_code);
INSERT INTO log_table VALUES ('event_reported_date: ' || to_char(s.event_payload.reported_date, 'D MON DD HH24:MI:SS'));
l_categories := s.event_payload.categories;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
INSERT INTO log_table VALUES ('category ' || c || ' - ' || l_categories(c));
END LOOP;
END IF;
l_category_codes := s.event_payload.category_codes;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_category_codes.COUNT
LOOP
INSERT INTO log_table VALUES ('category_code ' || c || ' - ' || l_category_codes(c));
END LOOP;
END IF;
l_attrs := s.event_payload.event_attrs;
IF l_attrs IS NOT NULL
THEN
FOR c IN 1..l_attrs.COUNT
LOOP
INSERT INTO log_table VALUES ('EV.ATTR name=' || l_attrs(c).name || ' value=' || l_attrs(c).value || ' nls_value=' || l_attrs(c).nls_value); END LOOP;
END IF;
COMMIT ;
END ;
/
Example 6-11 PL/SQL Script to Log Events to a Table
CREATE TABLE event_log (
notification_type VARCHAR2(32),
repeat_count NUMBER,
ruleset_name VARCHAR2(256),
rule_owner VARCHAR2(256),
rule_name VARCHAR2(256),
message VARCHAR2(4000),
message_url VARCHAR2(4000),
event_instance_guid RAW(16),
event_type VARCHAR2(20),
event_name VARCHAR2(512),
event_msg VARCHAR2(4000),
categories VARCHAR2(4000),
source_obj_type VARCHAR2(120),
source_obj_name VARCHAR2(256),
source_obj_url VARCHAR2(4000),
severity VARCHAR2(128),
severity_code VARCHAR2(32),
target_name VARCHAR2(256),
target_type VARCHAR2(128),
target_url VARCHAR2(4000),
host_name VARCHAR2(256),
timezone VARCHAR2(64),
occured DATE,
ca_guid RAW(16),
ca_name VARCHAR2(128),
ca_owner VARCHAR2(256),
ca_type VARCHAR2(256),
ca_status VARCHAR2(64),
ca_status_code NUMBER,
ca_job_step_output VARCHAR2(4000),
ca_execution_guid RAW(16),
ca_stage_change_guid RAW(16)
)
;
CREATE OR REPLACE PROCEDURE log_event(s IN GC$NOTIF_EVENT_MSG)
IS
l_categories gc$category_string_array;
l_ca_obj gc$notif_corrective_action_job;
l_categories_new VARCHAR2(1000);
BEGIN
-- save event categories
l_categories := s.event_payload.categories;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
l_categories_new := (l_categories_new|| c || ' - ' || l_categories(c)||',');
END LOOP;
END IF;
-- save event message
IF s.msg_info.notification_type = 'NOTIF_CA' AND s.event_payload.corrective_action IS NOT NULL
THEN
l_ca_obj := s.event_payload.corrective_action;
INSERT INTO event_log (notification_type, repeat_count, ruleset_name, rule_name, rule_owner, message, message_url, event_instance_guid, event_type, event_name, event_msg, categories, source_obj_type, source_obj_name, source_obj_url, severity, severity_code, target_name, target_type, target_url, host_name, timezone, occured, ca_guid, ca_name, ca_owner, ca_type, ca_status, ca_status_code, ca_job_step_output, ca_execution_guid, ca_stage_change_guid)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_name,s.msg_info.rule_owner, s.msg_info.message, s.msg_info.message_url, s.event_payload.event_instance_guid, s.event_payload.event_type, s.event_payload.event_name, s.event_payload.event_msg, l_categories_new, s.event_payload.source.source_type, s.event_payload.source.source_name, s.event_payload.source.source_url, s.event_payload.severity, s.event_payload.severity_code, s.event_payload.target.target_name, s.event_payload.target.target_type, s.event_payload.target.target_url, s.event_payload.target.host_name, s.event_payload.target.target_timezone, s.event_payload.occurrence_date, l_ca_obj.JOB_GUID, l_ca_obj.JOB_NAME, l_ca_obj.JOB_OWNER, l_ca_obj.JOB_TYPE, l_ca_obj.JOB_STATUS, l_ca_obj.JOB_STATUS_CODE, l_ca_obj.JOB_STEP_OUTPUT, l_ca_obj.JOB_EXECUTION_GUID, l_ca_obj.JOB_STATE_CHANGE_GUID); ELSE
INSERT INTO event_log (notification_type, repeat_count, ruleset_name, rule_name, rule_owner, message, message_url, event_instance_guid, event_type, event_name, event_msg, categories, source_obj_type, source_obj_name, source_obj_url, severity, severity_code, target_name, target_type, target_url, host_name, timezone, occured, ca_guid, ca_name, ca_owner, ca_type, ca_status, ca_status_code, ca_job_step_output, ca_execution_guid, ca_stage_change_guid)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_name, s.msg_info.rule_owner, s.msg_info.message, s.msg_info.message_url, s.event_payload.event_instance_guid, s.event_payload.event_type, s.event_payload.event_name, s.event_payload.event_msg, l_categories_new, s.event_payload.source.source_type, s.event_payload.source.source_name, s.event_payload.source.source_url, s.event_payload.severity, s.event_payload.severity_code, s.event_payload.target.target_name, s.event_payload.target.target_type, s.event_payload.target.target_url, s.event_payload.target.host_name, s.event_payload.target.target_timezone, s.event_payload.occurrence_date, null,null,null,null,null,null,null,null,null);
END IF;
COMMIT;
END log_event;
/
Example 6-12 PL/SQL Script to Log Incidents to a Table
CREATE TABLE incident_log (
notification_type VARCHAR2(32),
repeat_count NUMBER,
ruleset_name VARCHAR2(256),
rule_owner VARCHAR2(256),
rule_name VARCHAR2(256),
message VARCHAR2(4000),
message_url VARCHAR2(4000),
incident_id VARCHAR2(128),
ticket_url VARCHAR2(4000),
assoc_event_cnt NUMBER,
severity VARCHAR2(128),
severity_code VARCHAR2(32),
priority VARCHAR2(128),
priority_code VARCHAR2(32),
status VARCHAR2(32),
categories VARCHAR2(1000),
target_name VARCHAR2(256),
target_type VARCHAR2(128),
host_name VARCHAR2(256),
timezone VARCHAR2(64),
occured DATE
)
;
CREATE OR REPLACE PROCEDURE log_incident(s IN GC$NOTIF_INCIDENT_MSG)
IS
l_src_info_array GC$NOTIF_SOURCE_INFO_ARRAY;
l_src_info GC$NOTIF_SOURCE_INFO;
l_categories gc$category_string_array;
l_target_obj GC$NOTIF_TARGET;
l_target_name VARCHAR2(256);
l_target_type VARCHAR2(256);
l_target_timezone VARCHAR2(256);
l_hostname VARCHAR2(256);
l_categories_new VARCHAR2(1000);
BEGIN
-- Save Incident categories
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
l_categories_new := (l_categories_new|| c || ' - ' || l_categories(c)||',');
END LOOP;
END IF;
-- GET target info
l_src_info_array := s.incident_payload.incident_attrs.source_info_arr;
IF l_src_info_array IS NOT NULL
THEN
FOR I IN 1..l_src_info_array.COUNT
LOOP
IF l_src_info_array(I).TARGET IS NOT NULL
THEN
l_target_name := l_src_info_array(I).TARGET.TARGET_NAME;
l_target_type := l_src_info_array(I).TARGET.TARGET_TYPE;
l_target_timezone := l_src_info_array(I).TARGET.TARGET_TIMEZONE;
l_hostname := l_src_info_array(I).TARGET.HOST_NAME;
END IF;
END LOOP;
END IF;
-- save Incident notification message INSERT INTO incident_log(notification_type, repeat_count, ruleset_name, rule_owner, rule_name, message, message_url, incident_id, ticket_url, assoc_event_cnt, severity, severity_code, priority, priority_code, status, categories, target_name, target_type, host_name, timezone, occured)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_owner, s.msg_info.rule_name, s.msg_info.message, s.msg_info.message_url, s.incident_payload.incident_attrs.id, s.incident_payload.ticket_url, s.incident_payload.assoc_event_count, s.incident_payload.incident_attrs.severity, s.incident_payload.incident_attrs.severity_code, s.incident_payload.incident_attrs.priority, s.incident_payload.incident_attrs.priority_code, s.incident_payload.incident_attrs.STATUS, l_categories_new, l_target_name, l_target_type, l_hostname,l_target_timezone, s.incident_payload.incident_attrs.creation_date);
COMMIT;
END log_incident;
/
Example 6-13 PL/SQL Script to Log Problems to a Table
CREATE TABLE problem_log (
notification_type VARCHAR2(32),
repeat_count NUMBER,
ruleset_name VARCHAR2(256),
rule_owner VARCHAR2(256),
rule_name VARCHAR2(256),
message VARCHAR2(4000),
message_url VARCHAR2(4000),
problem_key VARCHAR2(850),
assoc_incident_cnt NUMBER,
problem_id NUMBER,
owner VARCHAR2(256),
severity VARCHAR2(128),
severity_code VARCHAR2(32),
priority VARCHAR2(128),
priority_code VARCHAR2(32),
status VARCHAR2(32),
categories VARCHAR2(1000),
target_name VARCHAR2(256),
target_type VARCHAR2(128),
host_name VARCHAR2(256), timezone VARCHAR2(64),
occured DATE
)
;
CREATE OR REPLACE PROCEDURE log_problem(s IN GC$NOTIF_PROBLEM_MSG)
IS
l_src_info_array GC$NOTIF_SOURCE_INFO_ARRAY;
l_src_info GC$NOTIF_SOURCE_INFO;
l_categories gc$category_string_array;
l_target_obj GC$NOTIF_TARGET;
l_target_name VARCHAR2(256);
l_target_type VARCHAR2(256);
l_target_timezone VARCHAR2(256);
l_hostname VARCHAR2(256);
l_categories_new VARCHAR2(1000);
BEGIN
-- Save Problem categories
l_categories := s.problem_payload.problem_attrs.categories;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_categories.COUNT
LOOP
l_categories_new := (l_categories_new|| c || ' - ' || l_categories(c)||',');
END LOOP;
END IF;
-- GET target info
l_src_info_array := s.problem_payload.problem_attrs.source_info_arr;
IF l_src_info_array IS NOT NULL
THEN
FOR I IN 1..l_src_info_array.COUNT
LOOP
IF l_src_info_array(I).TARGET IS NOT NULL
THEN
l_target_name := l_src_info_array(I).TARGET.TARGET_NAME;
l_target_type := l_src_info_array(I).TARGET.TARGET_TYPE;
l_target_timezone := l_src_info_array(I).TARGET.TARGET_TIMEZONE;
l_hostname := l_src_info_array(I).TARGET.HOST_NAME;
END IF;
END LOOP;
END IF;
-- save Problem notification message
INSERT INTO problem_log(notification_type, repeat_count, ruleset_name, rule_owner, rule_name, message, message_url, problem_key, assoc_incident_cnt, problem_id, owner, severity, severity_code, priority, priority_code, status, categories, target_name, target_type, host_name, timezone, occured)
VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_owner, s.msg_info.rule_name, s.msg_info.message, s.msg_info.message_url, s.problem_payload.problem_key, s.problem_payload.ASSOC_INCIDENT_COUNT, s.problem_payload.problem_attrs.id, s.problem_payload.problem_attrs.owner, s.problem_payload.problem_attrs.severity, s.problem_payload.problem_attrs.severity_code, s.problem_payload.problem_attrs.PRIORITY, s.problem_payload.problem_attrs.PRIORITY_CODE, s.problem_payload.problem_attrs.status, l_categories_new, l_target_name, l_target_type, l_hostname,l_target_timezone, s.problem_payload.problem_attrs.CREATION_DATE);
COMMIT;
END log_problem;
/Step 1: Define the PL/SQL Procedure
The procedure must have one of the following signatures depending on the type of notification that will be received.
For Events:
PROCEDURE event_proc(event_msg IN gc$notif_event_msg)
For Incidents:
PROCEDURE incident_proc(incident_msg IN gc$notif_incident_msg)
For Problems:
PROCEDURE problem_proc(problem_msg IN gc$notif_problem_msg)
Note:
The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a incident rule.
For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:
"Passing Information to a PL/SQL Procedure"
"Passing Corrective Action Status Change Information"
"Passing Job Execution Status Information"
Step 2: Create the PL/SQL procedure on the Management Repository.
Create the PL/SQL procedure on the repository database using one of the following procedure specifications:
PROCEDURE event_proc(event_msg IN gc$notif_event_msg)
PROCEDURE incident_proc(incident_msg IN gc$notif_incident_msg)
PROCEDURE problem_proc(problem_msg IN gc$notif_problem_msg)
The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)
If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user-defined exception that uses the error code -20000. The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.
Step 3: Register your PL/SQL procedure as a new notification method.
Log in as a Super Administrator. From the Setup menu, choose Notifications and then Notification Methods to access the Notification Methods page. From this page, you can define a new notification based on 'PL/SQL Procedure'. See Sending Notifications Using PL/SQL Procedures.
Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.
Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:
-
Name
-
Description
-
PL/SQL Procedure
You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.
An example of the required information is shown in Example 6-9.
Figure 6-1 illustrates how to add a PL/SQL-based notification method from the Enterprise Manager UI.
Figure 6-1 Adding a PL/SQL Procedure
Step 4: Assign the notification method to an incident rule.
You can edit an existing rule (or create a new incident rule). From the Setup menu, select Incidents and then select Incident Rules. The Incident Rules page displays. From here, you can add an action to a rule specifying the new PL/SQL procedure found under Advanced Notification Method.
There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.
See "Passing Information to a PL/SQL Procedure" for more information about how incident, event, and problem information is passed to the PLSQL procedure.
Migrating Pre-12c PL/SQL Advanced Notification Methods
Pre-12c notifications map to event notifications in Enterprise Manager 12c. The event types metric_alert, target_availability and job_status_alert correspond to the pre-12c notification functionality.
Note:
Policy Violations are no longer available beginning with Enterprise Manager 12c.
This section describes the mapping between Enterprise Manager 13c PL/SQL notification payload to the pre-12c PL/SQL notification payload. You can use this information for updating the existing pre-12c PL/SQL user callback procedures to use the 13c PL/SQL notification payload.Please note that Policy Violations are no longer supported in the 13c release.
Mapping for MGMT_NOTIFY_SEVERITY
When event type is metric_alert
Use the following map when gc$notif_event_payload .event_type='metric_alert'.
Table 6-8 Metric Alert Mapping
| MGMT_NOTIFY_SEVERITY | 13c Notification Payload |
|---|---|
|
TARGET_NAME |
gc$notif_target.target_name |
|
TARGET_TYPE |
gc$notif_target.target_type |
|
TIMEZONE |
gc$notif_target.target_timezone |
|
HOST_NAME |
gc$notif_target.host_name |
|
MERTIC_NAME |
gc$notif_event_attr.value where its name=' metric_group' in gc$notif_event_attr_array. |
|
METRIC_DESCRIPTION |
gc$notif_event_attr.value where its name=' metric_description' in gc$notif_event_attr_array. |
|
METRIC_COLUMN |
gc$notif_event_attr.value where its name=' metric_column' in gc$notif_event_attr_array. |
|
METRIC_VALUE |
gc$notif_event_attr.value where its name=' value' in gc$notif_event_attr_array. |
|
KEY_VALUE |
It is applied for multiple keys based metric when value of gc$notif_event_attr.name='num_keys' is not null and is greater than 0 in gc$notif_event_attr_array. See detail descriptions below. |
|
KEY_VALUE_NAME |
It is applied for multiple keys based metric when value of gc$notif_event_attr.name='num_keys' is not null and is greater than 0 in gc$notif_event_attr_array. See detail descriptions below. |
|
KEY_VALUE_GUID |
gc$notif_event_attr.value where its name='key_ value' in gc$notif_event_attr_array. |
|
CTXT_LIST |
gc$notif_event_context_array |
|
COLLECTION_TIMESTAMP |
gc$notif_event_payload. reported_date |
|
SEVERITY_CODE |
Derive from gc$notif_event_payload.severity_code, see Table 6-9. |
|
MESSAGE |
gc$notif_msg_info.message |
|
SEVERITY_GUID |
gc$notif_event_attr.value where its name=' severity_guid' in gc$notif_event_attr_array. |
|
METRIC_GUID |
gc$notif_event_attr.value where its name=' metric_guid' in gc$notif_event_attr_array. |
|
TARGET_GUID |
gc$notif_target.target_guid |
|
RULE_OWNER |
gc$notif_msg_info.rule_owner |
|
RULE_NAME |
gc$notif_msg_info.ruleset_name |
The following example illustrates how to obtain similar pre-12c KEY_VALUE and KEY_VALUE_NAME from an Enterprise Manager 13c notification payload.
Example 6-14 Extracting KEY_VALUE and KEY_VALUE_NAME
-- Get the pre-12c KEY_VALUE and KEY_VALUE_NAME from an Enterprise Manager 13c
-- notification payload
-- parameters
-- IN Parameters:
-- event_msg : The event notification payload
-- OUT Parameters
-- key_value_name_out : the KEY_VALUE_NAME backward compitable to pre-12c
-- notification payload
-- key_value_out : the KEY_VALUE backward compitable to pre-12c
-- notification payload
--
CREATE OR REPLACE PROCEDURE get_pre_12c_key_value(
event_msg IN GC$NOTIF_EVENT_MSG,
key_value_name_out OUT VARCHAR2,
key_value_out OUT VARCHAR2)
IS
l_key_columns MGMT_SHORT_STRING_ARRAY := MGMT_SHORT_STRING_ARRAY();
l_key_column_values MGMT_MEDIUM_STRING_ARRAY := MGMT_MEDIUM_STRING_ARRAY();
l_key_value VARCHAR2(1790) := NULL;
l_num_keys NUMBER := 0;
l_attrs gc$notif_event_attr_array;
l_key_value_name VARCHAR2(512);
BEGIN
l_attrs := event_msg.event_payload.event_attrs;
key_value_name_out := NULL;
key_value_out := NULL;
IF l_attrs IS NOT NULL AND
l_attrs.COUNT > 0
THEN
l_key_columns.extend(7);
l_key_column_values.extend(7);
FOR c IN 1..l_attrs.COUNT
LOOP
CASE l_attrs(c).name
WHEN 'num_keys' THEN
BEGIN
l_num_keys := to_number(l_attrs(c).value);
EXCEPTION
WHEN OTHERS THEN
-- should never happen, but guard against it l_num_keys := 0;
END;
WHEN 'key_value' THEN
l_key_value := substr(l_attrs(c).nls_value,1,1290);
WHEN 'key_column_1' THEN
l_key_columns(1) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_2' THEN
l_key_columns(2) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_3' THEN
l_key_columns(3) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_4' THEN
l_key_columns(4) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_5' THEN
l_key_columns(5) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_6' THEN
l_key_columns(6) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_7' THEN
l_key_columns(7) := substr(l_attrs(c).nls_value,1,64);
WHEN 'key_column_1_value' THEN
l_key_column_values(1) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_2_value' THEN
l_key_column_values(2) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_3_value' THEN
l_key_column_values(3) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_4_value' THEN
l_key_column_values(4) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_5_value' THEN
l_key_column_values(5) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_6_value' THEN
l_key_column_values(6) := substr(l_attrs(c).nls_value,1,256);
WHEN 'key_column_7_value' THEN
l_key_column_values(7) := substr(l_attrs(c).nls_value,1,256);
ELSE
NULL;
END CASE;
END LOOP;
-- get key_value and key_value_name when l_num_keys > 0
IF l_num_keys > 0
THEN
-- get key value name
IF l_key_columns IS NULL OR l_key_columns.COUNT = 0
THEN
key_value_name_out := NULL;
ELSE
l_key_value_name := NULL;
FOR i in l_key_columns.FIRST..l_num_keys
LOOP
IF i > 1
THEN
l_key_value_name := l_key_value_name || ';';
END IF;
l_key_value_name := l_key_value_name || l_key_columns(i);
END LOOP;
key_value_name_out := l_key_value_name;
END IF;
-- get key_value
IF l_num_keys = 1
THEN
key_value_out := l_key_value;
ELSE
l_key_value := NULL;
IF l_key_column_values IS NULL OR l_key_column_values.COUNT = 0
THEN
key_value_out := NULL;
ELSE
FOR i in l_key_column_values.FIRST..l_num_keys
LOOP
IF i > 1
THEN
l_key_value := l_key_value || ';';
END IF;
l_key_value := l_key_value || l_key_column_values(i);
END LOOP;
-- max length for key value in pre-12c = 1290
key_value_out := substr(l_key_value,1,1290);
END IF;
END IF;
END IF; -- l_num_keys > 0
END IF; -- l_attrs IS NOT NULL
END get_pre_12c_key_value;
/When the event type is metric_alert:
Use the following severity code mapping from 13c to pre-12c when the event type is metric_alert.
Table 6-9 Severity Code Mapping
| 13c Severity Code | Pre-12c Severity Code |
|---|---|
|
GC_EVENT_RECEIVER.FATAL 32 |
MGMT_GLOBAL.G_SEVERITY_CRITICAL 25 |
|
GC_EVENT_RECEIVER.CRITICAL 16 |
MGMT_GLOBAL.G_SEVERITY_CRITICAL 25 |
|
GC_EVENT_RECEIVER.WARNING 8 |
MGMT_GLOBAL.G_SEVERITY_WARNING 20 |
|
GC_EVENT_RECEIVER.CLEAR 0 |
MGMT_GLOBAL.G_SEVERITY_CLEAR 15 |
When event type is target_availability:
Use the following map when gc$notif_event_payload .event_type='target_availability'.
Table 6-10 Target Availability Mapping
| MGMT_NOTIFY_SEVERITY | 13c Notification Payload |
|---|---|
|
TARGET_NAME |
gc$notif_target.target_name |
|
TARGET_TYPE |
gc$notif_target.target_type |
|
TIMEZONE |
gc$notif_target.target_timezone |
|
HOST_NAME |
gc$notif_target.host_name |
|
MERTIC_NAME |
Use fixed value "Response". |
|
METRIC_DESCRIPTION |
NULL |
|
METRIC_COLUMN |
Use fixed value "Status". |
|
METRIC_VALUE |
gc$notif_event_attr.value where its name='target_status' in gc$notif_event_attr_array. |
|
KEY_VALUE |
NULL |
|
KEY_VALUE_NAME |
NULL |
|
KEY_VALUE_GUID |
NULL |
|
CTXT_LIST |
gc$notif_event_context_array |
|
COLLECTION_TIMESTAMP |
gc$notif_event_payload. reported_date |
|
SEVERITY_CODE |
gc$notif_event_attr.value where its name=' avail_severity' in gc$notif_event_attr_array. |
|
MESSAGE |
gc$notif_msg_info.message |
|
SEVERITY_GUID |
gc$notif_event_attr.value where its name=' severity_guid' in gc$notif_event_attr_array. |
|
METRIC_GUID |
gc$notif_event_attr.value where its name=' metric_guid_id' in gc$notif_event_attr_array. |
|
TARGET_GUID |
gc$notif_target.target_guid |
|
RULE_OWNER |
gc$notif_msg_info.rule_owner |
|
RULE_NAME |
gc$notif_msg_info.ruleset_name |
Mapping for MGMT_NOTIFY_JOB
Use the following map when gc$notif_event_payload .event_type=job_status_change'.
Table 6-11 Job Status Change Mapping
| MGMT_NOTIFY_JOB | 13c Notification Payload |
|---|---|
|
JOB_NAME |
gc$notif_source.source_name |
|
JOB_OWNER |
gc$notif_source.source_owner |
|
JOB_TYPE |
gc$notif_source.source_sub_type |
|
JOB_STATUS |
gc$notif_event_attr.value where its name=' execution_status_code' in gc$notif_event_attr_array. |
|
STATE_CHANGE_GUID |
gc$notif_event_attr.value where its name=' state_change_guid' in gc$notif_event_attr_array. |
|
JOB_GUID |
gc$notif_source.source_guid |
|
EXECUTION_ID |
gc$notif_event_attr.value where its name=' execution_id' in gc$notif_event_attr_array. |
|
TARGETS |
gc$notif_target.target_name, gc$notif_target.target_type |
|
RULE_OWNER |
gc$notif_msg_info.rule_owner |
|
RULE_NAME |
gc$notif_msg_info.ruleset_name |
|
OCCURRED_DATE |
gc$notif_event_payload. reported_date |
Mapping for MGMT_NOTIFY_CORRECTIVE_ACTION
Note that corrective action related payload is populated when gc$notif_msg_info.notification_type is set to NOTIF_CA.
For mapping the following attributes, use the mapping information provided for MGMT_NOTIFY_SEVERITY object Table 6-8
MERTIC_NAME
METRIC_COLUMN
METRIC_VALUE
KEY_VALUE
KEY_VALUE_NAME
KEY_VALUE_GUID
CTXT_LIST
RULE_OWNER
RULE_NAME
OCCURRED_DATE
For mapping the job related attributes in MGMT_NOTIFY_CORRECTIVE_ACTION object, use the following map.
Table 6-12 Corrective Action Mapping
| MGMT_NOTIFY_CORRECTIVE_ACTION | 13c Notification Payload |
|---|---|
|
JOB_NAME |
gc$ notif_corrective_action_job.job_name |
|
JOB_OWNER |
gc$ notif_corrective_action_job.job_owner |
|
JOB_TYPE |
gc$ notif_corrective_action_job.job_type |
|
JOB_STATUS |
gc$ notif_corrective_action_job.status_code |
|
STATE_CHANGE_GUID |
gc$ notif_corrective_action_job. job_state_change_guid |
|
JOB_GUID |
gc$ notif_corrective_action_job. job _guid |
|
EXECUTION_ID |
gc$ notif_corrective_action_job. job_execution_guid |
|
OCCURRED_DATE |
gc$ notif_corrective_action_job.occurred_date |
|
TARGETS |
There can be at most one target. Use the values from gc$notif_target.target_name, gc$notif_target.target_type for the associated target. |
Sending SNMP Traps to Third Party Systems
Enterprise Manager supports integration with third-party management tools through the Simple Network Management Protocol (SNMP). For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.
Note:
In order for a third-party system to interpret traps sent by the OMS, the omstrap.v1 file must first be loaded into the third-party SNMP console. For more information about this file and its location, see "MIB Definition".
The Enterprise Manager 13c version of the MIB file incorporates the 10g and 11g MIB content, thus ensuring backward compatibility with earlier Enterprise Manager releases.
Enterprise Manager supports both SNMP Version 1 and Version 3 traps. The traps are described by the MIB definition shown in Enterprise Manager MIB Definition. See "Management Information Base (MIB)" for an explanation of how the MIB works. If you are using Enterprise Manager 13c, see Interpreting Variables of the Enterprise Manager MIB and Enterprise Manager MIB Definition. If you are upgrading from a pre-12c version of Enterprise Manager, see SNMP Trap Mappings for specific version mappings.
For Enterprise Manager 12c, SNMP traps are delivered for event notifications. Starting in Enterprise Manager 13.5 RU16, SNMP trap notifications are supported for both events and incidents. SNMP trap notifications are not supported for problems.
Note:
Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a incident rule.
SNMP Version 1 Versus SNMP Version 3
SNMP Version 3 shares the same basic architecture of Version 1, but adds numerous enhancements to SNMP administration and security. The primary enhancement relevant to Enterprise Manager involves additional security levels that provide both authentication and privacy as well as authorization and access control.
User-based Security Model (USM)
USM defines the security-related procedures followed by an SNMP engine when processing SNMP messages. Enterprise Manager SNMP V3 support takes advantage of this added SNMP message-level security enhancement to provide a secure messaging environment.
USM protects against two primary security threats:
-
Modification of information: The modification threat is the danger that some unauthorized entity may alter in-transit SNMP messages generated on behalf of an authorized principal in such a way as to effect unauthorized management operations, including falsifying the value of an object.
-
Masquerade: The masquerade threat is the danger that management operations not authorized for some user may be attempted by assuming the identity of another user that has the appropriate authorizations.
For both SNMP versions, the basic methodology for setting up Enterprise Manager advanced notifications using SNMP traps remains the same:
-
Define the notification method based on an SNMP trap.
-
Assign the notification method to an incident rule.
Working with SNMP V3 Trap Notification Methods
The procedure for defining an SNMP V3 trap notification method differs slightly from that of V1. Beginning with Enterprise Manager Release 12.1.0.4, a separate interface consolidates key information and configuration functionality pertaining to SNMP V3 trap notification methods. The SNMP V3 Trap interface helps guide you through the process of creating SNMP notification methods, enabling the OMS to send SNMP traps, and defining user security settings for SNMP trap notifications.
Configuring the OMS to Send SNMP Trap Notifications
Before creating an SNMP Trap notification method, you must enable at least one OMS is your environment to handle SNMP Trap notifications. For SNMP V3, the OMS serves as an SNMP Agent which sends traps to the SNMP Manager that is monitoring all SNMP Agents deployed in the network.
Creating/Editing an SNMP V3 Trap Notification Method
Once an OMS has been enabled to send SNMP traps notifications, the next step is to create a notification method than can be used by an incident rule.
-
From the Setup menu, select Notifications and then SNMP V3 Traps. The Getting Started page displays.
Note:
If want to edit an existing Notification Method, select the desired method from the Notification Methods region and click Edit.
-
Click the Configuration tab. The Configuration page displays.
-
From the Notification Methods region, click Create. The SNMPv3 Traps: Create Notification Method page displays.
-
Enter the requisite Notification Method definition parameters. Note: You can enable Repeat Notifications at this point.
-
If you choose to create a new User Security Model entry, from the User Security Model region, ensure the Create New option is chosen.
-
Specify a Username that uniquely identifies the credential. SNMP V3 allows multiple usernames to be set in an SNMP Agent as well as SNMP Manager applications.
-
Select a Security Level from the drop-down menu. Available parameters become available depending on the security level. There are three levels from which to choose:
AuthPriv (Authentication + Privacy:) The sender's identity must be confirmed by the receiver (authentication). SNMP V3 messages are encrypted by the sender and must be decrypted by the receiver (privacy).
AuthNoPriv (Authentication only): The receiver must authenticate the sender's identity before accepting the message.
NoAuthNoPriv (no security): Neither sender identity confirmation nor message encryption is used.
-
For AuthPriv and AuthNoPriv security levels, choose a the desired Authentication Protocol. Two authentication protocols are available:
Secure Hash Algorithm (SHA)
Message Digest algorithm (MD5)
The authentication protocols are used to build the message digest when the message is authenticated.
Privacy Protocol (used for the AuthPriv security level) is used to encrypt/decrypt messages. USM uses the Data Encryption Standard (DES). The Privacy Password is used in conjunction with the Privacy Protocol. the privacy password on both the SNMP Agent and SNMP Manager must match in order for encryption/decryption to succeed.
If you have already have predefined User Security Model entries, choose the Use Existing option and select one of the USM entries from the drop-down menu. USM entries are listed by username.
Note:
Ensure that the USM credentials are identical in OMS and the external trap receiver. If they do not match, Enterprise Manager will still send the SNMP trap, but the trap will not be received. If the USM credentials are invalid, Enterprise Manager will still send the SNMP trap, however, the trap will not be received as the incorrect credentials will result in an authentication error at the SNMP receiver. This type of authentication error will not be apparent from the Enterprise Manager console.
-
-
Once you have entered the requisite Notification Method and USM parameters, click Save. The newly created notification method appears in the Notification Method region of the Configuration page.
Note:
Once you have defined the SNMP V3 Trap notification method, you must add it to a rule. See Creating a Rule to Send SNMP Traps for Events to Third Party Systems or Creating a Rule to Send SNMP Traps for Incidents to Third Party Systems for instructions.
Editing a User Security Model Entry
You can add USM entries at any time.
- From the User Security Model Entries region, click Create. The User Security Model Entries dialog displays.
Viewing Available SNMP V3 Trap Notification Methods
To view available SNMP V3 Trap notification methods:
- From the Setup menu, select Notifications, and then SNMP V3 Traps.
- Click on the Configurations tab.
- The Notification Methods region displays existing SNMP V3 Trap notification methods.
Deleting an SNMP V3 Trap Notification Method
To delete available SNMP V3 Trap notification methods:
- From the Setup menu, select Notifications, and then SNMP V3 Traps.
- Click on the Configurations tab.
- From the Notification Methods region, select an existing SNMP V3 Trap notification method.
- Click Delete.
Creating an SNMP V1 Trap
Step 1: Define a new notification method based on an SNMP trap.
Log in to Enterprise Manager as a Super Administrator. From the Setup menu, select Notifications and then select Scripts and SNMPv1 Traps.
You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. As shown in, the SNMP host will receive your SNMP traps.
Figure 6-2 SNMP Trap Required Information
Note:
A Test SNMP Trap button exists for you to test your setup.
Metric severity information will be passed as a series of variables in the SNMP trap.
Step 2: Assign the notification method to a rule.
You can edit an existing rule (or create a new incident rule), then add an action to the rule that subscribes to the advanced notification method. For instructions on setting up events or incidents rules using SNMP traps, see Creating a Rule to Send SNMP Traps for Events to Third Party Systems or Creating a Rule to Send SNMP Traps for Incidents to Third Party Systems respectively.
Example SNMP Trap Implementation
In this scenario, you want to identify the unique issues from the SNMP traps that are sent. Keep in mind that all events that are related to the same issue are part of the same event sequence. Each event sequence has a unique identification number.
An event sequence is a sequence of related events that represent the life of a specific issue from the time it is detected and an event is raised to the time it is fixed and a corresponding clear event is generated. For example, a warning metric alert event is raised when the CPU utilization of a host crosses 80%. This starts the event sequence representing the issue CPU Utilization of the host is beyond normal level. Another critical event is raised for the same issue when the CPU utilization goes above 90% and the event is added to the same event sequence. After a period of time, the CPU utilization returns to a normal level and a clear event is raised. At this point, the issue is resolved and the event sequence is closed.
The SNMP trap sent for this scenario is shown in Example 6-15. Each piece of information is sent as a variable embedded in the SNMP Trap.
This following example illustrates how OIDs are used during the lifecycle of an event. Here, for one event (while the event is open), the event sequence OID remains the same even though the event severity changes.
The OID for the event sequence is:
1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90
The OID for the event severity code is:
1.3.6.1.4.1.111.15.3.1.1.6.1: CRITICAL
When the event clears, these OIDs show the same event sequence with a different severity code:
The OID for the event sequence is:
1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90
The OID for the event severity code is:
1.3.6.1.4.1.111.15.3.1.1.6.1: CLEAR
The length of the SNMP OID value is limited to 2560 bytes by default. Configure the emoms property oracle.sysman.core.notification.snmp.max_oid_length to change the default limit.
Example 6-15 SNMP Trap
**************V1 TRAP***[1]***************** Community : public Enterprise :1.3.6.1.4.1.111.15.2 Generic :6 Specific :3 TimeStamp :67809 Agent adress :10.240.36.109 1.3.6.1.4.1.111.15.3.1.1.2.1: NOTIF_NORMAL 1.3.6.1.4.1.111.15.3.1.1.3.1: CPU Utilization is 92.658%, crossed warning (80) or critical (90) threshold. 1.3.6.1.4.1.111.15.3.1.1.4.1: https://sampleserver.oracle.com:5416/em/redirect?pageType=sdk-core-event-console-detailEvent&issueID=C77AE9E578F00773E040F00A6D242F90 1.3.6.1.4.1.111.15.3.1.1.5.1: Critical 1.3.6.1.4.1.111.15.3.1.1.6.1: CRITICAL 1.3.6.1.4.1.111.15.3.1.1.7.1: 0 1.3.6.1.4.1.111.15.3.1.1.8.1: 1.3.6.1.4.1.111.15.3.1.1.9.1: 1.3.6.1.4.1.111.15.3.1.1.10.1: Aug 17, 2012 3:26:36 PM PDT 1.3.6.1.4.1.111.15.3.1.1.11.1: Capacity 1.3.6.1.4.1.111.15.3.1.1.12.1: Capacity 1.3.6.1.4.1.111.15.3.1.1.13.1: Metric Alert 1.3.6.1.4.1.111.15.3.1.1.14.1: Load:cpuUtil 1.3.6.1.4.1.111.15.3.1.1.15.1: 281 1.3.6.1.4.1.111.15.3.1.1.16.1: 1.3.6.1.4.1.111.15.3.1.1.17.1: No 1.3.6.1.4.1.111.15.3.1.1.18.1: New 1.3.6.1.4.1.111.15.3.1.1.19.1: None 1.3.6.1.4.1.111.15.3.1.1.20.1: 0 1.3.6.1.4.1.111.15.3.1.1.21.1: sampleserver.oracle.com 1.3.6.1.4.1.111.15.3.1.1.22.1: https://sampleserver.oracle.com:5416/em/redirect?pageType=TARGET_HOMEPAGE&targetName=sampleserver.oracle.com&targetType=host 1.3.6.1.4.1.111.15.3.1.1.23.1: Host 1.3.6.1.4.1.111.15.3.1.1.24.1: sampleserver.oracle.com 1.3.6.1.4.1.111.15.3.1.1.25.1: SYSMAN 1.3.6.1.4.1.111.15.3.1.1.26.1: 1.3.6.1.4.1.111.15.3.1.1.27.1: 5.8.0.0.0 1.3.6.1.4.1.111.15.3.1.1.28.1: Operating System=Linux, Platform=x86_64, 1.3.6.1.4.1.111.15.3.1.1.29.1: 1.3.6.1.4.1.111.15.3.1.1.30.1: 1.3.6.1.4.1.111.15.3.1.1.31.1: 1.3.6.1.4.1.111.15.3.1.1.32.1: 1.3.6.1.4.1.111.15.3.1.1.33.1: 1.3.6.1.4.1.111.15.3.1.1.34.1: 1.3.6.1.4.1.111.15.3.1.1.35.1: 1.3.6.1.4.1.111.15.3.1.1.36.1: 1.3.6.1.4.1.111.15.3.1.1.37.1: 1.3.6.1.4.1.111.15.3.1.1.38.1: 1.3.6.1.4.1.111.15.3.1.1.39.1: SnmpNotifRuleset 1.3.6.1.4.1.111.15.3.1.1.40.1: SnmpNotifRuleset,SnmpNotifEvent 1.3.6.1.4.1.111.15.3.1.1.41.1: SYSMAN 1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90 1.3.6.1.4.1.111.15.3.1.1.43.1: 1.3.6.1.4.1.111.15.3.1.1.44.1: 1.3.6.1.4.1.111.15.3.1.1.45.1: 1.3.6.1.4.1.111.15.3.1.1.46.1: CPU Utilization is 92.658%, crossed warning (80) or critical (90) threshold., Incident created by rule (Name = Incident management Ruleset for all targets, Incident creation Rule for metric alerts.; Owner = <SYSTEM>). 1.3.6.1.4.1.111.15.3.1.1.61.1: Metric GUID=0C71A1AFAC2D7199013837DA35522C08 1.3.6.1.4.1.111.15.3.1.1.62.1: Severity GUID=C77AE9E578EC0773E040F00A6D242F90 1.3.6.1.4.1.111.15.3.1.1.63.1: Cycle GUID=C77AE9E578EC0773E040F00A6D242F90 1.3.6.1.4.1.111.15.3.1.1.64.1: Collection Name=LoadLinux 1.3.6.1.4.1.111.15.3.1.1.65.1: Metric Group=Load 1.3.6.1.4.1.111.15.3.1.1.66.1: Metric=CPU Utilization (%) 1.3.6.1.4.1.111.15.3.1.1.67.1: Metric Description= 1.3.6.1.4.1.111.15.3.1.1.68.1: Metric value=92.658 1.3.6.1.4.1.111.15.3.1.1.69.1: Key Value= 1.3.6.1.4.1.111.15.3.1.1.70.1: 1.3.6.1.4.1.111.15.3.1.1.71.1: 1.3.6.1.4.1.111.15.3.1.1.72.1: 1.3.6.1.4.1.111.15.3.1.1.73.1: 1.3.6.1.4.1.111.15.3.1.1.74.1: 1.3.6.1.4.1.111.15.3.1.1.75.1: 1.3.6.1.4.1.111.15.3.1.1.76.1: 1.3.6.1.4.1.111.15.3.1.1.77.1: 1.3.6.1.4.1.111.15.3.1.1.78.1: 1.3.6.1.4.1.111.15.3.1.1.79.1: 1.3.6.1.4.1.111.15.3.1.1.80.1: 1.3.6.1.4.1.111.15.3.1.1.81.1: 1.3.6.1.4.1.111.15.3.1.1.82.1: 1.3.6.1.4.1.111.15.3.1.1.83.1: 1.3.6.1.4.1.111.15.3.1.1.84.1: Number of keys=0 1.3.6.1.4.1.111.15.3.1.1.85.1: **************END V1 TRAP******************
SNMP Traps: Moving from Previous Enterprise Manager Releases to 12c and Greater
Note:
When you upgrade from a pre-Enterprise Manager 12c release to 12c and greater, SNMP advanced notification methods defined using previous versions of Enterprise Manager (pre-12c) will continue to function without modification.
For Enterprise Manager 11g and earlier, there were two types of SNMP traps:
-
Alerts
-
Job Status
Beginning with Enterprise Manager 12c there is now a single, comprehensive SNMP trap type that covers all available event types such as metric alerts, target availability, compliance standard violations, or job status changes. For more information about pre-12c to 12c SNMP trap mappings, see SNMP Trap Mappings. Traps will conform to the older Enterprise Manager MIB definition. Hence, pre-Enterprise Manager 12c traps will continue to be sent. See SNMP Trap Mappings for more information.
Also, for Enterprise Manager 12c, size of SNMP trap has increased in order to accommodate all event types and provide more comprehensive information. By default, the maximum SNMP packet size is 5120 bytes. If the third party system has a limit in the size of SNMP trap it can receive, you can change the default size of SNMP trap that Enterprise Manager sends. To change the default packet size, set this emoms oracle.sysman.core.notification.snmp_packet_length parameter, and then bounce the OMS.
Note:
When limiting the SNMP trap packet size, Oracle recommends not setting the oracle.sysman.core.notification.snmp_packet_length parameter any lower than 3072 bytes (3K).
The Enterprise Manager 12c MIB includes all pre-Enterprise Manager 12c MIB definitions. Hence, if you have an Enterprise Manager 12c MIB in your third party system, you can receive SNMP traps from both pre-Enterprise Manager 12c as well as Enterprise Manager 12c sites. For detailed information on version mapping, see SNMP Trap Mappings.
Sending Notifications Using Webhooks and Slack
Starting with Enterprise Manager 13c Release 5 RU 15, using the EM Generic Webhook, you can send event and incident notifications to any application that accepts webhook requests and can parse payload specific content. The webhook payload format depends on the type of the event or incident. Users will have to provide the structure for the payload to map EM to the external software.
To access the Webhooks and Slack page navigate to the Setup drop-down menu, hover over Notifications and locate the Webhooks and Slack option.
To use Webhooks or Slack as a notification method, you must enable it through Incident Rules. When on the Add Actions page for an incident or an event rule, locate the Advanced Notifications section to enable your preferred notification method (e.g., Slack).
Webhooks
Using a Webhook, you can configure EM to send events or incidents data directly into an application of your choice.
Add a Webhook Notification Method:
-
Get the incoming Webhook URL for your application.
- Click Add Notification Method in the Webhooks and Slack page.
- Select from the drop-down list to what you want the notification method to apply (Incidents or Events), and click Webhook:
- Incidents
- Events
- Incidents
- Enter a name and description for your notification method.
-
Enter the application’s incoming Webhook URL, and optionally other connection details.
- Optionally, modify the Webhook Payload. You can see the available payload variables by opening the Payload variable list on the right.
- Click Create.
Slack
Using a Slack Webhook, you can configure EM to send events or incidents data directly into a Slack channel.
Add a Slack Notification Method:
-
Get the Slack channel’s incoming Webhook URL.
Note:
You must be the workspace owner to get the incoming webhook URL. - Click Add Notification Method in the Webhooks and Slack page.
- Select from the drop-down list to what you want the notification method to apply (Incidents or Events), and click Slack:
- Incidents
- Events
- Incidents
- Enter a name and description for your notification method.
-
Enter the Slack channel’s incoming Webhook URL, and optionally other connection details.
Note:
Make sure the Slack channel’s incoming Webhook URL is in the
https://hooks.slack.com/services/...format. - Optionally, modify the Slack Payload. You can see the available payload variables by opening the Payload variable list on the right.
- Click Create.
Management Information Base (MIB)
Enterprise Manager Cloud Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.
About MIBs
A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).
The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.
MIB Definition
You can find the SNMP MIB file at the following location:
OMS_HOME/network/doc/omstrap.v1
Note:
The omstrap.v1 file is compatible with both SNMP V1 and SNMP V3.
The file omstrap.v1 is the OMS MIB.
For more information, see Interpreting Variables of the Enterprise Manager MIB.
A hardcopy version of omstrap.v1 can be found in Enterprise Manager MIB Definition.
The length of the SNMP OID value is limited to 2560 bytes by default. Configure emoms property oracle.sysman.core.notification.snmp.max_oid_length to change the default limit.
For Enterprise Manager 12c, SNMP traps are delivered for event notifications. Starting in Enterprise Manager 13.5 RU16, SNMP trap notifications are supported for both events and incidents. SNMP trap notifications are not supported for problems.
Note:
SNMP advanced notification methods defined using previous versions of Enterprise Manager (pre-12c) will continue to function without modification. Traps will conform to the older Enterprise Manager MIB definition.
Reading the MIB Variable Descriptions
This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 1, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.
Variable Name
- Syntax
-
Maps to the SYNTAX element of SNMP MIB definition, Version 1.
- Max-Access
-
Maps to the MAX-ACCESS element of SNMP MIB definition, Version 1.
- Status
-
Maps to the STATUS element of SNMP MIB definition, Version 1.
- Explanation
-
Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 1.
- Typical Range
-
Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.
- Significance
-
Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.
- Related Variables
-
Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.
- Suggested Presentation
-
Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.
Passing Corrective Action Status Change Information
Passing corrective action status change attributes (such as new status, job name, job type, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case, you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.
Passing Corrective Action Execution Status to an OS Command or Script
The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:
-
UNIX: $ENV_VARIABLE
-
MS Windows: %ENV_VARIABLE%
The notification system sets the following environment variables before calling the script. The notification system will set the environment variable $NOTIF_TYPE = NOTIF_CA for Corrective Action Execution. The script can then use any or all of these variables within the logic of the script.
Following table lists the environment variables for corrective action, they are populated when a corrective action is completed for an event.
Table 6-13 Corrective Action Environment Variables
| Environment Variable | Description |
|---|---|
|
CA_JOB_STATUS |
Corrective action job execution status. |
|
CA_JOB_NAME |
Name of the corrective action. |
|
CA_JOB_OWNER |
Owner of corrective action. |
|
CA_JOB_STEP_OUTPUT |
The value will be the text output from the corrective action execution. |
|
CA_JOB_TYPE |
Corrective action job type |
Passing Corrective Action Execution Status to a PLSQL Procedure
The notification system passes corrective action status change information to PL/SQL procedure - PROCEDURE p(event_msg IN gc$notif_event_msg). The instance gc$notif_corrective_action_job object is defined in event_msg.event_payload. corrective_action if event_msg. msg_info. notification_type is equal to GC$NOTIFICATIONNOTIF_CA. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the incident rule and passes the populated object to the procedure. The procedure is then able to access the fields of the object that has been passed to it. See Table 6-45 for details.
The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.
Table 6-14 Corrective Action Status Codes
| Name | Datatype | Value |
|---|---|---|
|
SCHEDULED_STATUS |
NUMBER(2) |
1 |
|
EXECUTING_STATUS |
NUMBER(2) |
2 |
|
ABORTED_STATUS |
NUMBER(2) |
3 |
|
FAILED_STATUS |
NUMBER(2) |
4 |
|
COMPLETED_STATUS |
NUMBER(2) |
5 |
|
SUSPENDED_STATUS |
NUMBER(2) |
6 |
|
AGENTDOWN_STATUS |
NUMBER(2) |
7 |
|
STOPPED_STATUS |
NUMBER(2) |
8 |
|
SUSPENDED_LOCK_STATUS |
NUMBER(2) |
9 |
|
SUSPENDED_EVENT_STATUS |
NUMBER(2) |
10 |
|
SUSPENDED_BLACKOUT_STATUS |
NUMBER(2) |
11 |
|
STOP_PENDING_STATUS |
NUMBER(2) |
12 |
|
SUSPEND_PENDING_STATUS |
NUMBER(2) |
13 |
|
INACTIVE_STATUS |
NUMBER(2) |
14 |
|
QUEUED_STATUS |
NUMBER(2) |
15 |
|
FAILED_RETRIED_STATUS |
NUMBER(2) |
16 |
|
WAITING_STATUS |
NUMBER(2) |
17 |
|
SKIPPED_STATUS |
NUMBER(2) |
18 |
|
REASSIGNED_STATUS |
NUMBER(2) |
20 |
Passing Job Execution Status Information
Passing job status change attributes (such as new status, job name, job type, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem. The job execution status information is one of event type - job_status_change event, and its content is in OS command and PL/SQL payload as described in Sending Notifications Using OS Commands and Scripts and Sending Notifications Using PL/SQL Procedures.
Passing Job Execution Status to a PL/SQL Procedure
The notification system passes job status change information to a PL/SQL procedure via the event_msg.event_payload object where event_type is equal to job_status_change. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL p(event_msg IN gc$notif_event_msg) procedure associated with the incident rule and passes the populated object to the procedure. The procedure is then able to access the fields of the event_msg.event_payload object that has been passed to it.
Table 6-15 lists all corrective action status change attributes that can be passed:
Table 6-15 Job Status Attributes
| Attribute | Datatype | Additional Information |
|---|---|---|
|
event_msg.event_payload.source.source_name |
VARCHAR2(128) |
The job name. |
|
event_msg.event_payload.source.source_owner |
VARCHAR2(256) |
The owner of the job. |
|
event_msg.event_payload.source.source_sub_type |
VARCHAR2(32) |
The type of the job. |
|
event_msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status' |
NUMBER |
The new status of the job. |
|
event_msg.event_payload. event_attrs(i).value where event_attrs(i).name='state_change_guid' |
RAW(16) |
The GUID of the state change record. |
|
event_msg.event_payload.source.source_guid |
RAW(16) |
The unique id of the job. |
|
event_msg target.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_id' |
RAW(16) |
The unique id of the execution. |
|
event_msg.event_payload.target |
gc$notif_target |
Target Information object.. |
|
event_msg.msg_info.rule_owner |
VARCHAR2(64) |
The name of the notification rule that cause the notification to be sent. |
|
event_msg.msg_info.rule_name |
VARCHAR2(132) |
The owner of the notification rule that cause the notification to be sent. |
|
event_msg.event_payload. reported_date |
DATE |
The time and date when the status change happened. |
When a job status change occurs for the job, the notification system creates an instance of the event_msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status' object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the event_msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status' object.
Table 6-16 Job Status Codes
| Name | Datatype | Value |
|---|---|---|
|
SCHEDULED_STATUS |
NUMBER(2) |
1 |
|
EXECUTING_STATUS |
NUMBER(2) |
2 |
|
ABORTED_STATUS |
NUMBER(2) |
3 |
|
FAILED_STATUS |
NUMBER(2) |
4 |
|
COMPLETED_STATUS |
NUMBER(2) |
5 |
|
SUSPENDED_STATUS |
NUMBER(2) |
6 |
|
AGENTDOWN_STATUS |
NUMBER(2) |
7 |
|
STOPPED_STATUS |
NUMBER(2) |
8 |
|
SUSPENDED_LOCK_STATUS |
NUMBER(2) |
9 |
|
SUSPENDED_EVENT_STATUS |
NUMBER(2) |
10 |
|
SUSPENDED_BLACKOUT_STATUS |
NUMBER(2) |
11 |
|
STOP_PENDING_STATUS |
NUMBER(2) |
12 |
|
SUSPEND_PENDING_STATUS |
NUMBER(2) |
13 |
|
INACTIVE_STATUS |
NUMBER(2) |
14 |
|
QUEUED_STATUS |
NUMBER(2) |
15 |
|
FAILED_RETRIED_STATUS |
NUMBER(2) |
16 |
|
WAITING_STATUS |
NUMBER(2) |
17 |
|
SKIPPED_STATUS |
NUMBER(2) |
18 |
|
REASSIGNED_STATUS |
NUMBER(2) |
20 |
Example 6-16 PL/SQL Procedure Using a Status Code (Job)
CREATE TABLE job_log (jobid RAW(16), status_code NUMBER(2), occured DATE);
CREATE OR REPLACE PROCEDURE LOG_JOB_STATUS_CHANGE(event_msg IN GC$NOTIF_EVENT_MSG)
IS
l_attrs gc$notif_event_attr_array;
exec_status_code NUMBER(2) := NULL;
occured_date DATE := NULL;
job_guid RAW(16) := NULL;
BEGIN
IF event_msg.event_payload.event_type = 'job_status_change'
THEN
l_attrs := event_msg.event_payload.event_attrs;
IF l_attrs IS NOT NULL
THEN
FOR i IN 1..l_attrs.COUNT
LOOP
IF l_attrs(i).name = 'exec_status_code'
THEN
exec_status_code := TO_NUMBER(l_attrs(i).value);
END IF;
END LOOP;
END IF;
occured_date := event_msg.event_payload.reported_date;
job_guid := event_msg.event_payload.source.source_guid;
-- Log all jobs' status
BEGIN
INSERT INTO job_log (jobid, status_code, occured)
VALUES (job_guid, exec_status_code, occured_date);
EXCEPTION
WHEN OTHERS
THEN
-- If there are any problems then get the notification retried
RAISE_APPLICATION_ERROR(-20000, 'Please retry');
END;
COMMIT;
ELSE
null; -- it is not a job_status_change event, ignore
END IF;
END LOG_JOB_STATUS_CHANGE;
/Passing Job Execution Status to an OS Command or Script
The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:
-
UNIX: $ENV_VARIABLE
-
MS Windows: %ENV_VARIABLE%
The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.
Table 6-17 Environment Variables
| Environment Variable | Description |
|---|---|
|
SOURCE_OBJ_NAME |
The name of the job. |
|
SOURCE_OBJ_OWNE |
The owner of the job. |
|
SOURCE_OBJ_SUB_TYPE |
The type of job. |
|
EXEC_STATUS_CODE |
The job status. |
|
EVENT_REPORTED_TIME |
Time when the severity occurred. |
|
TARGET_NAME |
The name of the target. |
|
TARGET_TYPE |
The type of the target. |
|
RULE_NAME |
Name of the notification rule that resulted in the severity. |
|
RULE_OWNER |
Name of the Enterprise Manager administrator who owns the notification rule. |
Passing User-Defined Target Properties to Notification Methods
Enterprise Manager allows you to define target properties (accessed from the target home page) that can be used to store environmental or usage context information specific to that target. Target property values are passed to custom notification methods where they can be processed using conditional logic or simply passed as additional alert information to third-party devices, such as ticketing systems. By default, Enterprise Manager passes all defined target properties to notification methods.
Note:
Target properties are not passed to notification methods when short email format is used.
Notification Reference
This section contains the following reference material:
EMOMS Properties
EMOMS properties can be used for controlling the size and format of the short email. The following table lists emoms properties for Notification System.
Table 6-18 emoms Properties for Notifications
| Property Name | Default Value | Description |
|---|---|---|
|
oracle.sysman.core.notification.emails_per_minute |
250 |
Email delivery limits per minute. The Notification system uses this value to throttle number of Email delivery per minutes. Customer should set the value lower if doesn't want to over flow the Email server, or set the value higher if the Email server can handle high volume of Emails. |
|
oracle.sysman.core.notification.cmds_per_minute |
100 |
OS Command delivery limits per minute. The Notification system uses this value to throttle number of OS Command delivery per minutes. |
|
oracle.sysman.core.notification.os_cmd_timeout |
30 |
OS Command delivery timeout in seconds. This value indicates how long to allow OS process to execute the OS Command delivery. Set this value higher if the OS command script requires longer time to complete execution. |
|
oracle.sysman.core.notification.plsql_per_minute |
250 |
PL/SQL delivery limits per minute. The Notification system uses this value to throttle number of PL/SQL delivery per minutes. |
|
em.notification.java_per_minute |
500 |
JAVA delivery limits per minute. The Notification system uses this value to throttle number of Java delivery per minutes. |
|
em.notification.ticket_per_minute |
250 |
Ticket delivery limits per minute. The Notification system uses this value to throttle number of Ticket delivery per minutes. |
|
oracle.sysman.core.notification.traps_per_minute |
250 |
SNMP delivery limits per minute. The Notification system uses this value to control the number of SNMP trap per minutes. |
|
oracle.sysman.core.notification.locale.plsql |
OMS Locale |
This property specifies the Locale delivered by advanced PL/SQL notification. The customer can define this property to overwrite the default Locale where the OMS is installed. Valid Locales:
|
|
oracle.sysman.core.notification.locale.email |
OMS Locale |
This property specifies the Locale delivered by Email. Customer can define this property to overwrite the default Locale where the OMS is installed. Valid Locales:
|
|
oracle.sysman.core.notification.locale.oscmd |
OMS Locale |
This property specifies the Locale delivered by OS Command. Customer can define this property to overwrite the default Locale where the OMS is installed. Valid Locales:
|
|
oracle.sysman.core.notification.locale.snmp |
OMS Locale |
This property specifies the Locale delivered by SNMP trap. Customer can define this property to overwrite the default Locale where the OMS is installed. Valid Locales:
|
|
oracle.sysman.core.notification.oscmd.max_env_var_length |
512 |
The maximum length of OS Common environment variable value. |
|
oracle.sysman.core.notification.snmp.max_oid_length |
2560 |
The maximum length of SNMP OID value. |
|
oracle.sysman.core.notification.min_delivery_threads |
6 |
The minimum number of active threads in the thread pool initially and number of active threads are running when system is in low activities. Setting the value higher will use more system resources, but will deliver more notifications. |
|
oracle.sysman.core.notification.max_delivery_threads |
24 |
The maximum number of active threads in the thread pool when the system is in the high activities. This value should greater than em.notification.min_delivery_threads. Setting the value higher will use more system resources and deliver more notifications. |
|
oracle.sysman.core.notification.short_format_length |
>=1 (155) |
The size limit of the total number of characters in short email format. The customer should modify this property value to fit their email or pager limit content size. The email subject is restricted to a maximum of 80 characters for short email format notifications. |
|
oracle.sysman.core.notification.snmp_packet_length |
<=1 (5120) |
The maximum size of SNMP Protocol Data unit. |
|
oracle.sysman.core.notification.email_content_transfer_encoding |
8-bit, 7-bit(QP), 7-bit(BASE64) (8-bit) |
The character set that can encode the Email. Oracle supports three character sets : 8-bit, 7-bit(QP), and7-bit(BASE64). |
|
oracle.sysman.core.notification.emails_per_connection |
>=1 (20) |
The maximum number of emails delivered to same email gateway before switching to the next available email gateway (assumes customers have configured multiple email gateways). This property is used for email gateway load balance. |
|
oracle.sysman.core.notification.short_format |
both, subject, body (both) |
Use short format on both subject and body, subject only, or body only.. |
|
oracle.sysman.core.notification.send_prior_status_after_agent_unreachable_clears |
True |
By default , a notification is sent indicating a target's status whenever the monitoring Agent comes out of unreachable status, even if the target's status has not changed. Use this emoms property to enable (True)/disable (False) the duplicate target status notification. To disable duplicate target status notifications, set this property to False:
To enable duplicate target status notifications, set the property to True.
|
You must establish the maximum size your device can support and whether the message is sent in subject, body or both.
You can modify the emoms properties by using the Enterprise Manager command line control emctl get/set/delete/list property command.
Note:
The following commands require an OMS restart in order for the changes to take place.
Get Property Command
emctl get [-sysman_pwd "sysman password"]-name oracle.sysman.core.notification.short_format_length
Set Property Command
emctl set property -name oracle.sysman.core.notification.short_format_length -value 155
Emoms Properties Entries for a Short Email Format
emctl set property -name oracle.sysman.core.notification.short_format_length -value 155 emctl set property -name oracle.sysman.core.notification.short_format -value both
Passing Event, Incident, Problem Information to an OS Command or Script
The notification system passes information to an OS script or executable using system environment variables.
Conventions used to access environmental variables vary depending on the operating system:
-
UNIX: $ENV_VARIABLE
-
Windows:%ENV_VARIABLE%
The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.
Environment Variables Common to Event, Incident and Problem
Table 6-19 Generic Environment Variables
| Environment Variable | Description |
|---|---|
|
NOTIF_TYPE |
Type of notification and possible values NOTIF_NORMAL, NOTIF_RETRY, NOTIF_DURATION, NOTIF_REPEAT, NOTIF_CA, NOTIF_RCA |
|
REPEAT_COUNT |
How many times the notification has been sent out before this notification. |
|
RULESET_NAME |
The name of the ruleset that triggered this notification. |
|
RULE_NAME |
The name of the rule that triggered this notification. |
|
RULE_OWNER |
The owner of the ruleset that triggered this notification. |
|
MESSAGE |
The message of the event, incident, or problem. |
|
MESSAGE_URL |
EM console URL for this message. |
Table 6-20 Category-Related Environment Variables
| Environment Variable | Description |
|---|---|
|
CATEGORIES_COUNT |
Number of categories in this notification. This value is equal to1 if one category is associated with event, incident or problem. It is equal to 0 if no category associated with event, incident or problem. |
|
CATEGORY_CODES_COUNT |
Number of category codes in this notification. |
|
CATEGORY_n |
Category is translated based on locale defined in OMS server. Valid values for the suffix "_n" are between 1.. $CATEGORIES_COUNT |
|
CATEGORY_CODE_n |
Codes for the categories. Valid values for the suffix "_n" are between 1..$CATEGORY_CODES_COUNT |
Table 6-21 lists the common environment variables for User Defined Target Properties. They will be populated under the following cases: (a) When an event has a related target, (b) When an incident or a problem have single event source and have a related target.
Table 6-21 User-Defined Target Property Environment Variables
| Environment Variable | Description |
|---|---|
|
ORCL_GTP_COMMENT |
Comment |
|
ORCL_GTP_CONTACT |
Contact |
|
ORCL_GTP_COST_CENTER |
Cost Center |
|
ORCL_GTP_DEPARTMENT |
Department |
|
ORCL_GTP_DEPLOYMENT_TYPE |
Deployment type |
|
ORCL_GTP_LINE_OF_BUS |
Line of Business |
|
ORCL_GTP_LOCATION |
Location |
Event Notification-Specific Environment Variables
Table 6-22 Event Notification-Specific Environment Variables
| Environment Variable | Description |
|---|---|
|
EVENT_NAME |
Event Name. |
|
EVENT_REPORTED_TIME |
Event reported date. |
|
EVENT_SOURCE_COUNT |
Number of Sources associated with this event. |
|
EVENT_TYPE |
Event type. |
|
EVENT_OCCURRENCE_TIME |
Event occurrence time. |
|
EVENT_TYPE_ATTRS |
The list of event type specific attributes. |
|
EVENT_CONTEXT_ATTRS |
Event context data. |
|
LAST_UPDATED_TIME |
Last updated time |
|
SEQUENCE_ID |
The unique event sequence identifier. An event sequence may consist of one or more events. All events in this sequence have the same event sequence ID. |
|
SEVERITY |
Severity of event, it is translated. |
|
SEVERITY_CODE |
Code for event severity. Possible values are the following. FATAL, CRITICAL, WARNING, MINOR_WARNING, INFORMATIONAL, and CLEAR |
|
ACTION_MSG |
Message describing the action to take for resolving the event. |
|
TOTAL_OCCURRENCE_COUNT |
Total number of duplicate occurrences |
|
RCA_DETAILS |
If RCA is associated with this events. |
|
CURRENT_OCCURRENCE_COUNT |
Total number of occurrences of the event in the current collection period. This attribute only applies to de-duplicated events. |
|
CURRENT_FIRST_OCCUR_DATE |
Time stamp when the event first occurred in the current collection period. This attribute only applies to de-duplicated events. |
|
CURRENT_LAST_OCCUR_DATE_DESC |
Time stamp when the e vent last occurred in the current collection period. This attribute only applies to de-duplicated events. |
Table 6-23 lists the environment variables for the incident associated with an event. They are populated when the event is associated with an incident.
Table 6-23 Associated Incident Environment Variables
| Environment Variable | Description |
|---|---|
|
ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER |
Set to yes, if associated incident was acknowledged by owner |
|
ASSOC_INCIDENT_ACKNOWLEDGED_DETAILS |
The details of associated incident acknowledgement. For example:No - if not acknowledged Yes By userName - if acknowledged |
|
ASSOC_INCIDENT_STATUS |
Associated Incident Status |
|
ASSOC_INCIDENT_ID |
Associated Incident ID |
|
ASSOC_INCIDENT_PRIORITY |
Associated Incident priority. Supported value are Urgent, Very High, High, Medium,Low, None. |
|
ASSOC_INCIDENT_OWNER |
Associated Incident Owner if it is existed. |
|
ASSOC_INCIDENT_ESCALATION_LEVEL |
Escalation level of the associated incident has a value between 0 to 5. |
Table 6-24 lists the common environment variables related to the Source Object. They are populated when $SOURCE_OBJ_TYPE is not TARGET.
Table 6-24 Source Object-Related Environment Variables
| Environment Variable | Description |
|---|---|
|
SOURCE_OBJ_TYPE |
Type of the Source object. For example, JOB, TEMPLATE. |
|
SOURCE_OBJ_NAME |
Source Object Name. |
|
SOURCE_OBJ_NAME_URL |
Source's event console URL. |
|
SOURCE_OBJ_SUB_TYPE |
Sub-type of the Source object. For example, it provides the underlying job type for job status change events. |
|
SOURCE_OBJ_OWNER |
Owner of the Source object. |
Table 6-25 lists the common environment variables for the target, associated with the given issue. They are populated when the issue is related to a target.
Table 6-25 Target-Related Environment Variables
| Environment Variable | Description |
|---|---|
|
TARGET_NAME |
Name of Target |
|
TARGET_TYPE |
Type of Target |
|
TARGET_OWNER |
Owner of Target |
|
HOST_NAME |
The name of the host on which the target is deployed upon. |
|
TARGET_URL |
Target's Enterprise Manager Console URL. |
|
TARGET_LIFECYCLE_STATUS |
Life Cycle Status of the target. Possible values: Production, Mission Critical, Stage, Test, and Development. It is null if not defined. |
|
TARGET_VERSION |
Target Version of the target |
Environment Variables Specific to Event Types
Events are classified into multiple types. For example, the mertc_alert event type is used for modeling metric alerts. You can use SQL queries to list the event types in your deployment as well as their event-specific payload. The following SQL example can be used to list all internal event type names that are registered in Enterprise Manager.
Select event_class as event_type, upper(name) as env_var_name from em_event_class_attrs where notif_order != 0 and event_class is not null union select event_class as event_type, upper(name) || '_NLS' as env_var_name from em_event_class_attrs where notif_order != 0 and event_class is not null and is_translated = 1 order by event_type, env_var_name;
The environment variable payload specific to each event type can be accessed via the OS scripts. The following tables list notification attributes for the most critical event types.
Table 6-26 Environment Variables Specific to Metric Alert Event Type
| Environment Variable | Description |
|---|---|
|
COLL_NAME |
The name of the collection collecting the metric. |
|
COLL_NAME_NLS |
The translated name of the collection collecting the metric |
|
KEY_COLUMN_X |
Internal name of Key Column X where X is a number between 1 and 7. |
|
KEY_COLUMN_X_NLS |
Translated name of Key Column X where X is a number between 1 and 7. |
|
KEY_COLUMN_X_VALUE |
Value of Key Column X where X is a number between 1 and 7. |
|
KEY_VALUE |
Monitored object for the metric corresponding to the Metric Alert event. |
|
METRIC_COLUMN |
The name of the metric column |
|
METRIC_COLUMN_NLS |
The translated name of the metric column. |
|
METRIC_DESCRIPTION |
Brief description of the metric. |
|
METRIC_DESCRIPTION_NLS |
Translated brief description of the metric. |
|
METRIC_GROUP |
The name of the metric. |
|
METRIC_GROUP_NLS |
The translated name of the metric |
|
NUM_KEYS |
The number of key metric columns in the metric. |
|
SEVERITY_GUID |
The GUID of the severity record associated with this metric alert. |
|
CYCLE_GUID |
A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear. |
|
VALUE |
Value of the metric when the event triggered. |
Table 6-27 Environment variables specific to Target Availability Event Type
| Environment Variable | Description |
|---|---|
|
AVAIL_SEVERITY |
The transition severity that resulted in the status of the target to change to the current availability status. Possible Values for AVAIL_SEVERITY
|
|
AVAIL_SUB_STATE |
The substatus of a target for the current status. |
|
CYCLE_GUID |
A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear. |
|
METRIC_GUID |
Metric GUID of response metric. |
|
SEVERITY_GUID |
The GUID of the severity record associated with this availability status. |
|
TARGET_STATUS |
The current availability status of the target. |
|
TARGET_STATUS_NLS |
The translated current availability status of the target. |
Table 6-28 Environment variables specific to Job Status Change event type
| Environment Variable | Description |
|---|---|
|
EXECUTION_ID |
Unique ID of the job execution.. |
|
EXECUTION_LOG |
The job output of the last step executed. |
|
EXECUTION_STATUS |
The internal status of the job execution. |
|
EXECUTION_STATUS_NLS |
The translated status of the job execution. |
|
EXEC_STATUS_CODE |
Execution status code of job execution. For possible values, see Table 6-16. |
|
STATE_CHANGE_GUID |
Unique ID of last status change |
You can use SQL queries to list the deployed event types in your deployment and the payload specific to each one of them. The following SQL can be used to list all internal event type names which are registered in the Enterprise Manager.
select class_name as event_type_name from em_event_class;
Following SQL lists environment variables specific to metric_alert event type.
select env_var_name
from
( Select event_class as event_type, upper(name) as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
union
select event_class as event_type, upper(name) || '_NLS' as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
and is_translated = 1)
where event_type = 'metric_alert';
You can also obtain the description of notification attributes specific to an event type directly from the Enterprise Manager console:
-
From the Setup menu, select Notifications, then select Customize Email Formats.
-
Select the event type.
-
Click Customize.
-
Click Show Predefined Attributes.
Environment variables, ending with the suffix _NLS, provide the translated value for given attribute. For example, METRIC_COLUMN_NLS environment variable will provide the translated value for the metric column attribute. Translated values will be in the locale of the OMS.
Environment Variables Specific to Incident Notifications
Table 6-29 Incident-Specific Environment Variables
| Environment Variable | Description |
|---|---|
|
SEVERITY |
Incident Severity, it is translated. Possible Values: Fatal, Critical, Warning, Informational, Clear |
|
SEVERITY_CODE |
Code for Severity. Possible values are the FATAL, CRITICAL, WARNING, MINOR_WARNING, INFORMATIONAL, and CLEAR |
|
INCIDENT_REPORTED_TIME |
Incident reported time |
|
INCIDENT_ACKNOWLEDGED_BY_OWNER |
Set yes, if incident is acknowledged by owner. |
|
INCIDENT_ID |
Incident ID |
|
INCIDENT_OWNER |
Incident Owner |
|
ASSOC_EVENT_COUNT |
The number events associated with this incident. |
|
INCIDENT_STATUS |
Incident status. There are two internal fixed resolution status. NEW CLOSED Users can define additional statuses. |
|
ESCALATED |
Is Incident escalated |
|
ESCALATED_LEVEL |
The escalated level of incident. |
|
PRIORITY |
Incident priority. It is the translated priority name. Possible Values: Urgent, Very High, High, Medium, Low, None |
|
PRIOTITY_CODE |
Incident priority code It is the internal value defined in EM. PRIORITY_URGENT PRIORITY_VERY_HIGH PRIORITY_HIGH PRIORITY_MEDIUM PRIORITY_LOW PRIORITY_NONE |
|
TICKET_STATUS |
Status of external ticket, if it exists. |
|
TICKET_ID |
ID of external ticket, if it exists. |
|
LAST_UPDATED_TIME |
Incident last update time. |
|
ADR_INCIDENT_ID |
Automatic Diagnostic Reposito ry (ADR) Incident ID: A unique numeric identifier for the ADR Incident. An ADR I ncident is an occurrence of a Problem. |
|
ADR_IMPACT |
Impact of the Automatic Diagnostic Repository (ADR) Incident. |
|
ADR_ECID |
Execution Context ID (ECID) associated with the associated Automatic Diagnostic Repository (ADR) incident. An ECID i s a globally unique identifier used to tag and track a single call through the Oracle software stack. It is used to correlate problems that could occur across multiple tiers of the stack. |
|
ASSOC_PROBLEM_KEY |
Problem key associated with the Automatic Diagnostic Repository (ADR) incident. Problems are critical error s in an Oracle product. The Problem key is a text string that describes the prob lem. It includes an error code and in some cases, other error-specific values. |
Table 6-30 Environment Variables Specific to Compressed Incident Notifications
| Variable Name | Description |
|---|---|
| MESSAGE_URL | Clicking on this link will navigate to compressed incident details in Enterprise Manager. |
| ASSOC_EVENT_COUNT | Total compressed events in the incident |
|
INC_COMPR_EVT_1_MSG INC_COMPR_EVT_2_MSG INC_COMPR_EVT_3_MSG INC_COMPR_EVT_4_MSG INC_COMPR_EVT_5_MSG |
Event Message of the five latest events that are part of the compressed incident. Latest event message is INC_COMPR_EVT_5_MSG. |
|
INC_COMPR_EVT_1_TYPE INC_COMPR_EVT_2_TYPE INC_COMPR_EVT_3_TYPE INC_COMPR_EVT_4_TYPE INC_COMPR_EVT_5_TYPE |
Event Type of the five latest events that are part of the compressed incident. Latest event type is INC_COMPR_EVT_5_TYPE. |
|
INC_COMPR_EVT_1_SEV INC_COMPR_EVT_2_SEV INC_COMPR_EVT_3_SEV INC_COMPR_EVT_4_SEV INC_COMPR_EVT_5_SEV |
Event Severity of the five latest events that are part of the compressed incident. SEVERITY: WARNING, CRITICAL, FATAL, CLEAR Latest event severity is INC_COMPR_EVT_5_SEV. |
|
INC_COMPR_EVT_1_REP_TIME INC_COMPR_EVT_2_REP_TIME INC_COMPR_EVT_3_REP_TIME INC_COMPR_EVT_4_REP_TIME INC_COMPR_EVT_5_REP_TIME |
Event Creation Date of the five latest events that are part of the compressed incident. Latest event creation date is INC_COMPR_EVT_5_REP_TIME. |
|
INC_COMPR_EVT_1_TGT_NAME INC_COMPR_EVT_2_TGT_NAME INC_COMPR_EVT_3_TGT_NAME INC_COMPR_EVT_4_TGT_NAME INC_COMPR_EVT_5_TGT_NAME |
The Target Name in the five latest events that are part of the compressed incident. Latest target name is INC_COMPR_EVT_5_TGT_NAME. |
|
INC_COMPR_EVT_1_TGT_TYPE INC_COMPR_EVT_2_TGT_TYPE INC_COMPR_EVT_3_TGT_TYPE INC_COMPR_EVT_4_TGT_TYPE INC_COMPR_EVT_5_TGT_TYPE |
The Target Type in the five latest events that are part of the compressed incident. Latest target type is INC_COMPR_EVT_5_TGT_TYPE. |
Table 6-31 lists the associated problem's environment variables, when the incident is associated with a problem.
Table 6-31 Associated Problem Environment Variables for Incidents
| Environment Variable | Description |
|---|---|
|
ASSOC_PROBLEM_ACKNOWLEDGED_BY_OWNER |
Set to yes, if this problem was acknowledged by owner |
|
ASSOC_PROBLEM_STATUS |
Associated Problem Status |
|
ASSOC_PROBLEM_ID |
Associated Problem ID |
|
ASSOC_PROBLEM_PRIORITY |
Associated Problem priority |
|
ASSOC_PROBLEM_OWNER |
Associated Problem Owner if it is existed. |
|
ASSOC_PROBLEM_ESCALATION_LEVEL |
Escalation level of the associated Problem has a value between 0 to 5. |
Environment Variables Specific to Problem Notifications
Table 6-32 Problem-Specific Environment Variables
| Environment Variable | Description |
|---|---|
|
SEVERITY |
Problem Severity, it is translated. |
|
SEVERITY_CODE |
Code for Severity. Possible values are : FATAL, CRITICAL, WARNING, MINOR_WARNING, INFORMATIONAL, and CLEAR |
|
PROBLEM_REPORTED_TIME |
Problem reported time. |
|
PROBLEM_ACKNOWLEDGED_BY_OWNER |
Set yes, if problem is acknowledged by owner. |
|
PROBLEM_ID |
Problem ID |
|
PROBLEM_KEY |
Problem Key |
|
PROBLEM_OWNER |
Problem Owner |
|
ASSOC_INCIDENT_COUNT |
The number incident associated with this problem.. |
|
PROBLEM_STATUS |
Incident status. They are STATUS_NEW STATUS_CLOSED Any other user defined status. |
|
ESCALATED |
Is Incident escalated. Yes if it is escalated, otherwise No. |
|
ESCALATED_LEVEL |
The escalated level of incident. |
|
PRIORITY |
Incident priority. It is the translated priority name.. |
|
PRIOTITY_CODE |
Incident priority code It is the internal value defined in Enterprise Manager. PRIORITY_URGENT PRIORITY_VERY_HIGH PRIORITY_HIGH PRIORITY_MEDIUM PRIORITY_LOW PRIORITY_NONE |
|
LAST_UPDATED_TIME |
Last updated time |
|
SR_ID |
Oracle Service Request Id, if it exists. |
|
BUG_ID |
Oracle Bug ID, if an associated bug exists. |
Environment Variables Common to Incident and Problem Notifications
An incident or problem may be associated with multiple event sources. An event source can be a Target, a Source Object, or both.
Environment Variables Related to Event Sources
The number of event sources is set by the EVENT_SOURCE_COUNT environment variable. Using the EVENT_SOURCE_COUNT information, a script can be written to loop through the relevant environment variables to fetch the information about multiple event sources. Environment variables for all event sources are prefixed with EVENT_SOURCE_. Environment variables for source objects are suffixed with SOURCE_<attribute_name> . For example, EVENT_SOURCE_1_SOURCE_TYPE provides the source object type of first event source. Environment variables for a target are suffixed with TARGET_<attribute_name>. For example, EVENT_SOURCE_1_TARGET_NAME provides the target name of first event source.
The following table lists the environment variables for source object of x-th Event Source.
Table 6-33 Source Object of the x-th Event Source
| Environment Variable | Description |
|---|---|
|
EVENT_SOURCE_x_SOURCE_GUID |
Source Object GUID. |
|
EVENT_SOURCE_x_SOURCE_TYPE |
Source Object Type |
|
EVENT_SOURCE_x_SOURCE_NAME |
Source Object Name. |
|
EVENT_SOURCE_x_SOURCE_OWNER |
Source Object Owner. |
|
EVENT_SOURCE_x_SOURCE_SUB_TYPE |
Source Object Sub-Type. |
|
EVENT_SOURCE_x_SOURCE_URL |
Source Object URL to EM console. |
Table 6-34 lists the environment variables for a target of xth Event Source.
Table 6-34 Target of x-th Event Source
| Environment Variable | Description |
|---|---|
|
EVENT_SOURCE_x_TARGET_GUID |
Target GUID |
|
EVENT_SOURCE_x_TARGET_NAME |
Target name |
|
EVENT_SOURCE_x_TARGET_OWNER |
Target Owner |
|
EVENT_SOURCE_x_TARGET_VERSION |
Target version |
|
EVENT_SOURCE_x_TARGET_LIFE_CYCLE_STATUS |
Target life cycle status |
|
EVENT_SOURCE_x_TARGET_TYPE |
Target Type |
|
EVENT_SOURCE_x_HOST_NAME |
Target Host Name |
|
EVENT_SOURCE_x_TARGET_URL |
Target URL to EM Console. |
Passing Information to a PL/SQL Procedure
Passing event, incident, and problem information (payload) to PL/SQL procedures allows you to customize automated responses to these conditions. All three types of notification payloads have a common element: gc$notif_msg_info. It provides generic information that applies to all types of notifications. In addition, each of the three payloads have one specific element that provides the payload specific to the given issue type.
gc$notif_event_msg (payload for event notifications)
gc$notif_event_msg contains two objects - event payload object and message information object.
Table 6-35 Event Notification Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
EVENT_PAYLOAD |
gc$notif_event_payload |
Event notification payload. See gc$notif_event_payload type definition for detail. |
|
MSG_INFO |
gc$notif_msg_info |
Notification message. See gc$notif_msg_info definition for detail. |
gc$notif_incident_msg (payload for incident notifications)
gc$notif_incident_msg type contains two objects - incident payload and message information. This object represents the delivery payload for Incident notification message, contains all data associated with Incident notification, and can be accessed by user's custom PL/SQL procedures.
Table 6-36 Incident Notification Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
INCIDENT_PAYLOAD |
gc$notif_incident_payload |
Incident notification payload. See gc$notif_incident_payload type definition for detail. |
|
MSG_INFO |
gc$notif_msg_info |
Envelope level notification information. See gc$notif_msg_info type definition for detail. |
gc$notif_problem_msg (payload for problem notifications)
This object represents the delivery payload for Problem notification message, contains all data associated with problem notification, and can be accessed by a user's custom PL/SQL procedures.
Table 6-37 Problem Notification Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
PROBLEM_PAYLOAD |
gc$notif_problem_payload |
Problem notification payload. See gc$notif_problem_payload type definition for detail. |
|
MSG_INFO |
gc$notif_msg_info |
Notification message. See gc$notif_msg_info type definition for detail. |
gc$notif_msg_info (common for event/incident/problem payloads)
This object contains the generic notification information including notification_type, rule set and rule name, etc. for Event, Incident or Problem delivery payload.
Table 6-38 Event, Incident, Problem Common Payload
| Attribute | Datatype | Description |
|---|---|---|
|
NOTIFICATION_TYPE |
VARCHAR2(32) |
Type of notification, can be one of the following values. GC$NOTIFICATION.NOTIF_NORMAL GC$NOTIFICATION.NOTIF_RETRY GC$NOTIFICATION.NOTIF_REPEAT GC$NOTIFICATION.NOTIF_DURATION GC$NOTIFICATION.NOTIF_CA GC$NOTIFICATION.NOTIF_RCA |
|
REPEAT_COUNT |
NUMBER |
Repeat notification count |
|
RULESET_NAME |
VARCHAR2(256) |
Name of the rule set that triggered the notification |
|
RULE_NAME |
VARCHAR2(256) |
Name of the rule that triggered the notification |
|
RULE_OWNER |
VARCAH2(256) |
EM User who owns the rule set |
|
MESSAGE |
VARCHAR2(4000) |
Message about event/incident/problem. |
|
MESSAGE_URL |
VARCHAR2(4000) |
Link to the Enterprise Manager console page that provides the details of the event/incident/problem. |
gc$notif_event_payload (payload specific to event notifications)
This object represents the payload specific to event notifications.
Table 6-39 Common Payloads for Events, Incidents, and Problems
| Attribute | Datatype | Additional Information |
|---|---|---|
|
EVENT_INSTANCE_GUID |
RAW(16) |
Event instance global unique identifier. |
|
EVENT_SEQUENCE_GUID |
RAW(16) |
Event sequence global unique identifier. |
|
TARGET |
gc$notif_target |
Related Target Information object. See gc$notif_target type definition for detail. |
|
SOURCE |
gc$notif_source |
Related Source Information object, that is not a target. See gc$notif_source type definition for detail. |
|
EVENT_ATTRS |
gc$notif_event_attr_array |
The list of event specified attributes. See gc$notif_event_attr type definition for detail. |
|
CORRECTIVE_ACTION |
gc$notif_corrective_action_job |
Corrective action information, optionally populated when corrective action job execution has completed. |
|
EVENT_TYPE |
VARCHAR2(20) |
Event type - example: Metric Alert. |
|
EVENT_NAME |
VARCHAR2(512) |
Event name. |
|
EVENT_MSG |
VARCHAR2(4000) |
Event message. |
|
REPORTED_DATE |
DATE |
Event reported date. |
|
OCCURRENCE_DATE |
DATE |
Event occurrence date. |
|
SEVERITY |
VARCHAR2(128) |
Event Severity. It is the translated severity name. |
|
SEVERITY_CODE |
VARCHAR2(32) |
Event Severity code. It is the internal severity name used in Enterprise Manager. |
|
ASSOC_INCIDENT |
gc$notif_issue_summary |
Summary of associated incident. It is populated if the event is associated with an incident. See gc$notif_issue_summary type definition for detail |
|
ACTION_MSG |
VARCHAR2(4000) |
Message describing the action to take for resolving the event. |
|
RCA_DETAIL |
VARCHAR2(4000) |
Root cause analysis detail. The size of RCA details output is limited to 4000 characters long. |
|
EVENT_CONTEXT_DATA |
gc$notif_event_context_array |
Event context data. See gc$notif_event_context type definition for detail. |
|
CATEGORIES |
gc$category_string_array |
List of categories that the event belongs to. Category is translated based on locale defined in OMS server. Notification system sends up to 10 categories. |
|
CATEGORY_CODES |
gc$category_string_array |
Codes for the categories. The size of array is up to 10. |
gc$notif_incident_payload (payload specific to incident notifications)
Contains the incident specific attributes, associated problem and ticket information.
Table 6-40 Incident Notification Payloads
| Attribute | Datatype | Additional Information |
|---|---|---|
|
INCIDENT_ATTRS |
gc$notif_issue_attrs |
Incident specific attributes. See gc$notif_issue_attrs type definition for detail. |
|
ASSOC_EVENT_COUNT |
NUMBER |
The total number of events associated with this incident. |
|
TICKET_STATUS |
VARCHAR2(64) |
The status of external Ticket, if it exists. |
|
TICKET_ID |
VARCHAR2(128) |
The ID of external Ticket, if it exists. |
|
TICKET_URL |
VARCHAR2(4000) |
The URL for external Ticket, if it exists. |
|
ASSOC_PROBLEM |
gc$notif_issue_summary |
Summary of the problem, if it has an associated problem. See gc$notif_issue_summary type definition for detail. |
gc$notif_problem_payload (payload specific to problems)
Contains problem specific attributes, key, Service Request(SR) and Bug information.
Table 6-41 Problem Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
PROBLEM_ATTRS |
gc$notif_issue_attrs |
Problem specific attributes. See gc$notif_issue_attrs type definition for detail. |
|
PROBLEM_KEY |
VARCHAR2(850) |
Problem key if it is generated. |
|
ASSOC_INCIDENT_COUNT |
NUMBER |
Number of incidents associated with this problem. |
|
SR_ID |
VARCHAR2(64) |
Oracle Service Request Id, if it exists. |
|
SR_URL |
VARCHAR2(4000) |
URL for Oracle Service Request, if it exists. |
|
BUG_ID |
VARCHAR2(64) |
Oracle Bug ID, if an associated bug exists. |
gc$notif_issue_attrs (payload common to incidents and problems)
Provides common details for incident and problem. It contains details such as id, severity, priority, status, categories, acknowledged by owner, and source information with which it is associated.
Table 6-42 Payload Common to Incidents and Problems
| Attribute | Datatype | Additional Information |
|---|---|---|
|
ID |
NUMBER(16) |
ID of the incident or problem. |
|
SEVERITY |
VARCHAR2(128) |
Issue Severity. It is the translated. |
|
SEVERITY_CODE |
VARCHAR2(32) |
Issue Severity Code.The possible values are defined in descending order of severity: GC$EVENT.FATAL GC$EVENT.CRITICAL GC$EVENT.WARNING GC$EVENT.MINOR_WARNING GC$EVENT.INFORMATIONAL GC$EVENT.CLEAR |
|
PRIORITY |
VARCHAR2(128) |
Issue Priority. It is the translated priority name. |
|
PRIORITY_CODE |
VARCHAR2(32) |
Issue Priority. It is the internal value defined in EM. The possible values are defined in descending order of priority: GC$EVENT.PRIORITY_URGENT GC$EVENT.PRIORITY_VERY_HIGH GC$EVENT.PRIORITY_HIGH GC$EVENT.PRIORITY_MEDIUM GC$EVENT.PRIORITY_LOW GC$EVENT.PRIORITY_NONE |
|
STATUS |
VARCHAR2(32) |
Status of Issue. The possible values are GC$EVENT.STATUS_NEW GC$EVENT.STATUS_CLOSED Any other user defined status. |
|
ESCALATION_LEVEL |
NUMBER(1) |
Escalation level of the issue, has a value between 0 to 5. |
|
OWNER |
VARCHAR(256) |
Issue Owner. Set to NULL if no owner exists. |
|
ACKNOWLEDGED_BY_OWNER |
NUMBER(1) |
Set to 1, if this issue was acknowledged by owner. |
|
CREATION_DATE |
DATE |
Issue creation date. |
|
CLOSED_DATE |
DATE |
Issue closed date, null if not closed. |
|
CATEGORIES |
gc$category_string_array |
List of categories that the event belongs to. Category is translated based on locale defined in OMS server. Notification system sends up to 10 categories. |
|
CATEGORY_CODES |
gc$category_string_array |
Codes for the categories. Notification system sends up to 10 category codes. |
|
SOURCE_INFO_ARR |
gc$notif_source_info_array |
Array of source information associated with this issue. See $gcnotif_source_info type definition for detail. |
|
LAST_MODIFIED_BY |
VARCHAR2(256) |
Last modified by user. |
|
LAST_UPDATED_DATE |
DATE |
Last updated date. |
gc$notif_issue_summary (common to incident and problem payloads)
Represents the associated incident summary in the event payload, or associated problem summary in the incident payload, respectively.
Table 6-43 Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
ID |
NUMBER |
Issue Id, either Incident Id or Problem Id. |
|
SEVERITY |
VARCHAR(128) |
The severity level of an issue. It is translated severity name. |
|
SEVERITY_CODE |
VARCHAR2(32) |
Issue Severity Code, has one of the following values. GC$EVENT.FATAL GC$EVENT.CRITICAL GC$EVENT.WARNING GC$EVENT.MINOR_WARNING GC$EVENT.INFORMATIONAL GC$EVENT.CLEAR |
|
PRIORITY |
VARCHAR2(128) |
Current priority. It is the translated priority name. |
|
PRIORITY_CODE |
VARCHAR2(32) |
Issue priority code, has one of the following values. GC$EVENT.PRIORITY_URGENT GC$EVENT.PRIORITY_VERY_HIGH GC$EVENT.PRIORITY_HIGH GC$EVENT.PRIORITY_MEDIUM GC$EVENT.PRIORITY_LOW GC$EVENT.PRIORITY_NONE |
|
STATUS |
VARCHAR2(64) |
Status of issue. The possible values are GC$EVENT.STATUS_NEW GC$EVENT.STATUS_CLOSED GC$EVENT.WIP (work in progress) GC$EVENT.RESOLVED any other user defined status |
|
ESCALATION_LEVEL |
VARCHAR2(2) |
Issue escalation level range from 0 to 5, default 0. |
|
OWNER |
VARCHAR2(256) |
Issue Owner. Set to NULL if no owner exists. |
|
ACKNOWLEDGED_BY_OWNER |
NUMBER(1) |
Set to 1, if this issue was acknowledged by owner. |
gc$category_string_array
gc$category_string_array is an array of string containing the categories which event, incident or problem is associated with. The notification system delivers up to 10 categories.
gc$notif_event_context_array
gc$notif_event_context_array provides information about the additional diagnostic data that was captured at event detection time. Note that notification system delivers up to 200 elements from the captured event context. Each element of this array is of the type gc$notif_event_context.
gc$notif_event_context: This object represents the detail of event context data which is additional contextual information captured by the source system at the time of event generation that may have diagnostic value. The context for an event should consist of a set of keys and values along with data type (Number or String only).
Table 6-44 Event Context Type
| Attribute | Datatype | Additional Information |
|---|---|---|
|
NAME |
VARCHAR2(256) |
The event context name. |
|
TYPE |
NUMBER(1) |
The data type of the value, which is stored (0) - for numeric data (1) - for string data. |
|
VALUE |
NUMBER |
The numerical value. |
|
STRING_VALUE |
VARCHAR2(4000) |
The string value. |
gc$notif_corrective_action_job
Provides information about the execution of a corrective action job. Note that the corrective actions are supported for metric alert and target availability events only.
Table 6-45 Corrective Action Job-Specific Attributes
| Attribute | Datatype | Additional Information |
|---|---|---|
|
JOB_GUID |
RAW(16) |
Corrective action job global unique identifier. |
|
JOB_NAME |
VARCHAR2(128) |
The value will be the name of the corrective action. It applies to Metric Alert and Target Availability Events. |
|
JOB_OWNER |
VARCHAR2(256) |
Corrective action job owner. |
|
JOB_TYPE |
VARCHAR2(256) |
Corrective action job type. |
|
JOB_STATUS |
VARCHAR2(64) |
Corrective action job execution status. |
|
JOB_STATUS_CODE |
NUMBER |
Corrective action job execution status code. It is the internal value defined in Enterprise Manager. For more information on status codes, see Table 6-14. |
|
JOB_STEP_OUTPUT |
VARCHAR2(4000) |
The value will be the text output from the corrective action execution. This will be truncated to last 4000 characters. |
|
JOB_EXECUTION_GUID |
RAW(16) |
Corrective action job execution global unique identifier. |
|
JOB_STATE_CHANGE_GUID |
RAW(16) |
Corrective action job change global unique identifier. |
|
OCCURRED_DATE |
DATE |
Corrective action job occurred date. |
gc$notif_source_info_array
Provides access to the multiple sources to which an incident or a problem could be related. NOTE: The notification system delivers up to 200 sources associated with an incident or a problem.
CREATE OR REPLACE TYPE gc$notif_source_info_array AS VARRAY(200) OF gc$notif_source_info;
gc$notif_source_info
Notification source information which is used for referencing source information containing either target or source, or both.
Table 6-46 Source Information Type
| Attribute | Datatype | Additional Information |
|---|---|---|
|
TARGET |
gc$notif_target |
It is populated when the event is related to a target. See gc$notif_target type definition for detail. |
|
SOURCE |
gc$notif_source |
It is populated when the event is related to a (non-target) source. See gc$notif_source type definition for detail. |
gc$notif_source
Used for referencing source objects other than a job target.
Table 6-47 Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
SOURCE_GUID |
RAW(16) |
Source's global unique identifier. |
|
SOURCE_TYPE |
VARCHAR2(120) |
Type of the Source object, e.g., TARGET, JOB, TEMPLATE, etc. |
|
SOURCE_NAME |
VARCHAR2(256) |
Source Object Name. |
|
SOURCE_OWNER |
VARCHAR2(256) |
Owner of the Source object. |
|
SOURCE_SUB_TYPE |
VARCHAR2(256) |
Sub-type of the Source object, for example, within the TARGET these would be the target types like Host, Database etc. |
|
SOURCE_URL |
VARCHAR2(4000) |
Source's event console URL. |
gc$notif_target
Target information object is used for providing target information.
Table 6-48 Target Information
| Attribute | Datatype | Additional Information |
|---|---|---|
|
TARGET_GUID |
RAW(16) |
Target's global unique identifier. |
|
TARGET_NAME |
VARCHAR2(256) |
Name of target. |
|
TARGET_OWNER |
VARCHAR2(256) |
Owner of target. |
|
TARGET_LIFECYCLE_STATUS |
VARCHAR2(1024) |
Life Cycle Status of the target. |
|
TARGET_VERSION |
VARCHAR2(64) |
Target Version of the target. |
|
TARGET_TYPE |
VARCHAR2(128) |
Type of a target. |
|
TARGET_TIMEZONE |
VARCHAR2(64) |
Target's regional time zone. |
|
HOST_NAME |
VARCHAR2(256) |
The name of the host on which the target is deployed upon. |
|
TARGET_URL |
VARCHAR2(4000) |
Target's EM Console URL. |
|
UDTP_ARRAY |
gc$notif_udtp_array |
The list of user defined target properties. It is populated for events that are associated with a target. It is populated for incidents and problems, when they are associated with a single source (gc$notif_source_info). |
gc$notif_udtp_array
Array of gc$notif_udtp type with a maximum size of 20.
CREATE OR REPLACE TYPE gc$notif_udtp_array AS VARRAY(20) OF gc$notif_udtp;
gc$notif_udtp
Used for referencing User-defined target properties. UDTP should consist of a set of property key names and property values.
Table 6-49 Payload
| Attribute | Datatype | Additional Information |
|---|---|---|
|
NAME |
VARCHAR2(64), |
The name of property. |
|
VALUE |
VARCHAR2(1024) |
Property value. |
|
LABEL |
VARCHAR(256) |
Property label. |
|
NLS_ID |
VARCHAR(64) |
Property nls id |
Notification Payload Elements Specific to Event Types
gc$notif_event_attr_array
Array of gc$notif_event_attr is used for referencing event-specific attributes. The array has a maximum size of 25. Each element of the array is of type gc$notif_event_attr (used for referencing event type-specific attributes).
Table 6-50 Event Attribute Type
| Attribute | Datatype | Additional Information |
|---|---|---|
|
NAME |
VARCHAR2(64) |
The internal name of event type specific attribute. |
|
VALUE |
VARCHAR2(4000) |
value. |
|
NLS_VALUE |
VARCHAR2(4000) |
Translated value for the attribute. |
You can use SQL queries to list the deployed event types in your deployment and the payload specific to each. The following SQL can be used to list all internal event type names which are registered in the Enterprise Manager.
Select event_class as event_type, upper(name) as env_var_name from em_event_class_attrs where notif_order != 0 and event_class is not null order by event_type, env_var_name;
You should convert the attribute name to upper case before using the name for comparison.
There is an attribute variable payload specific to each event type that can be accessed from a gc$notif_event_attr_array database type. The following tables list notification attributes for the most critical event types. You should convert the attribute name to uppercase before using the name for comparison.
Table 6-51 Environment variables specific to Metric Alert Event Type
| Environment Variable | Description |
|---|---|
|
COLL_NAME |
The name of the collection collecting the metric. |
|
KEY_COLUMN_X |
Internal name of Key Column X where X is a number between 1 and 7. |
|
KEY_COLUMN_X_VALUE |
Value of Key Column X where X is a number between 1 and 7. |
|
KEY_VALUE |
Monitored object for the metric corresponding to the Metric Alert event. |
|
METRIC_COLUMN |
The name of the metric column |
|
METRIC_DESCRIPTION |
Brief description of the metric. |
|
METRIC_GROUP |
The name of the metric. |
|
NUM_KEYS |
The number of key metric columns in the metric. |
|
SEVERITY_GUID |
The GUID of the severity record associated with this metric alert. |
|
VALUE |
Value of the metric when the event triggered. |
Table 6-52 Environment variables specific to Target Availability Event Type
| Environment Variable | Description |
|---|---|
|
AVAIL_SEVERITY |
The transition severity (0-6) that resulted in the status of the target to change to the current availability status. Possible Values for AVAIL_SEVERITY
|
|
AVAIL_SUB_STATE |
The substatus of a target for the current status. |
|
CYCLE_GUID |
A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear. |
|
METRIC_GUID |
Metric GUID of response metric. |
|
SEVERITY_GUID |
The GUID of the severity record associated with this availability status. |
|
TARGET_STATUS |
The current availability status of the target. |
Table 6-53 Environment variables specific to Job Status Change event type
| Environment Variable | Description |
|---|---|
|
EXECUTION_ID |
Unique ID of the job execution.. |
|
EXECUTION_LOG |
The job output of the last step executed. |
|
EXECUTION_STATUS |
The internal status of the job execution. |
|
EXEC_STATUS_CODE |
Execution status code of job execution. For possible values, see Table 6-16. |
|
STATE_CHANGE_GUID |
Unique ID of last status change |
Example 6-17 PL/SQL Script: Event Type Payload Elements
-- log_table table is created by following DDL to demostrate how to access
-- event notification payload GC$NOTIF_EVENT_MSG.
CREATE TABLE log_table (message VARCHAR2(4000)) ;
-- Define PL/SQL notification method for Events
CREATE OR REPLACE PROCEDURE log_table_notif_proc(s IN GC$NOTIF_EVENT_MSG)
IS
l_categories gc$category_string_array;
l_category_codes gc$category_string_array;
l_attrs gc$notif_event_attr_array;
l_ca_obj gc$notif_corrective_action_job;
BEGIN
INSERT INTO log_table VALUES ('notification_type: ' || s.msg_info.notification_type);
INSERT INTO log_table VALUES ('repeat_count: ' || s.msg_info.repeat_count);
INSERT INTO log_table VALUES ('ruleset_name: ' || s.msg_info.ruleset_name);
INSERT INTO log_table VALUES ('rule_name: ' || s.msg_info.rule_name);
INSERT INTO log_table VALUES ('rule_owner: ' || s.msg_info.rule_owner);
INSERT INTO log_table VALUES ('message: ' || s.msg_info.message);
INSERT INTO log_table VALUES ('message_url: ' || s.msg_info.message_url);
INSERT INTO log_table VALUES ('event_instance_guid: ' || s.event_payload.event_instance_guid);
INSERT INTO log_table VALUES ('event_type: ' || s.event_payload.event_type);
INSERT INTO log_table VALUES ('event_name: ' || s.event_payload.event_name);
INSERT INTO log_table VALUES ('event_msg: ' || s.event_payload.event_msg);
INSERT INTO log_table VALUES ('source_obj_type: ' || s.event_payload.source.source_type);
INSERT INTO log_table VALUES ('source_obj_name: ' || s.event_payload.source.source_name);
INSERT INTO log_table VALUES ('source_obj_url: ' || s.event_payload.source.source_url);
INSERT INTO log_table VALUES ('target_name: ' || s.event_payload.target.target_name);
INSERT INTO log_table VALUES ('target_url: ' || s.event_payload.target.target_url);
INSERT INTO log_table VALUES ('severity: ' || s.event_payload.severity);
INSERT INTO log_table VALUES ('severity_code: ' || s.event_payload.severity_code);
INSERT INTO log_table VALUES ('event_reported_date: ' || to_char(s.event_payload.reported_date, 'D MON DD HH24:MI:SS'));
IF s.event_payload.target.TARGET_LIFECYCLE_STATUS IS NOT NULL
THEN
INSERT INTO log_table VALUES ('target lifecycle_status: ' || s.event_payload.target.TARGET_LIFECYCLE_STATUS);
END IF;
-- Following block illustrates the list of category codes to which the event
-- belongs.
l_category_codes := s.event_payload.category_codes;
IF l_categories IS NOT NULL
THEN
FOR c IN 1..l_category_codes.COUNT
LOOP
INSERT INTO log_table VALUES ('category_code ' || c || ' - ' || l_category_codes(c));
END LOOP;
END IF;
--
-- Each event type has a specific set of attributes modeled. Examples of
-- event types include metric_alert, target_availability, job_status_change.
-- Following block illustrates how to access the attributes for job_status change
-- event type
--
IF s.event_payload.event_type = 'job_staus_chage'
THEN
l_attrs := s.event_payload.event_attrs;
IF l_attrs IS NOT NULL
THEN
FOR c IN 1..l_attrs.COUNT
LOOP
INSERT INTO log_table VALUES ('EV.ATTR name=' || l_attrs(c).name || ' value=' || l_attrs(c).value || ' nls_value=' || l_attrs(c).nls_value);
END LOOP;
END IF;
END IF;
-- Following block illustrates how to access corrective action job's attributes IF s.msg_info.notification_type = GC$NOTIFICATION.NOTIF_CA AND s.event_payload.corrective_action IS NOT NULL
THEN
l_ca_obj := s.event_payload.corrective_action;
INSERT INTO log_table VALUES ('CA JOB_GUID: ' || l_ca_obj.JOB_GUID);
INSERT INTO log_table VALUES ('CA JOB_NAME: ' || l_ca_obj.JOB_NAME);
INSERT INTO log_table VALUES ('CA JOB_OWNER: ' || l_ca_obj.JOB_OWNER);
INSERT INTO log_table VALUES ('CA JOB_TYPE: ' || l_ca_obj.JOB_TYPE);
INSERT INTO log_table VALUES ('CA JOB_STATUS: ' || l_ca_obj.JOB_STATUS);
INSERT INTO log_table VALUES ('CA JOB_STATUS_CODE: ' || l_ca_obj.JOB_STATUS_CODE);
INSERT INTO log_table VALUES ('CA JOB_STEP_OUTPUT: ' || l_ca_obj.JOB_STEP_OUTPUT);
INSERT INTO log_table VALUES ('CA JOB_EXECUTION_GUID: ' || l_ca_obj.JOB_EXECUTION_GUID);
INSERT INTO log_table VALUES ('CA JOB_STATE_CHANGE_GUID: ' || l_ca_obj.JOB_STATE_CHANGE_GUID);
INSERT INTO log_table VALUES ('CA OCCURRED_DATE: ' || l_ca_obj.OCCURRED_DATE); END IF;
COMMIT ;
END ;
/Passing Information to Webhooks and Slack
The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.
Token Variables Specific to Event Notifications
Incident Payload
| Token Name | Description |
|---|---|
${incident.incidentID} |
ID of the incident |
|
Display ID of the incident |
${incident.severity} |
Severity code of the incident |
|
Name of the target |
${incident.targetType} |
Target type |
${incident.creationTimestamp} |
Incident creation timestamp |
${incident.summary} |
Summary message for the incident |
${incident.resolutionStatus} |
Resolution state for the incident |
${incident.owner} |
Owner of the incident |
Event Payload
| Token Name | Description |
|---|---|
${event.eventID} |
ID for the event |
${event.eventClass} |
Event class |
${event.msg} |
Message for the event |
${event.severity} |
Severity code for the event |
${event.target} |
Name of the target |
${event.creationTimestamp} |
Evet creation timestamp |
${event.targetType} |
Target type |
Troubleshooting Notifications
To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.
General Setup
The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.
OS Command, PL/SQL and SNMP Trap Notifications
Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called, for example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.
Email Notifications
-
Make sure an email gateway is set up under the Notification Methods page of Setup. The Sender's email address should be valid. Clicking the Test button will send an email to the Sender's email address. Make sure this email is received. Note that the Test button ignores any Notification Schedule.
-
Make sure an email address is set up. Clicking the Test button will send an email to specified address and you should make sure this email is received. Note that the Test button ignores any Notification Schedule.
-
Make sure an email schedule is defined. No emails will be sent unless a Notification Schedule has been defined.
-
Make sure a incident rule is defined that matches the states you are interested and make sure email and notification methods are assigned to the rule.
Notification System Errors
For any alerts involving problems with notifications, check the following for notification errors.
-
Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. From the Setup menu, select Management Services and Repository to view these errors.
-
Check for any delivery errors. You can view them from Incident Manager. From the Enterprise menu, select Monitoring, then select Incident Manager. The details will give the reason why the notification was not delivered.
Notification System Trace Messages
The Notification System can produce trace messages in sysman/log/emoms.trc file.
Tracing is configured by setting the log4j.category.oracle.sysman.em.notification property flag using the emctl set property command. You can set the trace level to INFO, WARN, DEBUG. For example,
emctl set property -name log4j.category.oracle.sysman.em.notification -value DEBUG -module logging
Note: The system will prompt you for the SYSMAN password.
Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc and emoms_pbs.trc files using the grep command. For example,
grep em.notification emoms.trc emoms_pbs.trc
What to look for in the trace file.
The following entries in the emoms.trc file are relevant to notifications.
Normal Startup Messages
When the OMS starts, you should see these types of messages.
2011-08-17 13:50:29,458 [EventInitializer] INFO em.notification init.167 - Short format maximum length is 155 2011-08-17 13:50:29,460 [EventInitializer] INFO em.notification init.185 - Short format is set to both subject and body 2011-08-17 13:50:29,460 [EventInitializer] INFO em.notification init.194 - Content-Transfer-Encoding is 8-bit 2011-08-17 13:50:29,460 [EventInitializer] DEBUG em.notification registerAdminMsgCallBack.272 - Registering notification system message call back 2011-08-17 13:50:29,461 [EventInitializer] DEBUG em.notification registerAdminMsgCallBack.276 - Notification system message callback is registered successfully 2011-08-17 13:50:29,713 [EventInitializer] DEBUG em.notification upgradeEmailTemplates.2629 - Enter upgradeEmailTemplates 2011-08-17 13:50:29,735 [EventInitializer] INFO em.notification upgradeEmailTemplates.2687 - Email template upgrade is not required since no customized templates exist. 2011-08-17 13:49:28,739 [EventCoordinator] INFO events.EventCoordinator logp.251 - Creating event worker thread pool: min = 4 max = 15 2011-08-17 13:49:28,791 [[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO emdrep.pingHBRecorder initReversePingThreadPool.937 - Creating thread pool for reverse ping : min = 10 max = 50 2011-08-17 13:49:28,797 [[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG emdrep.HostPingCoordinator logp.251 - Creating thread pool of worker thread for host ping: min = 1 max = 10 2011-08-17 13:49:28,799 [[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG emdrep.HostPingCoordinator logp.251 - Creating thread pool for output of worker's output for host ping: min = 2 max = 20 2011-08-17 13:49:30,327 [ConnectorCoordinator] INFO connector.ConnectorPoolManager logp.251 - Creating Event thread pool: min = 3 max = 10 2011-08-17 13:51:48,152 [NotificationMgrThread] INFO notification.pbs logp.251 - Creating thread pool: min = 6 max = 24 2011-08-17 13:51:48,152 [NotificationMgrThread] INFO em.rca logp.251 - Creating RCA thread pool: min = 3 max = 20
Notification Delivery Messages
2006-11-08 03:18:45,387 [NotificationMgrThread] INFO em.notification run.682 - Notification ready on EMAIL1 2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO em.notification run.114 - Deliver to SYSMAN/admin@myco.com 2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO em.notification run.227 - Notification handled for SYSMAN/admin@myco.com
Notification System Error Messages
2011-08-17 14:02:23,905 [NotificationMgrThread] DEBUG notification.pbs logp.251 - Notification ready on EMAIL1 2011-08-17 14:02:23,911 [NotificationMgrThread] DEBUG notification.pbs logp.251 - Notification ready on PLSQL4 2011-08-17 14:02:23,915 [NotificationMgrThread] DEBUG notification.pbs logp.251 - Notification ready on OSCMD14 2011-08-17 14:02:19,057 [DeliveryThread-EMAIL1] INFO notification.pbs logp.251 - Deliver to To: my.admin@myco.com; issue type: 1; notification type: 1 2011-08-17 14:02:19,120 [DeliveryThread-OSCMD14] INFO notification.pbs logp.251 - Deliver to SYSMAN, OSCMD, 8; issue type: 1; notification type: 1 2011-08-17 14:02:19,346 [DeliveryThread-PLSQL4] INFO notification.pbs logp.251 - Deliver to SYSMAN, LOG_JOB_STATUS_CHANGE, 9; issue type: 1; notification type: 1 2011-08-17 14:02:19,977 [DeliveryThread-PLSQL4] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, LOG_JOB_STATUS_CHANGE, 9 2011-08-17 14:02:20,464 [DeliveryThread-EMAIL1] DEBUG notification.pbs logp.251 - Notification handled for To: my.admin@myco.com 2011-08-17 14:02:20,921 [DeliveryThread-OSCMD14] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, OSCMD, 8
Email Errors
The SMTP gateway is not set up correctly:
Failed to send email to my.admin@myco.com: For email notifications to be sent, your Super Administrator must configure an Outgoing Mail (SMTP) Server within Enterprise Manager. (SYSMAN, myrule)
Invalid host name:
Failed to connect to gateway: badhost.oracle.com: Sending failed; nested exception is: javax.mail.MessagingException: Unknown SMTP host: badhost.example.com;
Invalid email address:
Failed to connect to gateway: rgmemeasmtp.mycorp.com: Sending failed; nested exception is: javax.mail.MessagingException: 550 5.7.1 <smpemailtest_ie@example.com>... Access denied
Always use the Test button to make sure the email gateway configuration is valid. Check that an email is received at the sender's email address
OS Command Errors
When attempting to execute an OS command or script, the following errors may occur. Use the Test button to make sure OS Command configuration is valid. If there are any errors, they will appear in the console.
Invalid path or no read permissions on file:
Could not find /bin/myscript (machineb10.oracle.com_Management_Service) (SYSMAN, myrule )
No execute permission on executable:
Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute (machineb10.oracle.com_Management_Service) (SYSMAN, myrule )
Timeout because OS Command ran too long:
Timeout occurred running /bin/myscript (machineb10.oracle.com_Management_Service) (SYSMAN, myrule )
Any errors such as out of memory or too many processes running on OMS machine will be logged as appropriate.
Always use the Test button to make sure OS Command configuration is valid.
SNMP Trap Errors
Use the Test button to make sure SNMP Trap configuration is valid.
The OMS will not report an error if the SNMP trap cannot reach the third party SNMP console as this is sent via UDP. If the SNMP trap encounters problems when trying to reach the third party SNMP console, possible SNMP trap problems include: invalid host name, port, community for a machine running an SNMP Console or a network issue such as a firewall problem.
Other possible SNMP trap problems include: invalid host name, port, or community for a machine running an SNMP Console.
PL/SQL Errors
When attempting to execute an PL/SQL procedure, the following errors may occur. Use the Test button to make sure the procedure is valid. If there are any errors, they will appear in the console.
Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC
Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or procedure name (SYSMAN, myrule)
Procedure is not the correct signature. Example: PROCEDURE event_proc(s IN GC$NOTIF_EVENT_MSG)
Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)
Procedure has bug and is raising an exception.
Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized collection (SYSMAN, myrule)
Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to this condition will result in delivery failure with the message being displayed in the Details section of the alert in the Cloud Control console.
Always use the Test button to make sure the PL/SQL configuration is valid.
System Broadcasts
Enterprise Manager allows you to broadcast important instantly viewable system messages to Enterprise Manager consoles throughout your managed environment. These messages can be directed to specific users or all Enterprise Manager users. This feature can be useful when notifying users that Enterprise Manager is about to go down, when some part of your managed infrastructure has been updated, or when there is a system emergency.
Note:
Only Super Administrators can send system broadcasts.
Setting System Broadcast Preferences
Before you can send a system broadcast, you must first set your broadcast preferences.
-
Log in to Enterprise Manager as a Super Administrator.
-
From the <USERNAME> menu, select Preference and then System Broadcast. The System Broadcast User Preferences UI displays.
-
Check the desired broadcast message preferences then click Save.
Whenever you send a system broadcast message, these are the preferences that will be used.
Note:
The Number of seconds to show the System Broadcast setting will only work when the Do not automatically close System Broadcast sent by the super administrator option is disabled.
Creating a System Broadcast
Once your preferences are set, you use the EM CLI verb send_system_broadcast to send a system broadcast message.
emcli send_system_broadcast
-toOption="ALL|SPECIFIC"
[-to="comma separated user names"]
[-messageType="INFO|CONF|WARN|ERROR|FATAL" (default is INFO)]
-message="message details"
Options
-
toOption
Enter the value ALL to send the broadcast message to all users logged into the Enterprise Manager Console. Or enter SPECIFIC to send System Broadcast to users specified by -to.
-
to
Comma-separated list of users who are to receive the broadcast message. This option can only be used if the -toOption is set to SPECIFIC.
-
messageType
Type of System Broadcast, it can be one of following types
-
INFO (Information)
-
CONF (Confirmation)
-
WARN (Warning)
-
ERROR
-
FATAL
-
-
message
Message to be sent in the System Broadcast. The message has a maximum of 200 characters.
Example:
In this example, you want to broadcast an informational message indicating that you will be bringing down Enterprise Manager within an hour in order to perform an emergency patching operation.
emcli send_system_broadcast -messageType="INFO" -toOption="ALL" -message="Enterprise Manager will be taken down in an hour for an emergency patch"