Creating Corrective Actions for Events

Prior to Enterprise Manager release 13.1, corrective actions could only be associated with metric alerts. Enterprise Manager release 13.1 now allows script-based corrective actions to fire on an event by associating them with event rules. This greatly increases the number of situations where corrective actions can be used, such as compliance standard violations, metric errors, or target availability. By associating corrective actions with event rules, you can have the corrective action performed automatically.

You can also initiate the corrective action manually through the event details Guided Resolutions area of Incident Manager. For a detailed discussion about corrective actions, see "Creating Corrective Actions.".

Corrective Actions in Event Rules

When you create an event rule to be triggered when a matching event occurs, you can select an appropriate predefined corrective action from the Corrective Actions Library. The corrective actions available for selection will depend on the event type and target type selected for the rule.

When an event rule set is exported or imported, the associated corrective actions will be exported/imported as well. For more information about importing/exporting event rules, see "Exporting and Importing Incident Rules."

Create the Corrective Action

In order to associate a corrective action with an event rule, you must first add it to the Corrective Action Library. After a corrective action is in the library, you can reuse the corrective action definition whenever you define a corrective action for an event rule.

From the Enterprise menu, select Monitoring, and then Corrective Actions. The Corrective Action Library page appears.
Select a job type from the Create Library Corrective Action drop-down. For events, you must create an OS Command job type so that a script can be executed. Select OS Command, specify a name and then click Go. The Create OS Command Corrective Action page displays.

Specify a corrective action Name and a brief Description or event type.
From the Target Type drop-down menu, choose a target type. Click on the Parameters tab.
From the Command Type drop-down menu, choose Script.
Enter the OS script text.

All target and event Properties that can be used in the script are listed in the table to the right.

Tip: When accessing an Event Details page from Incident Manager, you can click Show Internal Values for Attributes to display the internal name and values for the event attributes. You can use this to determine what information you can access when writing the script for the corrective action. Just copy and paste the information from the dialog into a text editor and refer to this list of attributes when creating your script

Note:

If you are using an event context parameter, it must be prefixed with EVTCTX.
Specify an interpreter. For example, %perlbin%/perl
Once you have finished, click Save to Library. The Corrective Actions Library page displays and your corrective action appears in the library list.

At this point, the corrective action will be in draft status. At this stage, you can test and revise the corrective action. However, only you, as owner, can test the CA by running the CA manually from Incident Manager.

To test the corrective action, you must trigger an event that matches the event rule with the associated corrective action to see if the actions are what you expect. Once you are satisfied and are ready for other administrators to use the corrective action, proceed to the next step.

Note: The Access tab on the "Create 'OS Command' Corrective Action" page displays administrators and roles that have access to this corrective action. You can change access to this corrective action from this tab, if required.
Navigate to the Corrective Actions Library page and select the Corrective Action and then click Publish. A confirmation message displays. Click Yes to confirm publication.
Set the Preferred Credentials. From the Setup menu, select Security and then Preferred Credentials. The Preferred Credentials page displays. Note that the preferred credential of the rule set owner will be used by the corrective action linked to the rule.

Note:

The corrective action will use these credentials to access the system and carry out the actions (in this case, running the script). For example, set credential for host if your corrective action is going to perform corrective actions on a specific host.
If not already set, select the Target Type to be accessed by the corrective action and click Manage Preferred Credentials. You need to define the Default Preferred Credentials for the specific target type that the CA is going to perform the actions on. The target type's Preferred Credentials page displays. On the My preferences tab, navigate to the Default Preferred Credentials region and select the applicable credential. Click Set.

Note:

Preferred credentials must be set or the corrective action will fail.

Associate the Corrective Action with an Event Rule

Once you have created the corrective action to be associated with an event, you are now ready to create an event rule that uses the corrective action. You can only associate one corrective action per conditional action of the rule.

From the Setup menu, select Incidents and then Incident Rules. The Incident Rules - All Enterprise Rules page displays.
Click Create Incident Rule Set. The Create Rule Set page displays.
Enter a rule set Name and Description.
Select the appropriate Targets.
Scroll down to the Rules section and click Create... The Select Type of Rule dialog displays. Choose Incoming events and updates to events and click Continue. The Create Rule Set wizard appears.
From the Type drop-down menu, select the event Type. By default, Metric Alert is selected. Choose one of the event types, Compliance Standard Rule Violation, for example. Expand the Advanced Selection Options and set any event parameters to which the event rule should apply.
Click Next to proceed to the Add Actions page.
On the Add Actions page, click Add. The Add Conditional Actions page displays.
Scroll down to the Submit Corrective Action section and click Select Corrective Action. The corrective action selection dialog displays.
Choose the corrective action to be attached and click OK.

Note:

You are not prompted for credentials because the rules are run in the background and the rule set owner's preferred credentials are used to execute the corrective action.
Click Continue. You are returned to the main Add Actions page. Continue to add more actions, if necessary.
Complete the rule set definition and ensure that it appears in the list of incident rule sets on the Incident Rules - All Enterprise Rules page.

You will need to recreate the particular rule violation in order to test the CA.

Running the Corrective Action Manually

If you are aware that there exists a corrective action in the Corrective Action Library that can resolve the current event, you can run the corrective action manually from the library. In the Guided Resolution section of an Event Details page, the Corrective Actions area displays the Submit from Library link.

Click Submit from Library to display the Corrective Action Library dialog. This dialog lists ONLY those corrective actions that apply to the current event conditions. Select a corrective action from the list. The credential settings are displayed. By default, the preferred credentials are shown. You have the option of using alternate credentials.

Once set, click Submit. The Corrective action <CA name> submitted successfully dialog displays. Click the link Click here to view the execution details."to go to the job execution page. Here, you can view the job status and output.