Creating Dynamic Runbooks

Any Runbook Author can create a Dynamic Runbook

How to create a Dynamic Runbook - YouTube Video

  1. To create a Dynamic Runbook for an incident, from the Enterprise menu, select Monitoring and then Incident Manager. View incident details by clicking its row and then go to the Runbook Sessions region. If there is a Runbook session that is currently active, it will be listed.

    Alternatively, go to the All Metrics page to access Runbooks. To create a Dynamic Runbook for a metric, locate the metric in the All Metrics page of the appropriate target. You should see a Runbook Sessions region next to the metric's performance charts.

  2. In the Runbook Sessions region, click Create Runbook. The New Runbook authoring UI displays in the context of the incident or metric, thus allowing you to create Runbook steps.
  3. By default, the authoring UI opens with the Overview & Prerequisites as the first step. Here, you enter a description of what the new Runbook will accomplish and any prerequisites that are required to complete all Runbook steps. To aid readability, the Overview & Prerequisites step supports simple formatting via markdown notation. Click on the information icon (“I”) in the text entry area to view a list of markown elements supported by the text editor. The markdown formatting will be applied when you save the step.
  4. Click Save Step.
  5. From the Add a Step drop-down menu, select the desired step type you want to define. There are four step types:
    • Note: Use the Note step to provide instructions to the user. You can use markdown formatting with this step and also include Runbook variables. Refer to the Runbooks UI for details on the Runbook variables
    • Metric Data: Show a graphical view of a metric over time
    • Repository SQL: Allows you to run a SQL statement against documented Enterprise Manager repository schema views (MGMT$)

      Note:

      Accessing other elements from the Enterprise Manager repository schema is not supported.
    • Target SQL: Allows you to execute SQL against any database target in Enterprise Manager

    • OS Commands and Scripts: Allows you to execute a single operation, as you would do on the command line or to specify an interpreter to be used to execute a script which you enter as a part of the step definition.

    For more information about step types and how to use them, see Step Types.

  6. When you select a step type, the step authoring UI displays. Enter a step title and description of what the step will do, instructions, and any other information regarding interpretation of results.
  7. By default, the step will be prepopulated with settings from the context of the incident or metric. Depending on the step type, you can select other step parameters such as Metric Group Name by clicking on the current selection to display the pop-up selector and choose a new value from the drop-down list.

    Note:

    At the right, you can choose from a list of variables pulled from the incident context. If none of the variables meet your requirement, you can create your own variable by clicking Add a Runbook Variable.

    When defining a step, you can assign data from the output of the step to populate values of existing author-defined Runbook variables.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) details, such as target name and internal target type, for a different target instance, can be set directly or through the use of Runbook Variables.

  8. Save to Variable (Optional)

    When defining a step, you can dynamically assign data from the output of the step to populate values of author-defined Runbook variables, following the steps below:

    Note:

    A Runbook Variable can be assigned a value from only one cell of one single step's output.
    1. Click Save to Variable to open the tab.
    2. From the output of the step, click the cell containing the value you want to save to a variable. Once you do so, you will see a Save to Runbook Variable dialog appear.
    3. In the Save to Runbook Variable dialog box, click to open the dropdown list providing the available variables, and select the one you want to contain the data, then click Save. If you are saving a value to a Runbook Variable from another step, your author-defined Runbook Variable will not show up in the dropdown list. If the list is empty or you want to create a new variable, click Create New Runbook Variable at the top of the list, and use the pop-up window to create a new variable.

    The Saved Variables list, on the right side of the Save to Variables tab, shows the variables that take data from the step output.

  9. Links to EM Pages in Step Instructions (Optional)

    Use the MarkDown language to specify a link to an EM page. To construct the link, navigate to an EM page and then copy the part of the URL which begins with " / " after the host and port name. For example, if the url is http://my.emsite.com:1234/em/faces/si-host-home?type=oracle_database&target=Oemrep_Database then you could use it as a link as [Target Home Page](/em/faces/si-host-home?)

    Other examples:

    [Runbook Page](/em/faces/runbookPage)
    [Target Page](/em/faces/core-uifwk-targetSearch-home)

    You can use also use variables in your URLs. For example, if your target Oemrep_Database is type=oracle_database, they can be put into variables target_type and target_name respectively such that the link would be [Target Home Page](/em/faces/si-host-home?type=$target_type&target=$target_name)

  10. Click Run to execute the step and view any rendered visualizations such as charts or tables defined by the current parameter settings. You can run multiple tests to ensure that your settings provide the desired results. Once the step is run, you will see that any values selected to be saved to Runbook Variables will now be assigned to those Runbook Variables.
  11. Click Save Step. The step is now ready to use and will be added to the list of steps.
  12. Once you've finished adding steps, you've created a Runbook draft as it's still editable. It will appear in the Runbooks list page. (From the Enterprise menu, select Monitoring and then Runbooks). In draft mode, only the author can view/edit the Runbook.

Step Types

You can create the following types of steps:

Note:

The following step limitations apply to all step types:
  • 1MB of data (configurable via the oracle.sysman.core.runbooks.maxStepDataSizeKB is the OMS property)
  • 20 steps per Runbook (configurable via the oracle.sysman.core.runbooks.maxSteps OMS property)
  • Note Step

    Any type of text meant to provide information or instruction pertaining to the Runbook. The Note can be plain text, but can accommodate simple formatting via markdown language. A markdown reference table showing supported formatting is provided when creating a Note step.

    Input:

    1. Any type of text
    2. Basic formatting of text (markdown)
      1. bulleted lists (1 level)
      2. numbered lists (1 level)
      3. bold text
      4. italic text
      5. bold, italic text
      6. Header (h3, h4, h5)
      7. horizontal rule
      8. blockquote
    3. Runbook Variables
    4. External Links

      You can specify two types of external links within a Note step using the following markdown notation:

      Links to a specific URL: [Link Text](http//www.anothercompany.com)

      Links to My Oracle Support notes (only the MOS ID is required): [MOS Link Text](MOS:MOS ID). For example [Enterprise Manager configuration issues](MOS:1234567.1)

      Links to EM pages: [Target Page](/em/faces/core-uifwk-targetSearch-home)

      Note:

      External linking is available starting with Enterprise Manager 13c Release 5 Update 11 (13.5.0.11).
    5. Links to other steps

      Using markdown language, links to other steps can be added using (GOTO:step number).

      Here is an example which adds a link to another step based on the step input using an if-else clause:

      If the value is less than 100, (GOTO:5)
      Else, move ahead to the next step.

    Output:

    Formatted text with Runbook Variables substituted out for values.

    Example:

    Remember to have a named credential for the target database **$ora_target_name** on which the FRA incident has occurred. The Note step will show:Remember to have a named credential for the target database orcl_database on which the FRA incident has occurred.

  • Metric

    This step shows the time series chart for the specified metric. Defaults to the metric from passed in context, but it can be changed. When the Runbook is launched in context of an incident/event, the chart also shows when the event occurred.

    With Oracle Enterprise Manager 13c Release 5 Update 17 (13.5.0.17), the chart shows lines representing the current warning and critical thresholds for the metric. This allows you to evaluate the metric trend against warning and critical thresholds.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) details, such as target name and internal target type, for a different target instance, are available to be typed directly in the target selector input box or set through the use of Runbook Variables.

    Note:

    Configuration metrics are not supported.

    Input

    1. Target name/specification
    2. Metric Group
    3. Metric
    4. Metric Key Values
    5. Time range notation:

      You can also specify a metric time span by providing start and end dates. You can make those times relative to now or evt_time.

      • now corresponds to current time.
      • evt_time corresponds to the date and time when the event occurred.

      Well known Runbook variable named ora_event_time from the incoming context corresponds to the evt_time. Use the keyword now or evt_time followed by + or - and finally an integer and m for minutes, h for hours, or d for days.

      Examples:

      now : current time when the step is run

      now - 6h : 6 hours prior to the current time

      evt_time + 3d : 3 days after the time the event occurred

    Output:

    1. Time series line chart display
    2. Legend
    3. Links to other steps

      Using markdown language, links to other steps can be added using (GOTO:step number).

      Here is an example which adds a link to another step based on the step input using an if-else clause:

      If the value is less than 100, (GOTO:5)
      Else, move ahead to the next step.

    Note:

    If the Runbook was launched in context of an incident/event, the chart will show when the incident occurred so long as the specified time range includes when the incident happened.

  • Repository SQL

    You provide a SQL statement to be run against MGMT$ views in the repository schema. You are limited to running the SQL against the repository view tables (MGMT$ or GC$). You can use BIND parameters. There is a limit on step size data: The maximum limit for a SQL query is 20 columns and 1000 rows, or 1 MB of data (whichever limit is reached first).

    Using Bind Parameters

    You can bind values from Runbook Variables into your SQL statement using a colon followed by the Runbook Variable reference. For instance:

    select target_name from MGMT$TARGETS where target_type = :$ora_target_type_name

    This will create a bind parameter in the SQL statement and use whatever the value of $ora_target_type_name is as the value for that bind parameter when the SQL is executed.

    Starting with Enterprise Manager 13c Release 5 Update 21 (13.5.0.21) you can use a repository SQL step to find the details of another target instance (target name, and internal target type). These values can be assigned to Runbook variables, that can be used in the metric step edit screen to drive the values of the step.

    SQL Query Resultset formatting

    1. If the query column list contains a clob, wrap the CLOB column_name in TO_CHAR() function.

      Example:

      SELECT em_varchar_col, to_char(clob_column) from MGMT$CLOB_TABLE
    2. If the query column list contains a timestamp, wrap the date column name in a TO_CHAR() function and provide the proper date format mask.

      Example:

      SELECT em_varchar_col, to_char(em_timestamp_col,'DD-MON-YYYY HH24:MI:SS') from MGMT$DATE_TABLE
    3. If a user defined variable is a string that represents a date and the variable is used in the query WHERE clause, wrap the variable name in a TO_TIMESTAMP() or TO_DATE() function and provide the corresponding date format mask.

      Example:

      SELECT em_varchar_col,to_char(em_timestamp_col,'DD-MON-YYYY HH24:MI:SS') FROM MGMT$DATE_TABLE where em_timestamp_col
      < to_date(:$l_user_defined_date_variable,'DD-MON-YYYY')
  • Target SQL

    Ability to run predefined SQL (single statement only) on any database target. In addition to executing select statements, you can run Data Manipulation Language (DML) and Data Definition Language (DDL). The named credential is required whether it is a select statement or DDL/DML.

    Pre-requisites:

    In order to run the Target SQL step on a database target, you will need to have these privileges on your database targets:

    1. Run any sql on Database
    2. Connect Target
    3. Access to the named credential referenced in the step

    Input

    1. Target Name
    2. Target Type
    3. SQL
    4. Named Credential

    Using Runbook Variables

    You can use Runbook Variables in your Target SQL queries, DML or DDL statements.

    1. SQL queries (SELECT statements):

      Using Bind Parameters you can bind values from Runbook Variables into your SQL statement using a colon followed by the Runbook Variable reference.

      Example:

      select target_name from MGMT$TARGETS where target_type =:$ora_target_type_name

      This will create a bind parameter in the SQL statement and use whatever the value of $ora_target_type_name is as the value for that bind parameter when the SQL is executed.

    2. DML or DDL statements

      You can use Runbook Variables in your DML or DDL statements by using a colon followed by the Runbook Variable, such as :$ora_target_name or :$fra_size.

      Example:

      alter system set db_recovery_file_dest_size=:$fra_size  scope=both

      It will substitute the value of the variable before executing the DML or DDL statement.

    Using ISO 8601 formatted Runbook variables in SQL Query Where Clause

    If a Runbook variable's value is in ISO 8601 date format (such as 2022-03-01T19:21:00.000Z) and you need to use it in a where clause on a database table/view with date column:

    1. Use to_utc_timestamp_tz function to convert the value into a database timestamp with time zone value.
    2. Value from #1 can be converted to another time zone (such as target time zone) as needed.
    3. cast the output from #1 or #2 into date type, and use it as part of the where clause to join with tables having date column.
    4. Following examples assume the utc_time1 Runbook variable is set to a value in ISO 8601 date format (such as '2022-03-01T19:21:00.000Z').

      Examples:

      Example #1: Shows how to cast the variable value to date without taking timezone conversion into account.

      
      select to_char(min(collection_timestamp),'YYYY-MM-DD HH24:MI:SS') as min_time, to_char(max(collection_timestamp), 'YYYY-MM-DD HH24:MI:SS') as max_time, count(1) as total_count
      from mgmt$metric_current
      where collection_timestamp >= cast(TO_UTC_TIMESTAMP_TZ(:$ora_event_time_utc) as date) 

      Example #2: Shows how to convert the variable value to target timezone first, then cast it to date;

      
      select to_char(min(collection_timestamp),'YYYY-MM-DD HH24:MI:SS') as min_time, to_char(max(collection_timestamp), 'YYYY-MM-DD HH24:MI:SS') as max_time, count(1) as total_count
      from mgmt$metric_current
      where collection_timestamp >= cast((TO_UTC_TIMESTAMP_TZ(:$ora_event_time_utc) at time zone TIMEZONE_REGION) as date)

    Output

    1. Tabular display of return data
  • OS Commands and Scripts

    Allows you to execute a single operation, as you would do on the command line on the host of the specified target. or to execute a script which you enter as a part of the step definition using the specified interpreter.

    Pre-requisites:

    In order to run the OS Command and Scripts step, you will need to have these privileges:

    1. View privilege on the target, target's host and the agent that monitors the target's host
    2. Access to the named credential referenced in the step

    Input

    1. Target Name
    2. Target Type
    3. Named Credential
    4. Command Execution Type:

      Single Operation

      1. Command

        Note:

        The interpreter will be defaulted for whatever the OS is on the host.

      Script

      Note:

      For Windows platform, Script as a Command Execution Type is currently not supported.
      1. Script

      2. Interpreter

        Note:

        If left blank, the interpreter is defaulted according to the OS on the target host where the script is executed.

    Output:

    Returns the output from running the command or script on the target host.

Creating Runbook Variables

Runbook Variables are populated with values from the incident details when a session or draft is created. These values can be used by steps for display in the instruction text or as a part of the step definition which gets data or performs some task. This allows a Runbook session to provide information and perform tasks based on context specific to that session. And for a draft, it allows an author to define a Runbook using the Runbook Variables rather than hardcoded values.

For values which are not included as context when the draft is created, the author can create their own Runbook Variables. This Runbook Variable can be populated in these ways:

  • The runbook Variable can have a static value which remains the same across the sessions that will eventually be created on that Runbook

  • The Runbook Variable can require the session user to provide a value each time a new session is created for the Runbook. For instance, for target DB SQL steps, a named credential is required. Creating a Runbook Variable for this named credential and requiring the value to be set by the session user ensures that the SQL as a part of that step will be executed with a named credential the session user has access to for that session’s target DB.

  • The Runbook Variable can also by dynamically populated with data from the output of a step. This feature makes the use of Runbook Variables more powerful and is available starting with Oracle Enterprise Manager 13c Release 5 Update 18. As an example, you may have a step that determines the filesystem used by the database. The author can save the output of that step into a Runbook Variable, and then use that Runbook Variable in the next step that determines the available free space of the filesystem. Note that a Runbook Variable can save data from a specific value cell of a step's output, hence if you need to save multiple data values, you'll need to save these in multiple Runbook Variables.

There are some restrictions around the naming of Runbook Variables an author can create:

  1. It cannot start with ora_.
  2. It can contain only letters, numbers, and underscores.
  3. It cannot be longer than 64 characters.
  4. Additionally, for credentials, _cred should be included in the name (this allows us to filter for the specific case of named credentials used in the target DB step)

When creating a Runbook Variable, the author will provide:

  1. A name, which is how the Runbook Variable will be referenced in the step (for instance, user_target_db_cred)
  2. A display name, which is a short, user readable name (for instance, Target Database Named Credential)
  3. A description (optionally), which will be displayed in the session UI when the session user is asked to fill in a value (for instance, Provide a named credential for this target database)
  4. Value to be used in the draft, which allows you to create the Runbook Draft with a value the author can use to execute the step
  5. Options for Value used in a Runbook Session:

    There are options for how the value will be populated in a Runbook Session. The table below has a summary of these options and usage guidelines.

Value used in a Runbook Session When to use this option How the runbook variable value is populated Can runbook session user change it? Usage Notes
Specified by session user or by running a step Use this option when you want the runbook session user to specify the value of the runbook variable. Typically, this is done by running a runbook step to populate the runbook variable. For advanced users, the session user can manually enter the runbook value if needed. In a runbook session, the user runs a step to populate the runbook variable or he types in a value for the variable. Yes - by typing in a value or by running a step The initial value of the variable is empty (null).
Value provided by author or by running a step Use this option if you don't want the session user the ability to manually specify the runbook variable value. Instead, you want the session user to run a step to populate the value or you want to define the value as part of the runbook definition. The runbook author defines the value as part of runbook definition or he has the user run a step to populate it. Yes - only by running a step, if the runbook author creates a step that populates the runbook variable If the author expects the value of the variable to be populated by running a step, the initial value of the variable is null.

Graphic shows the Add a Runbook Variable dialog.

Publishing Runbooks

Once you are finished defining your Runbook draft, you're ready to publish it. Once published it will become available to all Runbook users. Only the author of the Runbook draft can publish it.

Navigate to the Runbooks list page (Enterprise->Monitoring->Runbooks) and go to the Drafts tab. From the Actions menu for your Runbook, select Publish. Select Keep Draft if the Runbook is still in development or the draft is still relevant, and click OK. Click on the Published tab to view your Runbook in the published Runbook list.

Creating New Versions of Runbooks

  • Create a draft from a published Runbook. Navigate to the Runbooks page. From the Actions menu for the desired Runbook, select Create New Version.
  • While the draft is being edited, the published version of the Runbook remains available for users to continue to create sessions.
  • The Runbook author will be required to provide a context ID to do so.
  • When the new draft is published, it replaces the previous published Runbook after user confirmation.
  • This new published Runbook will be used to create all future Runbook sessions. It has no impact on prior Runbook sessions.

Creating New Runbooks from Existing Runbooks

To save time and effort, you can create a new Dynamic Runbook from an existing one by using the Create Like action. Create Like can be used on either published or draft Runbooks.

Using Published Dynamic Runbooks

Note:

You do not have to be the owner of the published Runbook to use the Create Like action.
  1. From the Enterprise menu, select Monitoring and then Runbooks.
  2. Click on the Published tab to view the table of published Runbooks.
  3. From the Actions drop-down menu for the chosen Runbook, select Create Like. The Create Like dialog is displayed.
  4. Enter a context ID to use to create the new draft Runbook.
  5. Click OK. The Runbook editor screen is displayed with a copy of the published runbook in draft mode. As the owner of this draft, you can edit this runbook in the context of the specified context ID.

Using Draft Dynamic Runbooks

Since you own the draft version of the Runbook, a more complete copy can be made. You do not need to provide an context ID since the selected draft Runbook will already have that information.

  1. From the Enterprise menu, select Monitoring and then Runbooks.
  2. Click on the Drafts tab to view the table of draft Runbooks that you own.
  3. From the Actions drop-down menu for the chosen Runbook, select Create Like. The Create Like confirmation dialog is displayed.
  4. Click OK. The Runbook editor page displays the new draft Runbook.

Other Operations

After you've published the Runbook, you can perform other operations on it from the Actions menu.

You can:
  • Hide/Unhide the Runbook to control availability of published Runbooks for public use.
  • Delete the published Runbook.

Note:

These operations can only be performed by the Runbook author or Runbook administrator.

Adding External Links

You can add external links to information that may assist a future Runbook user with additional diagnostic or contextual information.