Incident Alert Management Guide
Incident Alert Management Guide
Incident Alert Management Guide
PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information. PDF generated at: Sun, 16 Mar 2014 17:26:36 PST
An example of this process is: 1. An ITIL user creates a high-priority incident regarding a serious issue with the server room. 2. The incident alert administrator creates a new incident alert for this source incident. 3. As a result of a conference call discussion, the incident is assigned to the problem management team, which agrees to investigate further and identify tasks to improve service and prevent the incident from happening again. 4. The problem management team resolves the source incident. The source incident may also be closed at this point. 5. The incident alert administrator resolves the incident alert. 6. The incident alert administrator convenes a post incident review meeting to ensure that all identified tasks are logged and tracked to completion. 7. The incident alert administrator can now close the incident alert.
Roles
Role Title [Name] ITIL user [itil_user] Description Can view the dashboard and incident alerts. Can subscribe to incident alerts.
Incident alert administrator [ia_admin] Can create and edit incident alerts and contact records.
Tables
Incident alert management adds or modifies the following tables.
Display Name [Table Name] Impacted CI [impacted_ci] Incident Alert [incident_alert] Description The CIs which have been impacted by the incident alert's source CI. The base table for incident alerts.
Plugins
The following additional plugins are activated with incident alert management.
Plugin Name Plugin ID Description
Contact Management com.snc.contact_management Provides contact functionality and enables contact administration for incident alerts.
The following additional plugins can optionally be installed to provide additional functionality.
Plugin Name NotifyNow Plugin ID Description
com.snc.notifynow Provides functionality to send SMS notifications and set up ad-hoc conference calls for an incident alert.
Properties
Incident alert management adds the following system properties.
Name com.snc.iam.log_level Description Logging level for the business rule MapUpstreamImpactedCI. Debug is the most detailed option with full trace of how the Impacted CI List is calculated. Error is the minimal logging option with only severe errors being logged Type: String Default value: info Possible Values: debug,info,error Location: System Properties [sys_properties] table
glide.ui.incident_alert_activity.fields Incident alert activity formatter fields. This is the list of fields tracked from the incident alert form in the activity formatter. Type: String Default value: opened_by, work_notes, comments, severity, estd_distruption_time, actual_disruption_time Location: System Properties [sys_properties] table
User Roles
Incident alert management adds the following user roles.
Role ia_admin Contains Roles notifynow_admin, contact_admin Description Can create and edit incident alerts, and manage contact information. The notifynow_admin role is only contained in ia_admin if Notify is active. [Requires ia_admin role] Can create and edit contact definitions and contact responsibilities. [Requires ia_admin role] Can view contacts, contact definitions, contact responsibilities and default overrides.
Note: Typically, incident alert administrators may need to have both ia_admin and itil roles, to have full access to incident alert functionality. For example, the itil and ia_admin role are both needed to be able to create incident alerts from within an incident form.
UI Actions
Incident alert management adds the following UI actions.
UI Action Create new incident alert Show Live Feed (1) Tables incident [incident] Description Creates new incident alert from an existing incident record.
Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert]
Follow on Live Feed (1) Follow on Live Feed (2) View PIR Report
Adds user to the live feed for this document. If no feed exists, it is created. This is for lists, forms have the redirect. Adds user to the live feed for this document. If no feed exists, it is created. This is for forms using the redirect. Shows the post incident review report.
Initiate Conference Call Incident Alert [incident_alert] Initiate a conference call for a incident alert.
UI Policies
Incident alert management adds the following UI policies.
UI Policy
Table
Make PIR section source incident fields read only Incident Alert [incident_alert] Closure info Resolution Info Capturing open / closed / resolved info Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert]
Script Includes
Incident alert management adds the following script includes.
Script Include Description
IncidentAlertConferenceCall Returns a list of frequent participants that have joined Notify conference calls.
Client Scripts
Incident alert management adds the following client scripts.
Script PIR visibility Table Description
Adding info from Source Incident Incident Alert [incident_alert] Bring in information from source incident.
Business Rules
Incident alert management adds the following business rules.
Business Rule Name Incident Alert insertion limitation Table Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert] Incident Alert [incident_alert] Contact [contact] Incident Alert [incident_alert] Description Only allow one active incident alert to be associated with an incident.
MapUpstreamImpactedCI
Make sure an incident alert can only be created with in a New state.
Automatically change the incident alert state to Work In Progress when comments are added. Make sure that the logged in user is an incident alert administrator. Map all impacted configuration items based on source CI.
Description Send an SMS notification when an incident alert is created. Check if a conference call can be initiated. Extend the Incident Alert activity log when a conference call finishes. Extend the Incident Alert activity log when a conference call is initiated.
Update Conference Call Finished IA NotifyNow Conference Call Activity [notifynow_conference_call] Update Conference Call Started IA Activity NotifyNow Conference Call [notifynow_conference_call]
3. Click Submit.
Field Number Severity Source incident Description Automatically generated incident alert ID, in the format IAxxxxxxxx. The severity for the incident alert. Values are Major, High, Medium, or Low. The source incident for this alert, if any. If you select a source incident, the Source CI, Short description, and Background fields are populated with data from this incident, unless there is existing data in those fields. The state of the alert. Values are New, Work In Progress, Resolved, Cancelled, or Closed. The source CI for this alert, if any. If there is a source incident selected, this field is populated with the source CI attached to that incident. If there is no source incident selected, select the source CI manually, if applicable. If the source CI has related CIs, these are automatically listed in the Impacted CIs related list. The assignment group, if any, for that incident alert. For example, there might be a group that represents a crisis management team, including a number of Incident Managers, Duty Directors and Duty Managers. The type of event. Values are: Outage, Degradation, Capacity, SLA/Delay, or Fail-Over. The assigned user for the alert. This can be an ITIL user or an incident alert administrator, and defaults to the user who creates the alert. Yes or No to indicate whether the business or a service is impacted.
State Source CI
Assignment group
Business/Service impact Short description Details section Opened Opened by Estimated disruption time Description
When the alert was created. The creator of the alert. This defaults to the user who creates the alert. The estimated duration of the disruption.
The details of all actions taken while working on the alert. Any separate work notes relevant to the alert that might help in communications.
Impacted CIs
The Impacted CIs related list shows all the CIs that the CMDB shows as related to the source CI for this alert.
Administrators and incident alert administrators can modify this list. Click the Edit button, then add and remove CIs, as appropriate.
Note: Administrators can adjust the com.snc.iam.log_level property to view more log information for how this list is determined. By default the value is info. Set this to debug to see more detailed log information.
User Contacts
After an incident alert is created, the following default user responsibilities are added to the User Contacts related list: Duty Manager Duty Director Incident Manager
From this list: Click New to add a new contact. Click the lookup icon beside the responsibility entry to edit the details for that responsibility. Select the check box for the entry, then select Actions on selected rows.. and click Delete, to delete that entry from the user contacts list. For more information, see Using Contacts with Incident Alerts.
Group Contacts
There are currently no default group contacts defined for incident alert management. However, you can define group responsibilities for your organization, then personalize the form layout to add the Group Contacts related list. You can then edit and modify this list, as for the user contacts.
This information is read-only. To make changes to this information, update the source incident.
10
Notify
If Notify is active, two additional related lists appear on the Incident Alert form. The SMS Messages related list gives information about the SMS notifications sent to users identified as contacts on the incident alert. For example, by default, SMS notifications are sent to users who are assigned to responsibilities when an incident alert is created. The SMS message content depends on the fields that were filled in when the alert is created, but is generally in the following format: IA0000001: a <severity> severity <event type> incident alert for '<CI name>' has been opened
Note: The CI name may be truncated to keep the content within 160 characters.
The Conference Calls related list shows details of any conference calls that have been launched for the incident alert. For more information, see Using Notify with Incident Alert Management.
Resolving an Alert
Typically, when the event that initiated the incident alert is resolved, the incident alert can also be marked as resolved. When an alert is resolved, the following fields are added to the Incident Alert form:
Field Resolved Resolved by Actual disruption time Description The date and time when the alert was resolved. Automatically set when the form is saved, but can be changed later. The user who resolved the alert. Automatically set when the form is saved, but can be changed later. The amount of disruption time recorded, based on the time between when the incident alert was created and the time it was marked as resolved. Information for discussion and review. For more information, see Running a Post Incident Review.
11
The Source Incident Details section contains read-only information, taken from the source incident Fill in the Incident Alert Details fields as follows:
Field Resolution code Resolution notes Summary Lessons learned Description [Required] Whether the incident alert has been completed. Values can be Complete, Complete with Actions, or Not complete.
[Required] Any notes about the resolution of the incident alert. After a user enters information in the resolution notes and saves the record, both the Resolution notes and Resolution code are set to read-only. A summary of the incident alert. Any lessons learned from the review process.
Use the View PIR Report related link to create a report that can be circulated or printed for the post incident review meeting.
12
Closing an Alert
Typically, when the post incident review is complete, the incident alert can be closed. To close an alert, mark the state as Closed.
The following values are then set in the Details section of the alert.
Field Closed Description The date and time when the alert was closed.
Open Alerts: displays all open alerts. Click an alert number to open the details for that alert. Open Alerts By Severity: groups open alerts by severity levels, as defined in the Incident Alert form. Open Alerts by Type: groups open alerts by event type, as defined in the Incident Alert form. Active Conference Calls: displays any active conference calls. This appears only if Notify is active.
13
4. Alter parameters, as required, and click Run Report to run the revised report.
Report Name Description Contains Number, Created by, Event Type, Severity, Title, Open time, Estimated Disruption time, Related Record, Assignee.
IAs opened in the All alerts, of any state, which have been opened in the last last 72 hours 72 hours. Open Alerts
Displays all open alerts. Click an alert number to open the Number, Severity, Short description, Source incident, State, details for that alert. Displayed on the dashboard by default. Business/Service impact, Assigned to. Groups open alerts by severity levels, as defined in the Incident Alert form. Displayed as a pie chart on the dashboard by default. Groups open alerts by alert type, as defined in the Incident Alert form. Displayed as a bar chart on the dashboard by default. Severity.
Event type.
All open alerts which have been created in the current week. Number, Created by, Event Type, Severity, Title, Time Created, Estimated Disruption time, Related Record number, Incident Manager. All alerts which have been resolved. This does not include closed alerts. Number, Resolved by, Event Type, Severity, Title, Actual Disruption time, Source Incident number, Source Incident status. Number, Resolved by, Event Type, Severity, Title, Actual Disruption time, Related Record, Assignee.
Resolved Alerts
Resolved IA's this All alerts which have been resolved in the current week, Week including any alerts closed this week.
14
To launch a conference call for an incident alert: 1. 2. 3. 4. Navigate to Incident Alert Management > Open. Open the relevant incident alert. Click the Initiate Conference Call related link. Within the dialog box that appears, select the participants for the conference
15
The dialog box displays the frequently-called and selected participants for the conference. All users from the User Contacts list in the incident alert are selected by default. Calls are placed to the number in the Mobile phone field on the user record. If that information is blank, the user cannot be contacted through Notify. The mobile phone number has to be an E.164 [1] compliant phone number. If the phone number is a local number, without the + prefix, the number will be retrieved based on the user's location and, if possible, converted into a valid E.164 number. 5. To select ad-hoc participants, do one of the following: Click the reference lookup icon, select the relevant user, and click Add to selected. Enter the participant's phone number in the field beside the telephone icon, and click Add to selected. 6. After the participant list is finalized, click OK 7. The conference call starts and a Conference call initiated message is displayed at the top of the Incident Alert form. Each user is called and can accept the call to join the conference. Note: Several response types are possible from users invited to join the conference call, apart from Accepted. 8. Click the Conference call initiated message to see details of that conference call.
Using Notify with Incident Alert Management 9. When the final participant leaves the conference, the conference call closes.
Note: VoIP phone systems, which do not use touch tone phones, may encounter issues with recognizing key presses. To avoid problems, ensure that conference call users use touch tone phones, or configure your VoIP system settings to recognize key presses, as described in your VoIP system documentation.
16
Adding Participants
If the conference participants decide that another user's input is required, that user can be invited to join the current conference call. Participants who may have involuntarily dropped out of the conference can also contact the conference call initiator, who can add them to the conference call. To add an ad-hoc participant to an active conference call: 1. Open the form for the relevant active conference call. 2. Click the Invite to Conference Call related link.
3. Select participants as described for launching a conference call. 4. The selected participant is called directly and can join the conference.
17
Note: Conference call information can also be accessed by navigating to NotifyNow > Conference Calls.
Unanswered Any other action, for example, missed call, or the contact took another action.
Note: Depending on the contact's phone service provider, the information the participant receives may vary. For example, contacts who have switched off their phones may or may not receive a missed call message.
18
References
[1] http:/ / en. wikipedia. org/ wiki/ E. 164
Contact Administration
Overview
Contacts allow incident alerts to be associated with users and groups based on conditions defining the reason for association: for example, the ownership of that incident alert. Multiple users and groups can be assigned as contacts. You can assign users or groups to incident alerts automatically based on the information provided in these records: Contact responsibilities: these provide a name, such as Incident Duty Manager, for a set of tasks related to incident alerts. The contact responsibility record also indicates whether those tasks are performed by an individual user or a group of users. Contact responsibilities can also be used to manually add contacts to an incident alert. Contact definitions: identify a set of conditions to determine which specific user or group is assigned to handle particular responsibilities for an incident alert. For example, All P1 Incidents must have an Incident manager, assigned to US Incident Management group. Contact responsibilities and contact definitions allow you to define and modify data-driven contact information for automatic notifications, rather than specifying individual users or groups directly for each incident alert.
Default Responsibilities
By default, contacts with the following responsibilities are notified when an incident alert is created: Duty Manager Incident Manager Duty Director These roles are involved with resolving the source incident or original event that the incident alert relates to, and so are seen as key contacts for the incident alert. The following sections describe typical operational roles for these responsibilities.
Contact Administration
19
Description The senior point of presence in the monitoring environment at the time an incident occurs. The Duty Manager assesses the incident against standard operating procedures, escalation triggers and personal knowledge and experience, to take corrective actions. To clarify the urgency and impact of an incident, the Duty Manager can contact the Incident Manager for advice. A senior technician, accountable for coordinating and managing all technical resources required to resolve incidents. After being notified by the Duty Manager of a serious incident, the Incident Manager assesses the seriousness and associated business impact. Based on this assessment, the Incident Manager decides whether to escalate the incident to the Duty Director. The Incident Manager may escalate to the Duty Director to gain access to resources outside of the department, if necessary. The escalation point for all issues that affect critical services The Duty Director works in partnership with the business directors in the organization to approve recovery plans developed by the Incident Manager, and to manage the senior level communications for the source incident.
Incident Manager
Duty Director
Other Responsibilities
Incident alert management provides the following additional responsibilities that can be added to incident alerts. You can also create contact responsibilities, as needed. The associated users receive notifications about the alert.
Title Business Director Communication Manager Crisis Action Manager Crisis Action Team Member Development Operations Service Owner Description Director within the business who is identified as a potential contact in the the event of an incident alert. Business-facing role in the event communication is required in an incident alert. Overall responsibility and accountability for managing incident alerts. Nominated department heads who are involved when an incident alert occurs.
Development personnel involved in the troubleshooting and resolving an incident alert. Second or third level operations support involved in troubleshooting and resolving an incident alert. Service owner or manager who is identified as a potential contact in the the event of an incident alert that relates to one or more of their services. Second or third level technical support personnel involved in troubleshooting and resolving an incident alert.
Technical Support
20
Name Type
The responsibility name. User or Group to indicate whether the responsibility appears in the User Contacts or Group Contacts related list of the Incident Alert form.
21
The field on the Incident Alert form that identifies the contact associated with the selected contact responsibility. Appears only when Form field is selected as the value for Source. For user contact types, values can be Assigned to, Closed by, Opened by, or Resolved by. For group contact types, the value is Assignment group.
Responsibility The contact responsibility associated with this definition. Quantity The maximum number of contacts that can be associated with the selected Responsibility per incident alert record. This field appears only when None is selected as the value for Source. A check box to indicate whether the definition is active or not. The conditions that must be met to associate this contact definition to a particular user or group. For example, ]Affected users] + [is] + [0-25]. If multiple conditions are defined, each condition is evaluated in the order listed.
Active Condition
22
Field Order User value The order in which the condition is to be evaluated. The user to assign as that contact if the condition matches.
Description
If the definition type is set to Group, this field is labelled Group value and defines the group to assign as that contact.
Condition The conditions defining whether the default override is to be applied. If multiple conditions are defined, each condition is evaluated in the order listed. If no conditions match, this default override is not applied.
4. Select a Responsibility and the User to have this responsibility for this incident alert. That contact information is now listed in the incident alert's User Contacts related list.
Note: If you delete an incident alert, all contacts associated with that incident alert are also deleted.
23
Notifications are sent by email. If Notify is active, notifications can also be sent by SMS message or voicemail. For example, a business manager does not log in to the system on daily basis, but needs to know when a new incident alert is created. The business manager can subscribe to receive notifications whenever a new incident alert is raised.
Subscribing to Notifications
1. 2. 3. 4. 5. Navigate to Self-Service > My Profile. Select Notification Preferences under Related Links. Under the device to receive notifications, click in the area labeled To subscribe to a new notification click here. Click the lookup icon beside Notification Message to display a list of available notifications. Select one of the following notifications:
New IA Raised IA Actions Taken IA Resolved Or Closed IA Cancelled 6. Fill in the details for the selected notification.
7. Click Submit. 8. The notification is then listed in the Notification Preferences list.
24
Filtering Notifications
If no filtering is applied to a subscription, then a subscribed user receives all notifications for that subscription. For example, a user subscribed to New IA Raised, with no filtering, receives notifications every time any incident alert is created. To make the notifications more relevant, select Advanced filter, then use the condition builder to create an appropriate filter. For example, you can choose to be notified only when an incident alert is created for a specific CI.
25
26
27