Atlassian Incident Handbook

Who is this guide for?

If you're on a development or operations team that looks after internet services for customers who require 24/7 availability, this handbook is for you.

What is an incident?

We define an incident as an event that causes disruption to or a reduction in the quality of a service which requires an emergency response. Teams who follow ITIL or ITSM practices may use the term major incident for this instead.

An incident is resolved when the affected service resumes functioning in its usual way. This includes only those tasks required to restore full functionality. 

The incident postmortem is done after the incident to determine the root cause and assign actions to ensure it is addressed before it can cause a repeat incident.

Our incident values

A process for managing incidents can't cover all possible situations, so we empower our teams with general guidance in the form of values. Similar to Atlassian's company values, our incident values are designed to:

• Guide autonomous decision-making by people and teams in incidents and postmortems. 
• Build a consistent culture between teams of how we identify, manage, and learn from incidents.
• Align teams as to what attitude they should be bringing to each part of incident identification, resolution, and reflection.

Tooling requirements

The incident management process described here uses several tools that are specific to Atlassian and can be substituted as needed:

• Incident tracking - every incident is tracked as a Jira issue, with a followup issue created to track the completion of postmortems (Atlassian uses a heavily customized version of Jira Software for this).
• Chat room - a real-time text communication channel is fundamental to diagnosing and resolving the incident as a team.
• Video chat - for many incidents, team video chat like Blue Jeans can help you discuss and agree on approaches.
• Alerting system - a tool such as OpsGenie manages on-call rotations and escalations.
• Documentation tool - we use Confluence for our incident state documents and sharing postmortem via blogs.
• Statuspage - communicating status with both internal stakeholders and customers through Statuspage helps keep everyone in the loop.

Incident tracking

Every incident is tracked as a Jira issue, with a followup issue created to track the completion of postmortems. The process in this handbook references our heavily customized version of Jira Software.

Incident issues are typically created by a support engineer in response to a customer ticket or by a developer recognizing a monitoring alert as being an incident. We urge people to create an issue if they're worried about something, rather than wait to escalate it.

In Jira, we have a simple workflow to track incidents through the resolution stage and to record all important actions taken during the incident response.

Incident manager

Each incident is driven by the incident manager (IM), who has overall responsibility for and authority for the incident. This person is indicated by the assignee on the incident issue. The incident manager is empowered to take any action necessary to resolve the incident, which includes paging anyone in the organization and keeping those involved in an incident focused on restoring service as quickly as possible. 

The incident manager is a role, rather than an individual on the incident. The advantage of defining roles during an incident is that it allows people to become interchangeable. As long as a given person knows how to perform a certain role, they can take that role for any incident.