Incident postmortems

What is postmortem?

A postmortem is a written record of an incident that describes:

• The incident's impact.
• The actions taken to mitigate or resolve the incident.
• The incident's root cause.
• Follow-up actions taken to prevent the incident from happening again.

At Atlassian, we track all postmortems with Jira issues to ensure they are completed and approved. You may decide to use a simpler system, like a Confluence page for each postmortem, if your needs are less complex.

Why do we do postmortems?

The goals of a postmortem are to understand all contributing root causes, document the incident for future reference and pattern discovery, and enact effective preventative actions to reduce the likelihood or impact of recurrence.

For postmortems to be effective at reducing repeat incidents, the review process has to incentivize teams to identify root causes and fix them. The exact method depends on your team culture; at Atlassian, we've found a combination of methods that work for our incident response teams: 

• Face-to-face meetings help drive appropriate analysis and align the team on what needs fixing.
• Postmortem approvals by delivery and operations team managers incentivize teams to do them thoroughly.
• Designated "priority actions" have an agreed Service Level Objective (SLO) which is either 4 or 8 weeks, depending on the service, with reminders and reports to ensure they are completed.

Attending to this process and making sure it is effective requires commitment at all levels in the organization. Our engineering directors and managers decided on the approvers and SLOs for action resolution in their areas. This system just encodes and tries to enforce their decisions.

When is a postmortem needed?

We carry out postmortems for severity 1 and 2 incidents. Otherwise, they're optional.

During or shortly after resolving the issue, the postmortem owner creates a new postmortem issue.

Who completes the postmortem?

The delivery team for the service that failed (the team that owns the "Faulty Service" on the incident issue) is responsible for completing the postmortem. That team selects the postmortem owner and assigns them the postmortem issue.

• The postmortem owner drives the postmortem through drafting and approval, all the way until it's published. They are accountable for completion of the postmortem. 
• One or more postmortem approvers review and approve the postmortem, and are expected to prioritize follow-up actions in their backlog.

We have a Confluence page which lists the postmortem approvers (mandatory and optional) by service group, which generally corresponds to an Atlassian product (e.g. Bitbucket Cloud).

How are postmortem actions tracked?

For every action that comes out of the postmortem, we:

• Raise a Jira issue in the backlog of the team that owns it. All postmortem actions must be tracked in Jira.
• Link them from the postmortem issue as "Priority Action" (for root cause fixes) or "Improvement Action" (for non-root-cause improvements).

We built some custom reporting using the Jira REST APIs to track how many incidents of each severity have not had their root causes fixed via the priority actions on the postmortem. The engineering managers for each department review this list regularly.

Postmortem process

Running the postmortem process includes creating a postmortem issue, running a postmortem meeting, capturing actions, getting approval and (optionally) communicating the outcome.

The postmortem owner is responsible for running through these tasks:

1. Create a postmortem and link it to the incident.
2. Edit the postmortem issue, read the field descriptions and complete the fields.
3. To determine the root cause of the incident, use the "Five Whys" technique to traverse the causal chain until you find a good true root cause. 
4. Schedule the postmortem meeting. Invite the delivery team, impacted teams and stakeholders, using the meeting invitation template.
5. Meet with the team and run through the meeting schedule below.
6. Follow up with the responsible dev managers to get the commitment to specific actions that will prevent this class of incident.
7. Raise a Jira issue for each action in the backlogs of the team(s) that own them. Link them from the postmortem issue as "Priority Action" (for root cause fixes) or "Improvement Action" (for other improvements).
8. Look up the appropriate approvers in Confluence and add them to the "Approvers" field on the postmortem. 
9. Select the "Request Approval" transition to request approval from the nominated approvers. Automation will comment on the issue with instructions for approvers. 
10. Follow up as needed until the postmortem is approved.
11. When the postmortem is approved, we have automation to create a draft postmortem blog in Confluence for you to edit and publish. Blogging postmortems share your hard-earned lessons, which multiplies their value.

Once the postmortem process is done, the actions are prioritized by the development team as part of their normal backlog according to the team's SLO.

Postmortem meetings

We find that gathering the team to discuss learnings together results in deeper analysis into root causes. This is often over video conference due to our distributed teams, and sometimes done in groups where incidents involve large groups of people.

Our suggested agenda:

1. Remind the team that postmortems are blameless, and why
2. Confirm the timeline of events
3. Confirm the root causes
4. Generate actions using "open thinking" - "What could we do to prevent this class of incident in the future?"
5. Ask the team "What went well / What could have gone better / Where did we get lucky"

Suggested calendar booking template:

Please join me for a blameless postmortem of , where we 

Postmortem issue fields

Our postmortem issue has an extensive series of fields to encourage collecting all the important details about the incident before holding the postmortem meeting. Below are some examples of how we fill out these fields.

Proximate and root causes

When you're writing or reading a postmortem, it's necessary to distinguish between the proximate and root causes.

• Proximate causes are reasons that directly led to this incident.
• Root causes are reasons at the optimal place in the chain of events where making a change will prevent this entire class of incident.

A postmortem seeks to discover root causes and decide how to best mitigate them. Finding that optimal place in the chain of events is the real art of a postmortem. Use a technique like Five Whys to go "up the chain" and find root causes. 

Here are a few select examples of proximate and root causes:

Root cause categories and their actions

We use these categories to group root causes and discuss the appropriate actions for each.  

Root causes with dependencies

When your service has an incident because a dependency fails, where the fault lies and what the root cause depends on whether the dependency is internal to Atlassian or 3rd party, and what the reasonable expectation of the dependency's performance is.

If it's an internal dependency, ask "what is the dependency's Service Level Objective (SLO)?":

• Did the dependency breach their SLO? 
  • The fault lies with the dependency and they need to increase their reliability.
• Did the dependency stay within their SLO, but your service failed anyway? 
  • Your service needs to increase its resilience.
• Does the dependency not have an SLO?
  • They need one!

If it's a 3rd party dependency, ask "what is our reasonable expectation* of the 3rd party dependency's availability/latency/etc?"

• Did the 3rd party dependency exceed our expectation (in a bad way)?
  • Our expectation was incorrect. 
    • Are we confident it won't happen again? E.g. We review and agree with their RCA. In this case, the action is their RCA.
    • Or, do we need to adjust our expectations? In this case, the actions are to increase our resilience and adjust our expectations.
    • Are our adjusted expectations unacceptable? In this case, we need to resolve the disconnect between requirements and solution somehow, eg find another supplier.
• Did the 3rd party dependency stay within our expectation, but your service failed anyway? 
  • In this case, your service needs to increase its resilience.
• Do we not really have an expectation?
  • The owner of the 3rd party dependency needs to establish this, and share it with teams so they know what level of resilience they need to build into their dependent services.

*Why "expectation"? Don't we have SLAs with 3rd parties? In reality, contractual SLAs with 3rd parties are too low to be useful in determining fault and mitigation. For example, AWS publishes almost no SLA for EC2. Therefore, when we're depending on a 3rd party service, we have to make a decision about what level of reliability, availability, performance, or another key metric we reasonably expect them to deliver. 

Postmortem actions

Sue Lueder and Betsy Beyer from Google have an excellent presentation and article on postmortem action items, which we use at Atlassian to prompt the team.

Work through the questions below to help ensure the PIR covers both short- and long-term fixes:

"Mitigate future incidents" and "Prevent future incidents" are your most likely source of actions that address the root cause. Be sure to get at least one of these.

Postmortem approvals

Atlassian uses a Jira workflow with an approval step to ensure postmortems are approved. Approvers are generally service owners or other managers with responsibility for the operation of a service. Approval for a postmortem indicates:

• Agreement with the findings of the post-incident review, including what the root cause was; and
• Agreement that the linked "Priority Action" actions are an acceptable way to address the root cause.

Our approvers will often request additional actions or identify a certain chain of causation that is not being addressed by the proposed actions. In this way, we see approvals adding a lot of value to our postmortem process at Atlassian.

In teams with fewer incidents or less complex infrastructure, postmortem approvals may not be necessary.

Blameless postmortems

When things go wrong, looking for someone to blame is a natural human tendency. It's in Atlassian's best interests to avoid this, though, so when you're running a postmortem you need to consciously overcome it. We assume good intentions on the part of our staff and never blame people for faults. The postmortem needs to honestly and objectively examine the circumstances that led to the fault so we can find the true root cause(s) and mitigate them. Blaming people jeopardizes this because:

• When people feel the risk to their standing in the eyes of their peers or to their career prospects, this usually outranks "my employer's corporate best interests" in their personal hierarchy, so they will naturally dissemble or hide the truth in order to protect their basic needs.
• Even if a person took an action that directly led to an incident, what we should ask is not "why did individual x do this", but "why did the system allow them to do this, or lead them to believe this was the right thing to do".
• Blaming individuals is unkind and, if repeated often enough, will create a culture of fear and distrust. 

In our postmortems, we use these techniques to create personal safety for all participants:

• Open the postmortem meeting by stating that this is a blameless postmortem and why 
• Refer to individuals by role (eg "the on-call Widgets engineer") instead of name (while remaining clear and unambiguous about the facts)
• Ensure that the postmortem timeline, causal chain, and mitigations are framed in the context of systems, process, and roles, not individuals.

Our inspiration for blameless postmortems and the useful concept of "second stories" comes from John Allspaw's seminal article.