Incident management in the age of DevOps

The DevOps incident management process

The DevOps approach to managing incidents isn’t radically different from the traditional steps to effective incident management. DevOps incident management includes an explicit emphasis on involving developer teams from the beginning--including on call--and assigning work based on expertise, not job titles.

1. Detection  
Instead of hoping incidents never happen (because they undoubtedly will), DevOps incident response teams place a high value on preparedness. They work collaboratively to plan their responses to potential incidents by identifying weaknesses in systems. They set up monitoring tools, alert systems, and runbooks that help each member know who to contact when an incident is detected and what to do next.

2. Response  
Rather than having one single on-call engineer responsible for responding to all incidents in an on-call shift, DevOps incident management teams designate multiple team members to be available for escalations. If the designated on-call engineer can't resolve an incident independently, there's a runbook ready to act as a guide. The on-call engineer can bring in the right people to assess the impact and severity level of the problem and escalate it to the right responders.

3. Resolution
When it comes time to respond to an incident, DevOps incident management teams can often get to resolution quickly. This is because, as a whole, they're more familiar with the application or system code—because they wrote it! And with the benefit of advanced preparation and good communication systems, together they can do the work that resolves the incident, reaching resolution faster than a third-party response team looking at the code for the first time.

4. Analysis  
DevOps incident management teams close out an incident with a blameless postmortem process. They come together to share information, metrics, and lessons-learned with a goal to continuously improve the resilience of their systems, as well as resolve future incidents quickly and efficiently.

5. Readiness
Once an incident is resolved, all remediation steps have been completed, and the system is restored, DevOps incident management teams take a step back to assess their readiness for the next incident. They take what they learned in their postmortem process and update their runbooks and make any necessary adjustments to monitoring tools and alert systems. And the DevOps focus on continuous improvement applies to the people and team, not just the technology. After an incident, each team member is better prepared for the next one.

Best practices for effective DevOps IM teams

Adopting a DevOps approach to incident response can lead to improved communication between development and IT operations teams, faster incident response and remediation, and a more resilient system.

Automate processes and workflows
Integrate your service desk, monitoring, ticketing, CMDB/asset management, and chat tools to streamline IT incident alerts and workflows to ensure the right people get notified with the information they need to get started on a resolution. Set up runbooks with pre-defined workflows so people can hit the ground running when an incident hits.

Communicate between teams
Ensure members of your teams can communicate across the organization with real-time chat tools. Use tools that create a record of the incident so anyone can jump in at any time and get up to speed on what's happened and what's being done.

Use the blameless approach
After you've resolved the incident come together as a team to review what happened for a blameless postmortem session. Avoid finger-pointing and focus on sharing information that helps everyone do their jobs better and contributes to a more reliable system.

Identify and focus on the business bottom line
DevOps incident response is more than a means to better communication; it's a way to ensure developers and operations are working together to deliver real business value. Track metrics such as mean time to detection (MTTD), mean time to repair (MTTR), and mean time between failures (MTBF) to understand your team's rate of improvement.

Utilize on-call scheduling to position developers and sysadmins as SREs
On DevOps teams, the lines between developer and sysadmin start to blur and those responding to the incident often become site reliability engineers (SRE). Still, individuals will likely have specialized knowledge either in the application code or the infrastructure code. Set up your on-call schedule to ensure you've got the right mix of expertise available to respond to incidents.

Learn more about how Jira Service Management can support a DevOps approach incident management.