Incident management for high-velocity teams
Understanding incident severity levels
Identify and prioritize incidents for faster resolution
There are three cardinal truths of incident management.
The first is that incidents are inevitable—especially for companies that are constantly growing and innovating.
The second is that a strong incident management practice is vital to a healthy business (and a weak one costs businesses big in both employee time and satisfaction and business revenues).
The third is that not all incidents are created equal. Losing data from one database is not the same as losing data from all of your databases. Dealing with an outage that impacts 20% of your users is a whole different ballpark than dealing with an outage that impacts 90 or 100%. Handling a system outage during peak hours is a lot more stressful than handling one when most of your customers are asleep. Even two incidents that look identical on paper are unique under the surface.
Incident management for high-velocity teams
Accelerate the flow of information between operations and development teams to respond and restore systems when incidents occur with Jira Service Management.
This is why companies with robust incident management programs have well-defined incident severity levels and clear best practices for prioritizing incidents as they arise.
What are severity levels?
Incident severity levels are a measurement of the impact an incident has on the business. Typically, the lower the severity number, the more impactful the incident.
For example: At Atlassian, we define a SEV (severity) 1 incident as “a critical incident with very high impact.” This could include a customer data loss, a security breach, or when a client-facing service is down for all customers.
A SEV 2 incident is a “major incident with significant impact,” including when a client-facing service is down for a sub-set of customers or a critical function within a system is not functioning.
And a SEV 3 incident is “a minor incident with low impact,” such as a system glitch that is causing customers slight inconvenience.
At Atlassian, SEV 3 incidents can be handled during daytime/working hours, while SEV 1 and SEV 2 incidents generate an alert for on-call professionals for an immediate fix no matter the time of day.
| Severity | Description | Examples |
|---|---|---|
| 1 | A critical incident with very high impact |
|
| 2 | A major incident with significant impact |
|
| 3 | A minor incident with low impact |
|
Severity levels are useful for understanding impact quickly and setting priorities for the IT and DevOps teams.
The more well-defined your SEV levels are, the more likely it is that your team will be on the same page and able to react quickly and appropriately when incidents happen. Without well-defined severity levels, it’s easy to waste vital time defining and explaining an incident’s urgency instead of resolving it.
When and where are severity levels useful?
The core value of SEV levels is that they save teams time. A team with severity levels and a clear roadmap for addressing each level is a team that can dive straight into a fix. A team without severity levels is likely to spend the first crucial minutes of a major incident figuring out how important it is, who should handle it, and how to handle it.
The more critical the incident, the more important it is to save as much time as possible by planning ahead not only with incident resolution and communication plans, but also clear SEV levels and priorities.
How is severity different from priority?
At first glance, incident severity seems like it would be the same as incident priority. After all, a severe incident with dire consequences should be dealt with before a less-severe incident, right?
But the truth is, for most businesses, it’s more complicated than that.
Severity is a measurement of impact. How much impact does an incident have on users? Does it take down their whole system? Keep them from completing a vital task? Or perhaps just irritate them and make tasks harder?
Priority, on the other hand, is a measurement of urgency. How quickly do we need to fix this issue? Which issue needs to be fixed first?
Sometimes the two measurements align perfectly. A high-severity incident that takes down the entire company is also probably the highest priority for DevOps and IT teams to focus on.
But sometimes priority and severity don’t align. For example: Let’s say there’s a typo in the headline on your website’s homepage. This is a low-severity issue because it doesn’t actually impact the function of the website. Users can still do whatever they need to do. Employees can still do what they need to do.
But the business might see the fix as high priority for brand standard reasons or because it causes confusion or simply because it makes them look bad. And so this incident could be low-severity and high-priority.
Similarly, let’s say there’s an incident that’s causing your app to crash. It’s high severity because it keeps users from doing what they need to do. But let’s also say that incident is only affecting .05% of your users. If there are other incidents with wider impact, an issue like this might not be the top priority, even though generally the words “system is crashing” would mean all hands on deck.
Both measurements matter in incident management, but it’s important to recognize when they align and when they don’t. High severity doesn’t automatically push something to the top of the priority list and high priority doesn’t always mean a system is down.
Because priority is more actionable than severity, it’s the primary measurement we use. And because severity is often a key factor that drives priority, we’ve set clear definitions in our incident handbook for our own incident management practice.
Defining incident severity levels for your organization
Not all incidents are created equal and not all organizations handle them the same way. When setting severity levels and the processes and expectations around them, in addition to the impact of an incident, you’ll need to factor in:
- The size of your tech team
- On-call schedules
- High- and low-traffic times of day for your service
- The frequency of incidents
Why? Because each of these things might impact how you define SEV levels.
For example, an app that services local markets in a single time zone might have a big gap in use between 2 a.m. and 7 a.m. So, if your whole system goes down at 3 a.m., is it the same SEV level as a system outage during peak hours?
Or what about a startup with a small team and a lot of what we might consider SEV 3 incidents? Should they stick with a SEV 3 label and let the team work out which incidents take priority when three happen at once? Or should they split them out further into SEV 3, 4, or even 5 to distinguish between a partial loss of functionality in a customer-facing system, performance issues, and something even more minor, like a bug that doesn’t impact the usability of the system but does eventually need to be fixed?
The vital thing here is understanding your business, your team, and what kind of SEV-level and priority-level definitions work for you.
| Atlassian 3-tier | 4-tier | 5-tier | |
|---|---|---|---|
| SEV 1 | A critical incident with very high impact. | ||
| SEV 2 | A major incident with significant impact. | ||
| SEV 3 | A minor incident with low impact. For example: A system bug is creating a minor inconvenience to customers. | An incident with the potential to become a major incident if not quickly addressed. | |
| SEV 4 | A support request that’s irritating a customer but does not impact overall system function. | A minor incident that impact product usability but don’t bring it to a halt. | |
| SEV 5 | Bugs or support issues that don’t impact product usability. | ||
It's essential to have an incident management solution to escalate incidents, so the right teams can immediately swarm and begin resolution.
Discover how all of Jira Service Management's incident management capabilities, including alerting and on-call scheduling, collaborative communication, and robust reporting and analytics features enable teams to respond to, resolve, and learn from incidents.
Learn incident communication with Statuspage
In this tutorial, we’ll show you how to use incident templates to communicate effectively during outages. Adaptable to many types of service interruption.
Read this tutorialIncident communication templates and examples
When responding to an incident, communication templates are invaluable. Get the templates our teams use, plus more examples for common incidents.
Read this article