How Atlassian does operational readiness

Tier-0: Critical back-bone infrastructure

A tier-0 service provides supporting infrastructure and shared service components that tier-1 services rely on to function. Components are considered critical if one of the following is true:

• They are required for a tier-1 service to run or be accessed by its users
• They are required for a customer to sign up for a tier-1 service
• They are required for staff to support or perform key operational functions on a tier-1 service, such as:

     - Start / Stop / Restart the service 
     - Perform a deployment, upgrade, roll-back, or hot-fix     
     - Determine the current state (up / down / degraded)

Tier-1: Essential services

A tier-1 service provides a vital business, customer, or product function. These are customer-facing services or business-critical internal services. When the service is degraded or unavailable, the company loses money, or is unable to perform critical business functions, and/or core functionality from a customer perspective is lost. Tier-1 services require a 24/7 support roster, have high SLAs for key metrics, and a stringent set of go-live requirements.

Tier-2: Non-core services

A tier-2 service provides a customer-facing service that are not part of core functionality. Tier-2 services provide added value or additional user experience that might be considered optional or "nice to have."

A tier-2 service includes public services that function mainly in a marketing capacity, such as public company websites. They don’t offer customers direct business-grade services and internal services used by teams to perform aspects of their roles, such as collaboration tools, issue tracking, and more.

Tier-2 services may or may not require a 24/7 support roster, have lower SLAs, and fewer go-live requirements.

Tier-3: Internal only or non-critical features

A tier-3 service provides internal functionality to the company or experimental beta services. This class may also include services that are currently an experimental feature for early adopters, where an expectation has been set that the quality of the service may degrade during beta. This level provides a low SLA bucket for services that are supported by best-efforts only.

Data loss (RPO)

This number represents the maximum amount of data that will be lost by the service in the event of a service failure. A tier-0 service will lose less than one hour of data in the event of a service failure.

Service restoration (RTO)

This number represents the maximum amount of time before the service is back up and running. A tier-0 service will be fully recovered in less than four hours.

Backups

When a service breaks, teams may need to use backups to restore data to a certain point in time. It’s important to take regular backups of data, implement a restoration process, and routinely test the backup and restoration process. The backup and restoration process should be reliable and effective. Documentation and testing are key here.

Definition of done

• Implement a backup and restoration process
• Document and test the backup and restoration process
• Regularly test the backup and restore process
   

Capacity management

Clearly outline what capacities the service provides to consumers. In particular, identify any limits the service imposes on consumers. Implement performance testing to ensure the service operates within expected limits.

Here are some examples of information to test and make available to consumers.

• Service is limited to X requirements per second
• Service guarantees a response time of X
• X function of the service is or is not replicated cross region
• Consumer should not do X
  - overload the service
  - upload files larger than X

Definition of done

•  Service limits are identified and documented
•  Performance testing is in place to verify the limits are enforced
   

Customer awareness

Supportability is an important aspect of service quality that sits alongside reliability and usability. Teams must build support processes for a service or new feature of a service before it goes live. Supportability can include a customer support process, a change control process, support runbooks, and other support-focused items.

Customer support process

Developers must understand what happens when customers contact the support team for support and they must understand their responsibilities with respect to the support process. This can include being part of an on-call rotation or being asked to address production problems as they occur.

Change control process

Not all changes impact customers in the same way. Some changes are imperceptible to customers. For example, a small bug fix. Some result in high customer effort to adopt, such as a complete rewrite of an API. Change control helps assess the magnitude of the customer impact changes might have.

Support runbooks

Runbooks provide a high-level overview of how a service works, as well as detailed explanations of problems that can occur and how to resolve them. It’s important to update runbooks regularly and verify that documented support procedures are accurate as the service changes over time.

Definition of done

• Documentation answering most of the questions that the support team would require to investigate an issue
• A working change control process
  
   

Disaster recovery

Part of a disaster is losing an availability zone. Services need to be sufficiently resilient to operate normally in the event of an availability zone failure. Disaster recovery has two components: First, to develop and document a disaster recovery process and second, to perform ongoing testing of the documented process. Disaster recovery needs to be tested regularly. Test the ability to handle an availability zone failure using the documented disaster recovery plan.

Definition of done

• DR plan is documented
• DR plan is tested
• Recurring tests of the DR plan are scheduled
   

Logging

Logs are useful for a multitude of reasons such as detection of anomalies, investigation during or after a service outage, and tracing malicious activity by connecting related events between services using unique identifiers. There are many kinds of logs. A couple of very useful logs that most services should include are:

• Access Logs
• Error logs

Definition of done

• Appropriate logs are generated
• Logs are stored somewhere they are easily findable and searchable
   

Logical access checks

Logical access checks focus on how to manage internal users access, external users access, service to service access, and data encryption. How will the service prevent unauthorized access to functionality and data? How are user permissions defined, verified, updated, and deprecated? Do these controls provide sufficient protection to sensitive data?

Internal Users

Some important questions to answer are: How are internal users authenticated? How is access granted/provisioned? How is it taken away? How does an escalation of privileges work? What is the process for regular access reviews and audits?

External users

How is authentication handled for customers? How is access granted/provisioned? How is it taken away? How does an escalation of privileges work? What is the process for regular access reviews and audits?

Service-to-service

This is similar to internal and external users. Teams must determine how services are going to authenticate to each other. For example, by setting up OAuth 2.0.

Encryption

Teams likely want to encrypt their data at-rest and in-transit. Explain how the service manages encryption of data. How does the team manage keys? What is the key rotation policy?

Definition of done

• Logical access checks are documented, implemented, and tested for internal users, external users, and service-to-service
• Data is encrypted at rest
• Data is encrypted in transit
• Encryption is implemented and tested
   

Releases

Deployment of a new version of the service must not disrupt customer traffic beyond what is defined in the services SLA. All changes must be peer-reviewed, tested, and deployed via CI/CD pipelines. After each deployment, verify the deployment was successful and didn't break any functionality. Automated post-deployment verification is preferred. Have multiple environments such as test, staging, pre-production, and production so deployments can be tested.

Definition of Done

• The service has a zero-downtime deployment
• There are environments where the service must be deployed and tested before going to production
   

Security checklist

The security checklist is a set of practices and standards for developing and maintaining secure infrastructure and software. These standards reduce the likelihood of privacy violations and data breaches and, in turn, lead to enhanced customer trust. Teams must develop a security checklist that addresses the nature of the service they are building. A few example requirements are listed:

Definition of done

• Evidence that no open critical or high vulnerabilities exist for the service
• Use of encryption at rest for all datastores
• Evidence that the service does not allow insecure HTTP connections
   

Service metrics

Service metrics provide essential health and diagnostic information about a service and empowers teams to monitor and respond to incidents. Start by defining a set of metrics that are monitored for each service. Then, create a dashboard with these metrics in a tool like Datadog or New Relic. Raise alarms when a metric moves out of bounds and raise trouble tickets in the event of an alarm.

Definition of done
Here are some examples of things to measure:

• Latency: the time taken to handle a request
• Traffic: load places on the service by external users
• Errors: number of user affecting errors or failures
• Saturation: how busy is the service and how much more can it handle
• Underlying resource usage: CPU, memory, disk
• Application internals such as queues, timings, and flow
• Usage and core functionality of your service: active users, actions per minute
   

Service resilience

Service resilience requirements determine whether or not a service can handle changes in load and/or failures of various components. A service that is resilient will likely auto-scale and be resistant to single node failure.

Auto-scaling

If the service has the ability to scale automatically, ensure the auto-scaling is configured properly and tested. Determine what metric will trigger auto-scaling and test to make sure it works. For example, if the service requires storing data on disk, it can monitor the percentage of free space of the disks and can auto-scale by adding storage when the percentage of free space falls below a threshold.

Single node failure testing

It is desirable to have services that can survive single node failures. If a single service node goes down, the service should continue to function, possibly with reduced capacity. Test this by terminating at least one node in the service and observe system behavior. It is expected that your service will handle a single node failure. The environment where you will simulate a single node failure must be monitored.

Definition of done

• Evidence of auto-scaling implemented and tested
• Evidence that the production and/or staging environments run multiple nodes
• Evidence that the service is resilient to single node failure
   

Support

Support is the process of supporting a service after release. Teams need to have runbooks, ops tools, and on-call rotations in place and working before going live so that services experiencing issues have a process in place to fix them.

Runbooks

Runbooks provide on-call responders with the context and instructions they need to lead rapid incident response and remediation efforts.

Ops tools

Running a service to a sufficient standard means that there is an on-call roster in place and that an ops tool like Opsgenie is setup to alert on-call when the service has issues.

On-call

For a Tier 2 & Tier-3 service - an on-call roster is required

For a Tier 1 & 0 service - a 24x7 on-call roster is required

Definition of done

• Runbooks are written and findable by support
• Ops tool is configured and tested
• On-call rotations are in place and being paged in the event of issues
   

Tier-3 suggested operational readiness checks

• Backups
• Logging
• Releases
• Security checklist
• Service metrics
• Support
   

Tier-3 services start with the most basic operational readiness requirements.

Tier-2 and Tier-1 suggested operational readiness checks

• Backups
• Disaster recovery
• Logging
• Releases
• Security checklist
• Service metrics
• Service resilience
• Support

Tier-2 and tier-1 services add disaster recovery and service resilience operational readiness requirements. It is important to note that tier-2 and tier-1 services could have different operational readiness requirements. It is not required that the tiers have different requirements. If another operational readiness requirement is deemed necessary for a specific service tier, then add it. Tier-2, and tier-1 could diverge depending on the team’s needs.

Tier-0 suggested operational readiness checks

• Backups
• Capacity management
• Customer awareness
• Disaster recovery
• Logging
• Logical access checks
• Releases
• Security checklist
• Service metrics
• Service resilience
• Support

Tier-0 services add capacity management, customer awareness, and logical access checks.