This post is part of a series of blogs on Atlassian QA. We will cover how the QA strategy has been implemented in different teams, the tools and techniques we use, and the personal experiences from members of the team.
The Jira engineering team is large, consisting of 78 developers and team leads, 10 product managers, six UI designers, and three technical writers. To assist this vast crowd with quality, we have a team of only six QA engineers.
In Introducing Atlassian QA, we described the overall approach that the Atlassian QA team takes to ship high-quality software quickly. However, different product teams implement this same strategy in different ways. This allows us to flex to the different needs of the teams, and to easily experiment with new ideas. This blog describes how the Jira QA team currently implements the QA strategy.
The overall vision of the process is for every feature to be coded and tested only once, by the developer responsible for implementing it. No bugs, no bouncing back and forth between tester and dev, and no extra testing stages.
Jira development Process
The development of an average story looks roughly like this:
If a story follows the most efficient development process, it follows the solid black line from Planning to Done. However, occasionally there are issues along the way, and the story may follow one of the alternative paths marked with black dotted lines.
During this process, a QA engineer has multiple points at which he or she provides input into the way the story is developed and tested – providing every form of quality improvement except actually testing the story themselves. The opportunities for a QA engineer to influence the development of a story are as follows:
Planning feedback
Occasionally, a story will seem simple, but the QA engineer knows in advance that it will actually be very difficult. To get this feedback in as early as possible, the QA engineer attends the iteration planning meeting. For example, a story might say, “As an administrator, I want to create a Jira sample project at any time.”
The QA planning feedback for this story might be, “The sample project data makes assumptions about the available issue types, resolutions, and other configuration of the system. We can’t rely on the system being configured in this way when the instance isn’t brand new. So this story will also need to cover rewriting the way that the sample data is injected.”
Based on this feedback, the team might choose to increase their estimates, change the scope, or move the story further down the backlog in favor of lower-hanging fruit.
Pre-development testing notes
As QA, we can often make accurate predictions about what bugs we’d find in a completed story. If so, why not warn the developer in advance and save everyone the bother of dealing with the bug?
In order to achieve this, we introduced pre-development testing notes. They are a set of hints, written before development starts, about the type of bugs we might have expected to find in the story once it is complete. The goal is to enable the developer to avoid introducing the bugs in the first place.
Pre-development testing notes can vary significantly in size, depending on factors such as the scope and complexity of the story, the experience level of the developer, and the level of risk associated with the changes. Here is a typical example of pre-development testing notes for a low-risk story:
Even for a simple story such as this, the QA engineer has found a little-used Jira configuration option that was not known to the developer. By bringing this up before work starts, she has saved the story from being rejected later on – saving time, effort, and irritation all around.
The expectation is that the developer will perform exploratory testing of the feature while they are developing it. Not all bugs can be predicted, after all. We work to train our developers in exploratory testing techniques so they can do this well.
QA/developer demo
Once a developer is happy that they have completed a story, he or she will call over their QA engineer for a demo session. This has multiple purposes:
- To allow the QA engineer to understand the implementation details of the story, down to the code level. This informs their decisions on risks, what testing needs to be added, and what testing is unnecessary.
- To get a second opinion on the usability of the feature: the UI (if there is one), the API (if it’s a new public API), and any output to the logs.
- To discuss what testing the developer has done and, what testing they still intend to carry out before they will push the changes to the repository.
The demo is a discussion between equals, and not a test that the developer or story can “pass” or “fail.” However, sometimes concerns are noticed during the demo, and these get quickly noted as comments on the story. This is much more lightweight than raising issues at this stage – they don’t need to be separately tracked, assigned, triaged, resolved, or tested – because they will be fixed within a few hours, while the developer is still working in his or her original context.
At the end of the demo, the QA engineer and developer will make a joint decision on what should happen to the story next. If both parties are confident about its quality, the story is moved directly to Done. A story being marked as Done is a big deal – it means that the feature will be deployed in a release to production with no further manual testing or review. This isn’t a decision we take lightly, but the time saved by skipping another testing stage is significant. In some Jira teams this is the norm for most stories, and our metrics consistently show that these teams produce the highest-quality work.
(Optional) Post-development testing notes
If either party feels that further testing is needed, the QA engineer will update the testing notes, removing any testing that’s already been done by the developer, and adding new risks based on the implementation of the story. This eliminates duplication of effort and ensures efficient testing.
Here is a portion of the post-development testing notes for a high-risk story:
(Optional) Developer testing
Once the testing notes have been updated, another developer will pick up the story for testing. This testing is carried out using the testing notes as a guide, but intelligently; not blindly following a checklist.
We are actively working on improvements upstream so we can eliminate this second round of testing. It is much more efficient to find and/or prevent issues earlier in the process. Currently, only two to four percent of all Jira stories fail this second round of testing.
Conclusion
Using this process, we manage to maintain a high standard of quality with a small number of QA engineers, across a very large amount of active stories at any given time. The process allows for active feedback, ensuring that knowledge gained from previous bugs is fed back in to the testing notes for future stories. Finally, it’s much more efficient (and enjoyable!) for all parties when we help developers to do their work right the first time, rather than waiting until they are done and then pointing out all the mistakes they made.
Check out more agile tips and tricks from Atlassian:
What agile means for the everyday developer
How to deliver quality assurance at speed