This is a guest blog post by Xpand IT, maker of add-ons in the Atlassian Marketplace.
When companies today are trying to ship as fast as possible, test automation is becoming more and more essential. Test automation is when you use an automation tool to test the functionality of your software and find bugs. Many developers use test automation as a way to save time so they don’t have to manually test each aspect of their software as an end-user.
According to a recent survey by PractiTest & Tea Time with Testers, 86% percent of organizations have some kind of test automation in place. It’s being used mostly for functional or regression testing, but also for load testing, unit testing, and Continuous Integration. Similar to the build and deploy phases, software testing is another part of the Continuous Integration and Continuous Deployment pipelines and can be automated.
But why is test automation that relevant? Can it really help deliver faster and better software?
Together we’ll look at the benefits of test automation and where to start. We will then explore automated testing at different levels and some frameworks and automation libraries that may be used to achieve full test automation. Finally, we’ll see how a test management tool helps to provide visibility of your tests, whether you’re doing manual testing or automated testing.
Why test automation?
As shown in Gartner’s World Quality Report 2016-17, the benefits of test automation include quality, process, timing, awareness, and costs related aspects. Test automation:
- Reduces time to ship by validating builds faster.
- Reduces testing effort by running most of the software tests as automated ones, thus decreasing the time needed for testing.
- Reduces costs not just by demanding less human resources for test execution, but also by detecting bugs sooner in the development lifecycle.
- Makes Continuous Integration and Continuous Delivery practices possible.
- Avoids partial/cumulative testing. When there is no time or resources left, user stories are validated independently on different code revisions, and the overall status of the user stories in a given cycle (sprint/version) is inferred.
- Makes regression testing banal since automated regression tests can be run alongside other tests.
- Avoids human error by means of repeatable automated tests executed on clean “resettable” environments.
- Makes everyone happy, including developers by detecting bugs as early as possible and avoiding greater problems afterward.
Studies done by organizations such as NASA show that the cost of fixing bugs grows exponentially depending on the phase in which they are detected. The fix cost also depends on the stage where bugs are initially introduced (Code Complete, Steve McConnell, 2004). Overall, the bugs that are found later or introduced sooner have greater costs. Thus, by involving testers from the start, namely in the requirements specification phase and moving testing to the first development phases, by means of Shift Left Testing practices, will help reduce overall bug fixing costs.
Test automation may help substantially here since it can be implemented early in the coding stage, in the form of unit tests. Moreover, it can be used to implement component, integration, functional and UI tests, covering the whole testing spectrum.
Get started with test automation
Mike Cohn’s famous and widely-adopted Test Pyramid summarizes how the testing automation effort should be spent, depicting great focus on unit tests and significantly less on UI testing; special relevance should be given to “service tests” for validating the business logic. WatirMelon presents a variant of this pyramid concept, giving focus on the different types of automated tests and their aim.
When we scale up the pyramid, costs increase and the overall test run time increases. On the other side, tests at the bottom are simpler, faster to run, and highly focused.
Some organizations tend to implement the inverted Test Pyramid though, giving greater focus to GUI tests. This is mostly a side effect of a waterfall model or a shift right testing approach, where testing is done later in the software development lifecycle, at the end of sprints or releases.
Types of tests and automation frameworks
Unit tests
Unit tests are easier and faster to implement and maintain, thus they are the preferable place to start with test automation. This type of test will essentially evaluate if the system is being built correctly, and at the same time, assess the code coverage. Each language or platform has its own preferred unit testing frameworks, such as JUnit for Java or Nunit for C#.
Integration and Component tests
Integration or component tests will evaluate if the individual parts that make up the solution are working and integrating with each other as expected. They are relatively easy to implement and provide the means to prevent errors that are harder to track further ahead.
Service tests
Service or functional tests validate the business logic without involving the UI, thus their focus is more on the “Are we building the right system?” question. These tests complement unit tests and can be run in a Continuous Integration scenario since their overhead in terms of effort and execution duration is relatively low. Some helpful BDD (Behaviour Driven Development) frameworks for this purpose include Cucumber or RSpec.
UI tests
On the other hand, UI or end-to-end tests are the ones that validate business’ or user’s demands simulating real usage. Typical examples of these make use of web automation frameworks such as Selenium for driving browser tests. Browser or UI tests, in general, are easy to “sell” but have many drawbacks: they’re expensive, hamper finding bug root causes, and are slower and harder to stack into the Continuous Integration pipeline.
There are some other frameworks that allow complementary UI tests such as visual regression testing (e.g. PhantomCSS), for which some cases may be relevant.
Acceptance tests
On top, or in parallel, you may have acceptance tests as a way of formally ensuring the specifications are being met. Cucumber, or any BDD tool, eases the writing of automated acceptance tests. In fact, these tools distinguish the test specification from their implementation. In Cucumber, for example, tests are specified as a sequence of phrases written in natural language, which will be individually matched by the tool to find the proper test code. Each of the phrases may then be reused to specify another test scenario. BDD frameworks are great because they promote specification, closing the gap between business stakeholders, developers, and testers.
Use a testing tool to deliver better software
Choosing a Test Management Platform that allows both requirement and test management will make your team’s life easier and give enhanced visibility into the progress of your projects.
Jira is great for this purpose, among other things, it:
- fosters collaboration
- is highly flexible, extensible and can be effortlessly integrated with
- keeps track of changes, ensures accountability and traceability
An add-on such as Xray for Jira facilitates the management of both manual and automated tests, independently of the testing frameworks and automation libraries being used, providing clear visibility of the requirement coverage and test progress at any moment in time.
Xray also eases Continuous Integration, no matter what automated testing frameworks you use and your approach to the test pyramid, giving you full visibility of your tests.
Whether you’re doing manual testing or automated testing, or a mix of both, it’s important to have a consolidated overview of the status of your requirements and the progress of your testing. With a proper test management add-on, you will be able to answer, “Did we test every requirement? Have we missed any important features?” and the ultimate question: “Are we ready to go live?”
Automation and automated tests will help you obtain better software, faster and consistently, and reduce costs along the way. And using the proper Jira add-on will make it manageable. Check out this e-book that will help you find and choose the best test management add-on for Jira.