This is a guest blog from Daniel Wester. Daniel is the Chief Product Architect at Wittified Atlassian Add-ons (an Appfire company). With over 20 products on the Atlassian Marketplace, Wittified certainly knows its way around software development. Their award-winning add-on portfolio includes administrator favorites like Delegated Project Admin Pro, Delegated Project Creator, CallCenter for Jira Service Desk, and the Announcer series.
Testing is fundamental to the life of a software developer. The more code you write, the more testing it requires. The more testing you employ, the more important automation becomes. The more automation you leverage, the more infrastructure you need. And of course, more infrastructure means more cost. A yin for every yang. In software, as in life, there’s always a balance.
To develop is to test. If Confucius wrote code, that’d be his mantra.
But what if we could tip the scale in our favor? Maximize our investment in automated testing, while minimizing our cost of infrastructure. Sounds impossible, right? Well, not only is it possible, we’re already doing it here at Wittified, and I’m excited to show you how!
Tipping the scale
Once your code base grows to a certain point, automation becomes a requirement for testing. If your product integrates with another system, then integration testing is another requirement. And those integration tests typically require a test harness and/or dedicated servers (which often need to be reset between test runs).
At Wittified, we currently develop over 20 products. At that volume, having a dedicated server sitting around for each product wasn’t efficient.
Enter the art of Infrastructure as a Service (IaaS).
There are several IaaS providers out there, but our current favorite is Digital Ocean. If you’d like to try them out and follow along with our walkthrough, feel free to use our referral code and save some bucks: https://m.do.co/c/910721bc1503
For each basic platform we want to test on, we create a new Digital Ocean “Droplet” (their name for a server). We configure the Droplet as needed, then shut it down and create a snapshot. We repeat this flow for each basic platform combination.
This enables us to quickly spin up a new instance whenever necessary to do some quick testing against a real platform configuration.
On their own, these snapshots can be a major time saver, but the biggest value comes when we make the leap into automated testing. There’s an awesome command line tool for Digital Ocean called Tugboat that allows you to interact with Droplets through scripts. Leveraging Tugboat in conjunction with a few wisely placed Bamboo Jobs (see below), we’re able to automate the creation of new test servers, and once we’re done, easily destroy them.
So how do we actually do this? To demonstrate, let’s test an example Jira add-on against all minor versions of Jira from 6.0 to 7.0.
In each of our Bamboo Jobs, our tasks focus on 4 important areas:
1. Spinning up of the Droplet using Tugboat, and generating a “droplets.properties” file.
2. Reading the IP from droplets.properties, using: Bamboo Inject Variables Plugin
3. Executing our tests using the IP in step 2.
4. Killing the Droplet once complete.
Wait a minute. Doesn’t that mean we’re going to have image id’s spread out across all of our build jobs? Nope. Bamboo to the rescue! We’ll just use Global Variables to centralize them. This enables us to easily upgrade our test data by creating a new snapshot and then simply updating a Global Variable.
You might say, cool, but what’s the point? Why not just have the necessary servers always up, ready, and waiting for our tests? The answer is pretty easy: Cash. Rather than paying good money for a server that’s just sitting there whether it’s active or not, each build can now automatically trigger a new server whenever it needs one, and then effortlessly kill it once complete. We also no longer have to worry about multiple builds colliding with each other. Every build always gets its own server to test against.
Pay less, get more. Now that’s a mantra to live by!
Priorities
So where do we go from here?
Since we live in a world of fast feedback, the next step is to prioritize our builds. First we run our unit tests and basic integration tests which don’t rely on any server resources. Then by using Bamboo‘s Stage functionality we automatically test against the “most important” version of Jira (our host product for this example) after the quick tests have finished. This version is often the latest, or the one being used by most of our customers. This allows us to quickly test the waters and gain a bit of comfort with each build before proceeding against all intended versions. Using the Manual Stage functionality creates a gate before executing our tests against the remaining versions of the host product we support. This gate allows the developer to receive fast feedback, and thus specify when the tests should be run against all of the remaining supported versions. While it is a manual step, using this approach allows us to reduce the noise of preventable failed builds. In addition, the entire team can easily see the health of the build without having to look in multiple places. As I always say, an efficient developer is a happy developer!
And you know who else is happy? Finance and management. That’s because server resources can now be tracked right down to the individual build. And because Digital Ocean’s charge model supports billing based upon hourly usage, you know you’re always getting the most for your money. For Jira testing, our per-version cost is usually about $0.03, which is pretty low. Even as it applies to our User Limit Manager add-on for Confluence (with up to 60 independent servers testing against all kinds of variables like LDAP, Crowd, and more) our total cost is only about $2.00. That’s an impressively small price, for an enormous benefit — automated validation that our code is doing what it’s supposed to do.
Walking the path
If you’re looking for a lean and nimble strategy to scale your Continuous Integration infrastructure, this is the path I recommend. The rest is up to you. As with any new journey, just take it one step at a time and feel free to use the examples above. As Confucius may or may not have said: “I hear and I forget. I see and I remember. I do and I understand.” In other words, let’s get testing!
Did you find this post useful? Share it on your social network of choice so others can conquer their CI infrastructure!