CI/CD Git: 5 tips for CI-friendly Git repos

An external storage system like Artifactory (who make an add-on for Bamboo), Nexus, or Archiva can help for artifacts that are generated by your team or the teams around you. The files you need can be pulled into the build directory at the beginning of your build – just like the 3rd-party libraries you pull in via Maven or Gradle.

If you don't already have an external storage system on your network, it's easiest to take advantage of Git large file support (LFS).

Git LSF is an extension that stores pointers to large files in your repository, instead of storing the files themselves. The files themselves are stored on a remote server. As you can imagine, this drastically reduces clone time.

Chances are, you already have access to Git LFS – both Bitbucket and Github support it.

But let’s say your build requires the full repo history – if, for example, one of the steps in your build updates the version number in your POM (or similar), or you’re merging two branches with each build. Both these cases require Bamboo to push changes back to your repo.

With Git, simple changes to files (like updating a version number) can be pushed without the entire history present. But merging still requires the repo's history because Git needs to look back and find the common ancestor of the two branches – that’s going to be a problem if your build uses shallow cloning. Which leads me to tip #3.

This also makes the cloning operation much faster, and Bamboo actually does this by default.

Note that repo caching only benefits you if you're using agents that persist from build to build. If you create and tear down build agents on EC2 or another cloud provider every time a build runs, repo caching won’t matter because you’ll be working with an empty build directory and will have to pull down a full copy of the repo every time anyway.

Shallow clones plus repo caching, divided by persistent vs. elastic agents, equals an interesting web of factors. Here’s a little matrix to help you strategize.

It goes (almost) without saying that running CI on all your active branches is a good idea. But is it a good idea to run all builds on all branches against all commits? Probably not. Here’s why.

Let’s take Atlassian, for example. We have upwards of 800 developers, each pushing changes to the repo several times a day – mostly pushes to their feature branches. That’s a lot of builds. And unless you scale your build agents instantly and infinitely, it means a lot of waiting in the queue.

One of our internal Bamboo servers houses 935 different build plans. We plugged 141 build agents into this server, and used best practices like artifact passing and test parallelization to make each build as efficient as possible. And still: building after each and every push was clogging up the works.

Instead of simply setting up another Bamboo instance with another 100+ agents, we stepped back and asked if this was truly necessary. And the answer was no.

So we gave the developers the option of making their branch builds push-button instead of always triggering then automatically. It's a good way to balance testing rigor with resource conservation, and branches are where most of the change activity is happening, so there's a big opportunity for savings.

Many developers like the extra control that push-button builds offer, and find fit naturally into their workflow. Others prefer not to think about when to run a build, and stick with automated triggers. Either approach can work. The important thing is to get your branches under test in the first place, and make sure you've got a clean build before merging upstream.

Critical branches like main and stable release branches are a different story, though. Builds there are triggered automatically, either by polling the repo for changes or by sending a push notification from Bitbucket to Bamboo. Since we use dev branches for all our work-in-progress, the only commits coming into main should (in theory) be dev branches getting merged in. Plus, these are the code lines we release from and make our dev branches from. So it’s really important that we get timely test results against each merge.

Polling your repo every few minutes looking for changes is a pretty cheap operation for Bamboo. But when you're scaling up to hundreds of builds against thousands of branches involving dozens of repos, it adds up fast. Instead of taxing Bamboo with all that polling, you can have Bitbucket call out when a change has been pushed and needs to be built.

Typically, this is done by adding a hook to your repository, but as it happens, the integration between Bitbucket and Bamboo does all the under-the-hood set-up for you. Once they're linked on the back end, repo-driven build triggers Just Work™ right out of the box. No hooks or special configs required.

Regardless of tooling, repo-driven triggers carry the advantage of automatically fading into the sunset when the target branch goes inactive. In other words, you’ll never waste your CI system’s CPU cycles polling hundreds of abandoned branches. Or waste your own time manually turning off branch builds. (Though it’s worth noting that Bamboo can easily be configured to ignore branches after X days of inactivity, if you still prefer polling.)