“Ouch. Dammit…” Maria said out loud, after closing yet another browser tab containing yet another job posting she wasn’t going to apply for. Turns out actually banging her head against her desk hurt more than she’d anticipated. Good grief: how did it come to this?
The worst part is that she used to love her job. Great team, great company, and as Head of IT & Engineering, her corner office has a great view. Then SplinterTech went public. Wrapping her head around the alphabet soup of regulations her team now has to comply with (PCI, SOX, HIPAA) was easy. But actually getting anything done these days? Different story.
The miasma of documentation alone is enough to drain a person’s soul in two releases or less. Then there’s code review. Not that her team resists code reviews. Just that they can’t seem to enforce it effectively, which is a liability at audit time, which is a risk to SplinterTech’s reputation, which puts 2500 people’s jobs on the line.
So yeah: ever so slightly stressful.
“Ugh, what’s the use?” she thought. “Job-hunting my way out of this mess isn’t exactly the noble thing to do anyway.” She eyed the lovingly-framed diplomas on her wall, reminding her she’s too accomplished to bail now.
. . .
If Maria’s story seems hauntingly familiar, you’re not alone. Regulatory compliance is about as exciting as a root canal. The good news is that it doesn’t need to be nearly as painful.
You’ve probably heard about DevOps by now. You may even buy the argument (as I do) that DevOps actually helps teams meet compliance standards because automation is not only an integral part of DevOps, but a great way to make sure development and deploy practices are reliable, repeatable, and traceable. Not to mention the fact that automation speeds up the whole process, which helps you move as fast (or faster) than your competitors. And DevOps’ emphasis on a culture of transparency pays off when it’s time for an audit.
I believe the technical term for pulling off dev speed and transparency and regulatory compliance is “threading the m-fing needle”. It doesn’t happen overnight, but you can get there in four not-necessarily-easy-but-totally-doable steps.
Step 1: identify and document all the rules affecting your team
It’s common sense – which means it’s easy to overlook. Make sure your team members and stakeholders are aware of what you’re required to comply with, how you’ll go about doing so, and where they can refer back to this info on the off-chance they don’t have a photographic memory. To avoid “document version hell” (also a technical term, by the way), document it all on wiki pages and put links to these pages on team portals, dashboards, etc. Bonus points if you designate key people as “watchers” on the pages so they get notified each time there’s an update.
Step 2: build your automation backlog
Once people understand what’s important, and why, you can set yourself up for the kind of automation that makes compliance easier. Start by choosing repetitive tasks that could be converted from manual to automated. Typically things like:
- Pull requests – While there should always be a thoughtful, human review, you can automate the tedious bits like making sure two or more reviewers approved the PR, and there are no failing builds or test runs associated with that commit.
- Code coverage – Any CI/CD tool worth it’s salt can be configured to fail builds in which test coverage drops below a certain threshold.
- Connecting of dots – Make sure code changes, code reviews, build results, deploys, and issues are all linked together. (As in, you can get from one to the other with a single mouse click.) Day-to-day work is easier, as are audits.
- Permissions – Oh, security… the struggle is real. But you can set up your repository manager such that only certain people can make changes in specific repos and/or branches – and such that nobody can make changes directly in production.
Note: all repos and branches should be open for read-only access by default. Because DevOps. And, because that level of transparency is just plain practical.
- Audit trails – Save a tree as well as your sanity by automatically logging build, test, and deploy results. Then complete the circle by linking in commits, reviews, and, issues as mentioned above.
- Failover and recovery – Automatically rolling back a deployment gone bad is way less error-prone than doing it by hand. And it’s traceable, and it’s faster, which helps you meet your service level agreements (SLAs).
Make sure you weave some of this work into each iteration so you’re chip-chip-chipping away at it on the regular. And reflect on it regularly, too. “Are our access controls on branches effective, or overkill?”… “Did our code coverage increases satisfy the auditors?”… etc. The trick is to meet your obligations without getting in your own way. Here again, there’s some needle-threading to do.
Step 3: get cultured
To be fair, this should really be happening in parallel with steps 1 and 2 because in the DevOps world, culture is always a priority. (But “always” would mess up my nice little numbered list, so here we are.)
If your team suffers from poor communication, finger-pointing, and/or unclear roles and responsibilities, simply automating a few things and calling it a day ain’t gonna do a lick of good. Pull your team together to identify your trouble spots and figure out how you’ll work through them. Sounds daunting, but fear not: the Atlassian Team Playbook can help.
Teams who know exactly where they’re hurting and why can head right to the collection of plays and filter by pain point. Suffering from “Communication breakdown”? A case of “Leadership Deficit Disorder”, perhaps? The Team Playbook will suggest a handful of techniques to get you back on the right track.
Keep in mind, the plays are not extra work. They’re simply different (and effective) ways of approaching the work you already do.
In many cases, though, a team is dysfunctional for reasons it doesn’t yet understand. That’s where the team Health Monitor comes in. It’s a chance for your team to self-assess against eight attributes we’ve found to be common amongst high-performing teams, and figure out where you need to improve. You’ll also get suggestions for plays that can help build up the weak muscles.
[slideshare id=95605688&doc=atlassianteamhealthmonitorshowtoguide-180501154000]
And for your convenience, the Team Playbook offers a “game plan” specifically for building DevOps culture – popular plays like “Pre-mortem” or “Rules of Engagement“. Game plans are a nifty little short cut, but you should still use the Health Monitor to keep tabs on how your team is doing. Seriously.
Last, don’t forget to celebrate those incremental wins! Even a quick high-five at stand-up for stuff that seems like table stakes (“Code coverage is up 20 percentage points – yessss 
“) does wonders for keeping momentum and morale high.
Step 4: continuously improve your development workflow
Now that you’re on the path to better team health, it’s time to work on your dev workflow. Take a step back and look for quick wins you can start with. For example, it’s common for teams practicing DevOps to adopt a feature branching workflow using a Git (or occasionally, Mercurial).
Free tutorials like this one make the switch to feature branching way easier.
Keeping all that messy work in progress isolated on a feature branch until it’s proven to be production-worthy darn near guarantees you won’t have an “oh $#!τ” moment – the one when you realize you’ve just de-obsfucated customer data in your production logs (for example). Furthermore, you can use those automated permissions I mentioned earlier to better control the flow of changes from branch to branch, and environment to environment.
Look also at whether you have the level of code coverage you need where you need it. “Specification by example” works great here. Work with your compliance lead to understand what a compliance failure would look like (i.e., how would it manifest in the logs?), then write tests that will fail the build if those conditions are detected.
Once you have strong auto-tests, you may have the option to have passing builds automatically promoted to the next environment, depending on the specific regulations you’re working with. Again: automated deployments are your friend. They’re reliable and traceable – traits auditors love, both.
And what about other bits of overhead that could be eliminated with automation? If, for instance, you’re using Jira Software to track your work, you might integrate it with Bitbucket to take advantage of “smart commits” that automatically transition the relevant issues to the next step in the workflow and save you a trip back to your agile board. Or you could integrate your repository manager with your CI/CD tool to trigger builds automatically when a pull request is created.
It’s a journey, not a flip of the switch
Is your head is swimming by now? Don’t stress. Tackling one thing at a time and iterating as you go makes the transition far more manageable from a logistical standpoint. Remember Maria from above? Steady improvement is exactly how her team got back on track. I’d be lying if I said it was easy. But they did it, and now they have a good groove going.
To help with the technical aspects, consider switching to a Git repository management tool like Bitbucket Server or Bitbucket Data Center. Teams like yours in regulated industries are taking advantage of Bitbucket’s support for workflow enforcement (like requiring green builds or multiple approvers on code reviews), project-level admin controls, and granular permissions capabilities.
If you’re already on Bitbucket Server/Data Center (look at you, smarty pants) and want to beef up your continuous integration practice, check out Bamboo. It’s got a complementary set of compliance-flavored capabilities like project organization, permissions, and deep integrations with Bitbucket Server and Jira Software.
A little bit of slick tooling, combined with a hefty dose of the right culture and practices, is great way to enjoy all the benefits of a DevOps approach without violating compliance rules. Who says you can’t have your cake and eat it, too? (Mmmmm… caaaaake…)
Check out Bitbucket Data Center
Wanna learn more about DevOps? We’ve got all the info you need to start your journey.