Is GitOps the next big thing in DevOps?

Git is a mission critical tool for software development that enables pull request and code review workflows. Pull requests promote visibility into incoming changes to a codebase and encourages communication, discussion, and review of changes. 

Pull requests are a pivotal feature in collaborative software development and changed the way teams and businesses build software. Pull requests bring transparency and measurability to a formerly opaque process. Git pull requests helped enable the evolution of DevOps processes into software development. System administrators, who typically were hesitant to change, are now embracing new software development practices such as agile and DevOps. 

Systems administration as a craft has a sloppy history. System administrators previously would manage hardware manually by either connecting to and provisioning machines in a physical server rack or over a cloud provisioning API. In addition to the manual provisioning process, large amounts of manual configuration work was a regular routine. Administrators would keep custom collections of imperative scripts and configurations, cobble them together, and place them in various places. These scripts could break at any time or get lost. Collaboration was challenging as the custom tool chains were not regularly documented or shared.

The DevOps movement arose from this primordial swamp of systems administration. DevOps borrowed the best ideas from software engineering and applied them to systems administration, where the cobbled-together tools became version-controlled code. 

IaC is one of the biggest revelations of DevOps. Previously system administrators favored custom imperative scripts to configure systems. Imperative software follows a sequence of steps to achieve a desired state, such as:

Imperative software is often error prone and easy to break by changing the sequence of events. Modern software development has trended away from imperative patterns and towards declarative software patterns.

Declarative software follows a declaration of an expected state instead of a sequence of commands. Here’s a comparison of imperative vs declarative devops statements. 

While the imperative statements might read: 

1. Install an operating system on this machine
2. Install these dependencies
3. Download code from this URL
4. Move the code to this directory
5. Do this 3 times for 3 other machines

The declarative version of this would simply read: 4 machines have software from this URL, installed at this directory.

GitOps shares many of the same benefits as an agile feature branch software development workflow. The first major benefit is ease of adoption due to the usage of common tools. Git is the de facto standard of version control systems and is a common software development tool for most developers and software teams. This makes it easy for developers familiar with Git to become cross functional contributors and participate in GitOps. 

Using a version control system lets a team track all modifications to the configuration of a system. This gives a “source of truth” and valuable audit trail to review if something breaks or behaves unexpectedly. Teams can review the GitOps history and see when a regression was introduced. Additionally this audit trail can be used as a reference for compliance or security auditing. The GitOps history can be used as proof when things like encryption certificates are modified or updated.

GitOps brings transparency and clarity to an organization's infrastructure needs around a central repo. Containing all systems configurations in a central repository helps scale contribution input from team members. Pull requests made through hosted Git services like Bitbucket have rich tools for code review and discussion commentary. This builds passive communication loops that allows the full engineering team to observe and monitor infrastructure changes.

GitOps can greatly increase productivity for a DevOps team. It allows them to quickly experiment with new infrastructure configurations. If the new changes don’t behave as expected, a team can use Git history to revert changes to a known good state. This is incredibly powerful since it enables the familiar “undo” functionality in a complicated infrastructure. 

GitOps procedures are performed by an underlying orchestration system. GitOps itself is an agnostic best practice pattern. Many popular GitOps solutions today primarily use Kubernetes as the orchestration system. Some alternative GitOps tool sets are coming to market that support direct Terraform manipulation. 

To achieve a full GitOps install, a pipeline platform is required. Jenkins, Bitbucket Pipelines, or CircleCi are some popular pipeline tools that are complementary to GitOps. Pipelines automate and bridge the gap between Git pull requests and the orchestration system. Once pipeline hooks are established and triggered from pull requests, commands are executed to the orchestration piece. 

A new pattern or component that is specifically introduced with GitOps is the GitOps “operator,” which is a mechanism that sits between the pipeline and the orchestration system. A pull request starts the pipeline that then triggers the operator. The operator examines the state of the repository and the start of the orchestration and syncs them. The operator is the magic component of GitOps.

Imagine a team identified a performance bottleneck or a spike in traffic and the team notices the load balancer is not working as expected. They look into the GitOps repo that holds the infrastructure configuration and find a specific file that configures and deploys the load balancer. They can review it on their online Git hosting site. After some review and discussion they identify that some of the configuration values for the load balancer are not optimal and need to be adjusted. 

A member of the team opens up a new pull request that optimizes the load balancer values. The pull request is reviewed and approved by a second team member and merged into the repository. The merge kicks off a GitOps pipeline, which triggers the GitOps operator. The operator sees the load balancer configuration was changed. It confirms with the systems orchestration tool that this does not match what is live on the teams cluster.

The operator signals the orchestration system to update the load balancer configuration. The orchestrator handles the rest and automatically deploys the newly configured load balancer. The team then monitors the newly updated live system to see it return to a healthy state. This is an ideal GitOps scenario. Let’s expand on it further to demonstrate GitOps utility.

Let’s imagine that instead of slightly tweaking the load balancer values to be more optimal, the team makes an aggressive decision to deploy an entirely new load balancer type. They feel the current load balancer is fundamentally flawed and want to try an alternative option. The workflow is the same as the value tweak. The team creates a pull request that introduces an entirely new load balancer configuration and deletes the old configuration. It is approved and deployed through the pipeline.

Unfortunately the team finds that this new type of load balancer is incompatible with some other services within their cluster. The new load balancer causes critical traffic failures and halts user operations. Luckily because the team has a complete GitOps pipeline they can quickly undo these load balancer changes. The team will make another pull request that reverts the repository back to the old known functional load balancer. This again will be noted by the GitOps pipeline and automatically deployed. It will rapidly improve the infrastructure and improves the reliability score of the team.

GitOps is an incredibly powerful workflow pattern for managing modern cloud infrastructure. Though primarily focused on Kubernetes cluster management, the DevOps community is applying and publishing GitOps solutions to other non-Kubernetes systems. GitOps can bring many benefits to an engineering team including improved communication, visibility, stability, and system reliability. One of the core requirements for a GitOps experience is a modern hosted Git platform like Bitbucket.