Enforcing best practice on self-serve infrastructure with Terraform, Atlantis and Policy As Code

Policy as Code

As developers are now empowered to self-serve their infrastructure, we are losing sight on what is actually being built. Are our custom modules being used? Are resources created in the correct zone / region? Are we exposing GCS buckets publicly?

Without assigning ourselves as code owners on all Terraform repositories and gatekeeping all Pull Requests, we have little visibility of what infrastructure is being provisioned every day. Having an infrastructure team that oversees every decision does not scale, and just serves as a bottleneck where users are more likely to ask for us to do something, rather than feeling empowered to do it themselves.

We decided that a better approach would be to build compliance directly into our Terraform pipelines. We carried out a series of Proof of Concepts on a number of open source Policy as Code tools, and eventually settled on a pattern that we have rolled out to Production and is helping us keep our infrastructure secure (and compliant) today.

What is policy as code?

Hashicorp define Policy as Code very well here:

Policy as code is the idea of writing code in a high-level language to manage and automate policies.

In the same way we define our infrastructure as code using Terraform, we want to write policies in code that enforce our best practices on Terraform. An example of this would be to ensure every resource has a valid team label, and the corresponding value of the team label is a valid team within our organisation.

What did we look for in a tool?

We evaluated the tools based on the following criteria:

• Well supported, with a strong community or a proactive development team
• Allows writing custom policies (some tools only have built-in policies for CIS benchmarks)
• Easy to integrate into our existing pipelines
• Easy to pick up and learn
• Easy to write new policies
• Ability to test our policies with unit tests

What tools did we trial?

• checkov
• terrascan
• terraform-compliance
• OPA (Open Policy Agent)
• regula
• Conftest

There is a fantastic article written by Cristophe Tafani-Dereeper that compares each of these tools in depth, and does an excellent job of explaining the different types of code analysis. We used Cristophe’s work as the base for our testing, and would recommend reading this if you are also interested in starting this journey yourself.

What did we learn?

First and foremost, there is no one size fits all solution, and definitely no correct answer.

It was apparent very quickly that the built-in checks that are packaged with these tools cater more towards AWS than GCP. We also realised quickly that the capability to build our own custom checks was far more important than having a comprehensive suite of tests built in.

We found that static code analysis (that is, analysing Terraform code without producing a plan) was not particularly reliable, often missing resources inside nested modules, or not correctly populating variables from tfvars files. Analysis on a plan produced by Terraform produced far more reliable results.

It’s also worth noting that some of these open source tools are a compliment to a paid SaaS product, so some key features for us would be locked unless we opted to subscribe. We love open-source here at loveholidays, and give preference to open-source first tools.

Which tool won?

We eventually settled on using Fugue’s Regula as the framework for writing our custom checks. Regula is a library written on top of the Open Policy Agent engine, utilising the Rego language to write policies.

A terraform plan command can output a JSON file to your local filesystem, containing the resources in your configuration and the planned changes Terraform wants to apply. This json file is a very complicated, nested structure with many layers of resources. When using the Open Policy Agent engine directly, or an abstraction such as Conftest, it relies on the user to navigate and loop (“walk”) through this complex file, which is no small feat and prone to mistakes.

Regula provides a series of helper functions written in Rego, which do the heavy lifting of navigating the complex Terraform plan file. This means the focus can be on using Regula’s functions to write simple policies, rather than fighting the plan file in each policy.

We found the Fugue team to be extremely responsive and helpful over on Github. A special thanks goes out to them for the excellent product they’ve built.

The product has seriously evolved since we started using it, and now includes a full CLI tool written in Go. Currently, we are only using Regula’s core Rego library, however the CLI and other recent features look promising.

Integrating into our pipeline

Once we had a number of policies written using the Regula framework, we had to work out how to actually incorporate these into our deploy pipelines.

We use Conftest to validate Kubernetes Manifests elsewhere in our stack, so we had some in-house knowledge of this already. Conftest also uses the Open Policy Agent engine, so it can use the libaries provided by Regula and execute any policies written in Rego.

Crucially, Atlantis also supports Conftest policy checks as part of its “plan” workflow.

So now, we have Atlantis running a Terraform plan, passing that plan file to Conftest, Conftest then pulls the Regula framework and our custom policies written in Rego, and comments the output to the PR, all in one single action:

Each policy is documented in a markdown file in our policy-as-code repository, and linked to as part of a violation warning as seen above. This means that, when a developer finds a violation in their code, they can go to our documentation and see exactly why their resource violates any particular rule.

We are still fairly early in our journey with Policy as Code, and our policies change frequently. Due to this, we download the policies from our policy-as-code Github repository on each plan via Atlantis, which means every Terraform Plan is evaluated against the latest set of policies we have released. This allows us to create/modify policies, and release them to production, extremely quickly.

Testing our policies

Our policies are kept in a dedicated Github repository. Each policy file has a corresponding <file>_test.rego and <file>_infra.tf file, where the .tf file defines a set of Terraform resources that pass and fail a policy, and the _test file sets the pass/fail result for each Terraform resource against the policy.

We then utilise a combination of Regula and Conftest to execute unit tests against each policy.

These unit tests run via Github Actions as part of the Pull Request process for adding or changing a Policy, so we can be confident that every policy passes or raises violations as we would expect.

Example Policies

The following are some example policies we have incorporated here at loveholidays:

• Ensure resources have a ‘team’ label, with a correct value that is a real team at loveholidays
• Ensure resources are not created outside of the EU
• Ensure we are only using whitelisted external Terraform Modules
• Ensure uses of our loveholidays’ custom Terraform Modules meet a minimum version requirement (if we release a mandatory module upgrade, we can bump the minimum tag in the policy and force users to upgrade on a subsequent Terraform plan)
• Ensure resources have all required labels based on resource type
• Deny public GCS buckets
• Allow only Google Groups we manage in Terraform to appear in a project’s IAM list
• Deny any use of over-privileged roles such as project owner

Most of our rules will look at the contents of all resources in the Terraform configuration on each plan, so if we add a new policy, it will raise all existing violations in the code base.

The other style of policy would be to only look at the resources present in the planned changes for a particular PR. For example, if a new firewall rule is added that exposes port 22 for a particular VM, we could violate a rule that says “fail when a VM will be exposed on port 22”, which will require a predetermined owner to approve the changes.

Atlantis has a concept of Github owners. This group of approvers can type atlantis approve on a PR following a policy violation, which will enable the requestor to then comment atlantis apply and proceed with the PR. On a subsequent PR, this firewall rule would not raise a violation, because it will not appear in the planned changes of that plan.

This enables us to catch changes to the infra as they are being applied, but not falsely report them on every plan.

For our static policies that evaluate all resources, where we do have existing violations, we utilise Regula’s waivers feature to make special exceptions / whitelists against a particular resource ID and policy ID.