How to Govern Terraform States Using GitLab Enterprise

Terraform has become the nearly ubiquitous way to provision services in a cloud native era. However, when we start to build our infrastructure using Terraform’s as-code approach, there are a few things we need to consider in order to be able to manage these processes at scale, for a diversity of decentralized services, and for distributed teams. At Firefly, we often encounter the challenges of managing IaC at scale as part of our effort to help organizations discover and manage their many cloud assets.

Terraform Management at Scale

With the mass adoption of Terraform, it has become the de facto tool for developers to build and manage their cloud infrastructure at scale. Most companies today, who rely heavily on Terraform for their infrastructure management, choose to do so with an orchestration tool. These tools complement the suite of tools Hashicorp provides to help get a handle on the many modules and providers, frameworks, and services being provisioned with the sheer scale of cloud operations — as well as a remote backend to maintain the state for your infrastructure.

The companies that choose a fully managed orchestration tool oftentimes will select Hashicorp’s very own Terraform Cloud (SaaS solution). However, like all tools that gained widespread popularity, there are many non-Hashicorp alternatives that ride the wave of the tool’s success, as well. Terraform Cloud’s benefits are a fully remote back end, native integration with GitHub, state versioning, and advanced features for infrastructure stakeholders, such as platform engineers, DevOps teams and cloud engineers.

However, there is another option that is gaining popularity for large-scale Terraform operations, and that is the GitOps approach, ones who decide to deploy their infrastructure using GitHub Actions or other built-in CI pipeline applications. The most popular tool for this use case is Atlantis. Atlantis is a basic solution that integrates automatically with each pull request (PR) and enforces best practices for infrastructure deployments as they are defined in the company policies, such as the code owner, code reviewers, and unit tests using tools like TerraTest, among others .

When you choose the GitOps method, this will not come with the managed back end, and therefore, this will still be required for those looking to maintain state for their IaC. Terraform currently supports out-of-the-box integration with AWS S3, GCS, Hashicorp Consul, Kubernetes, and HTTP.

As we all know, though, many companies today have decided to work with GitLab on-prem, for many reasons, and therefore, all of the GitHub and GitHub Actions integrations become less relevant with this choice.

Terraform States Using GitLab Enterprise

Those companies who choose GitLab as their primary source code management (SCM) platform, will also many times choose to deploy their infrastructure using dedicated GitLab pipelines. This leaves us with the question: What about the Terraform state?

We introduced a new feature for remote back ends inside GitLab. We knew this was just what we needed as a GitLab shop. However, when we came to try and enable it, we found very little documentation to help us…and so we had to go down the rabbit hole of researching how to configure and setup remote back ends with specific requirements dictated in the GitLab API. We’d like to share with you some of the excellent intel we uncovered.

We’ll start with some of the challenges we immediately encountered. Configuring the GitLab back end proved itself quite complex, having to understand the GitLab configuration syntax in-depth and the various S3 configurations to actually get this set up. Once we managed to configure our S3 bucket as the dedicated data store for our Terraform states, we found that these are all encrypted using AES 256 inside the S3 by GitLab. What this means is that once encrypted, this state is no longer accessible inside Terraform. This requires you to use GitLab APIs to download them and be able to use them in your environment.

This is where it gets tricky. So we’ve decided to deploy and orchestrate our code using GitLab. Great. Next we want to leverage their new capability of managing state — but this means we can’t actually manage our Terraform state if they are encrypted and not accessible to Terraform.

This adds a particular layer of complexity for this use case, because when you work in the modern engineering format of CI/CD, GitLab will increase your version number with each deployment to maintain the log and change history of versions. All of this is fine, and important as an engineering best practice — however, this introduces a few gaps when it comes to Terraform state management.

If we take a look at the GitLab API documentation the way to download the state is as follows:

curl --header "Private-Token: " ""

This means that in order to be able to download the state you have to have a few critical pieces of information:

  • Your Access Token
  • Your Project ID
  • Your State Name
  • Your Version Number

Not only does one rarely know the specific name of their deployment, it’s very rare to know the latest version number (GitLab doesn’t this in the UI — only if you hover over the deployment or click on it will you see this number in the URL). This is constantly changing with continuous deployment.

In very large-scale operations, there are hundreds of environments running Terraform all the time, and news ones constantly being running. Not to mention different kinds of environments — development, staging, production, with all of these having multiple dev accounts. It’s a needle in a haystack.

We felt like we hit a wall. We knew there had to be a better way. We went back to research.

GitLab GraphQL API for Terraform State Management

After digging deeper, we found a gold mine. There IS another way.

We found a hidden GraphQL API that reveals all of your GitLab environments built through GitLab Pipelines which enables you to extract quite simply all of the critical information you will need to be able to download and access the Terraform state.

See it in action — below is the GraphQL code snippet that enables you to query and extract the required data.

POST https://{{GITLAB-HOST}}/api/graphql { "operationName": "getStates", "variables": { "projectPath": "sefi/tf-demo", "first": 50, "after": null, "last": null, "before": null }, "query": "query getStates($projectPath: ID!, $first: Int, $last: Int, $before: String, $after: String) {n project(fullPath: $projectPath) {n idn terraformStates(first: $first, last: $last, before: $before, after: $after) {n countn nodes {n ...Staten __typenamen }n pageInfo {n ...PageInfon __typenamen }n __typenamen }n __typenamen }n}nnfragment State on TerraformState {n idn namen lockedAtn updatedAtn deletedAtn lockedByUser {n ...Usern __typenamen }n latestVersion {n ...StateVersionn __typenamen }n __typenamen}nnfragment User on User {n idn avatarUrln namen usernamen webUrln __typenamen}nnfragment StateVersion on TerraformStateVersion {n idn downloadPathn serialn updatedAtn createdByUser {n ...Usern __typenamen }n job {n idn detailedStatus {n idn detailsPathn groupn iconn labeln textn __typenamen }n pipeline {n idn pathn __typenamen }n __typenamen }n __typenamen}nnfragment PageInfo on PageInfo {n hasNextPagen hasPreviousPagen startCursorn endCursorn __typenamen}n" }

This API returns the latest version of all environments in the project.

{ "data": { "project": { "id": "gid://gitlab/Project/", "terraformStates": { "count": 1, "nodes": [ { "id": "gid://gitlab/Terraform::State/", "name": "", "lockedAt": null, "updatedAt": "2022-08-02T19:55:26Z", "deletedAt": null, "lockedByUser": null, "latestVersion": { "id": "gid://gitlab/Terraform::StateVersion/", "downloadPath": "/api/v4/projects//terraform/state//versions/0", "serial": 0, "updatedAt": "2022-08-02T19:55:26Z", "createdByUser": null, "job": null, "__typename": "TerraformStateVersion" }, "__typename": "TerraformState" } ], "pageInfo": { "hasNextPage": false, "hasPreviousPage": false, "startCursor": "", "endCursor": "", "__typename": "PageInfo" }, "__typename": "TerraformStateConnection" },

Using the response, we can download the latest version of the Terraform state leveraging the previously mentioned GitLab API:


That’s it! It’s that easy.

For anyone using GitLab On-Prem or Enterprise, leveraging GitLab Pipelines, there really is no need to add more tooling to the stack for Terraform orchestration and management. You can leverage the built-in GitLab support and Terraform’s integration with S3. With the GraphQL API access, you can now access the required info to download your state from storage via the GitLab API.


Leave a Comment