This tutorial walks you through the complete steps of setting up a Terraform workspace in Github for provisioning infrastructure in Google Cloud using Terraform cloud.

Pre-requisites Link to heading

  1. A Github account
  2. A Google account with Google cloud console setup
  3. Setup CLI tools
    1. gcloud
    2. Install terraform. Consider using a tool like tfenv so that you can have flexible terraform versions
  4. Terraform Cloud account
  5. Read about Infrastructure as code
  6. Read about Terraform Basics

Terraform Cloud Link to heading

Terraform needs to manage and persist the state of the infrastructure. With each change in the Terraform workspace, a new version of state is created. When working with cloud providers like AWS and GCP, the state can be stored in a S3 or a GCP bucket. This allows developers in an organization to freely collaborate by fetching the state from the buckets and then running plans and applies through a CI/CD pipeline. Although this is solution for state management works there are few caveats:

  1. The state of the workspace and all the resources can be checked with only the state file. This might not be intuitive enough for people not expert in Terraform.
  2. There is additional overhead of creating, managing and scaling a CI/CD pipeline to run Terraform plan on commits and Terraform plan and apply on merges.
  3. For small individual projects using a S3 or GCP buckets can incur additional monetary costs with for state management.

Terraform Cloud solves all of these problems by providing a managed solution for state management which helps in collaboration. Terraform cloud is free for upto an organization with 5 users with full API access and unlimited organization which makes it perfect to use as a playground to learn Terraform without going into the complexity of setting up buckets for remote state management.

Setting up a Terraform workspace for Google Cloud Link to heading

  • Set a unique project name

    export GCP_PROJECT="anshumantripathi-my-project"
    
  • Create a GCP project.

    gcloud project create $GCP_PROJECT
    
  • Create a service account for terraform with the name terraform-former

    gcloud iam service-accounts create terraform-former --description='Service account for Terraform' --display-name='terraform-former' --project $GCP_PROJECT
    
  • Create a service account key

    gcloud iam service-accounts keys create key.json --iam-account=terraform-former@$GCP_PROJECT.iam.gserviceaccount.com
    
  • Enable required services on your GCP project.

    gcloud services enable cloudresourcemanager.googleapis.com --project $GCP_PROJECT
    gcloud services enable iam.googleapis.com --project $GCP_PROJECT
    
    • cloudresourcemanager.googleapis.com is required to allow Terraform to manage resources in the GCP project.
    • iam.googleapis.com is required to allow Terraform to manage IAM.
  • Create a new Github repository for your workspace

  • Clone the repository in your local environment and create the following files. See this example for reference

    1. variables.tf
    2. versions.tf
    3. backend.tf
    4. project.tf
  • In the versions.tf add the following:

    terraform {
      required_version = "1.3.9"
      required_providers {
        google = {
          source  = "hashicorp/google"
          version = "~> 4.54.0"
        }
        google-beta = {
          source  = "hashicorp/google-beta"
          version = "~> 4.54.0"
        }
      }
    }
    

    This sets up the Google and Google-beta providers. We add the Google beta provider to get access to beta level APIs. The required_version sets the Terraform version requirement to 1.3.9.

  • Push the changes to your repo

  • Create a workspace in Terraform Cloud.

    1. Sign in to Terraform Cloud
    2. Create a workspace. This should add the Terraform cloud Github app to your workspace Github repository.
    3. Login to terraform cloud from your local environment using terraform login
  • Setup Google cloud authentication

    1. Create an environment variable called GOOGLE_CREDENTIALS in your Terraform Cloud workspace.
    2. Remove the newline characters from your JSON key file and then paste the credentials into the environment variable value field. You can use the tr command to strip newline characters. cat key.json | tr -s '\n' ' '
    3. Mark the variable as Sensitive and click Save variable.
  • In the backend.tf file add the following to setup backend as Terraform cloud:

    terraform {
      backend "remote" {
        organization = "anshumantripathi"
    
        workspaces {
          name = "gcp-terraform-workspace"
        }
      }
      required_version = ">= 1.3.9"
    }
    
  • In the variables.tf file add the following local variables:

    locals {
      # Replace with project name created in step 1
      google_project = "anshumantripathi-my-project"
      services = [
        "cloudresourcemanager.googleapis.com",
        "iam.googleapis.com",
      ]
      terraform_former_project_roles = [
        "roles/owner"
      ]
    }
    
    • Create a project.tf file and define your project and service account. See this example for reference
      # Create project
      resource "google_project" "my_project" {
          name            = local.google_project
          project_id      = local.google_project
      }
      # Create service account
      resource "google_service_account" "terraform_former" {
          account_id   = "terraform-former"
          project      = local.google_project
          display_name = "terraform-former"
          description  = "Service account for terraform"
      }
      resource "google_project_service" "my_project_services" {
          for_each = toset(local.services)
          project  = local.google_project
          service  = each.key
      }
      resource "google_project_iam_binding" "terraform_former_project_iam_binding" {
          for_each = toset(local.terraform_former_project_roles)
          project = local.google_project
          role = each.key
          members = ["serviceAccount:${google_service_account.terraform_former.email}"]
      }
      

      Note: It should be kept in mind that google_project_iam_binding is authoritative!. Consider using google_project_iam_member

  • Setup workspace locally

    terraform init -upgrade
    terraform fmt -recursive # Format your code
    
  • Import all your changes to terraform

    terraform import google_project.my_project my-project
    terraform import 'google_project_service.my_project_services["iam.googleapis.com"]' $GCP_PROJECT/iam.googleapis.com
    terraform import 'google_project_service.my_project_services["cloudresourcemanager.googleapis.com"]' $GCP_PROJECT/cloudresourcemanager.googleapis.com
    terraform import google_service_account.terraform_former "$(gcloud iam service-accounts describe terraform-former@$GCP_PROJECT.iam.gserviceaccount.com --project $GCP_PROJECT --format='value(name)')"
    terraform import 'google_project_iam_binding.terraform_former_project_iam_binding["roles/owner"]' "$GCP_PROJECT roles/owner $(gcloud iam service-accounts describe terraform-former@$GCP_PROJECT.iam.gserviceaccount.com --project $GCP_PROJECT --format='value(email)')"
    
  • Commit and push your changes to your repository. The Terraform cloud Github app will submit a Terraform in your Terraform cloud workspace to provision the required resources.

Now all your resources are tracked with Terraform. Sometimes it can be desirable to not manage the resources used by Terraform itself, like the terraform-former service account, through Terraform. That might be true, but, I would recommend managing all resources through Terraform because:

  1. If you make a mistake with Terraform’s own resources, they can be fixed and corrected manually and the state can be reverted. On the other hand, if you mess up a manually created resource, there is a possibility you might not be able to revert it (assuming you realize what went wrong).
  2. Terraform allows IaC which gives version control. This way even your mistakes are documented which helps you avoid them and take active measures to not run into them.