Bootstrapping your Terraform automation with Amazon CodeCatalyst

Terraform is awesome to manage all your infrastructure, but when you have more than one developer trying to make changes to the infrastructure, things can get messy very quickly if there isn't a mechanism (CI/CD pipeline) in place to manage it. Without one, making changes to any infrastructure requires coordination and communication, and the challenge quickly scales the more people that are involved with making these changes. Imagine having to run around shouting "Hey Bob! Hey Jane! You done yet with that DB change? I need to add a new container build job!". As Jeff Bezos said:

"Good intentions never work, you need good mechanisms to make anything happen."

This tutorial will show you how to set up a CI/CD pipeline using Amazon CodeCatalyst and Terraform. The pipeline will utilize pull requests to submit, test, and review any changes requested to the infrastructure. We will cover the following topics in this tutorial:

Using S3 as a backend for Terraform state files, with DynamoDB for locking, and encrypting the state file at rest with KMS
CodeCatalyst to run our CI/CD pipelines to create and update all your infrastructure

About
✅ AWS experience	200 - Intermediate
⏱ Time to complete	30 minutes
💰 Cost to complete	Free tier eligible
🧩 Prerequisites	- AWS Account - CodeCatalyst Account - Terraform 1.3.7+ - (Optional) GitHub account
💻 Code Sample	Code sample used in tutorial on GitHub
📢 Feedback	Any feedback, issues, or just a 👍 / 👎 ?
⏰ Last Updated	2023-02-22

Chicken and egg problem

Automating your infrastructure is a great idea, but you need infrastructure to automate your infrastructure. There are three approaches to doing this:

Clicking in the console to set everything up, aka "ClickOps"
Using a CLI to create the resources for you with scripts, "Procedural"
Using Terraform without storing the state file to bootstrap, then add in the state file configurations to store it

We will be using the 3rd option, have a look at the Stack Overflow discussion around approaches for more details on the trade-offs.

Getting started

Let's get started setting this up! Make sure you are logged into your AWS, and CodeCatalyst accounts in the same browser.

Setting up a CodeCatalyst Space, Project, Repo, and Environment

Now, let's set up our CodeCatalyst Space and Project. Create a new space by clicking on Create Space on the CodeCatalyst Dashboard, add a name (we will use Terraform CodeCatalyst), add the AWS Account ID to link to for billing (111122223333 is a placeholder), you can find your account ID in the top right of your AWS Console, and follow the prompts to link your AWS Account with CodeCatalyst.

Dialog showing a CodeCatalyst Space after successfully adding an AWS account to it

Next, we need to create a new Project, click on the Create Project button, select Start from scratch, and give your project a name - we will use TerraformCodeCatalyst.

Dialog in CodeCatalyst to create a new project from scratch with name "TerraformCodeCatalyst"

Now we need to create a new repository for our code. Click Code in the left-side navigation menu, then on Source repositories, Add repository and choose Create repository. Set a repository name (we will use bootstrapping-terraform-automation-for-amazon-codecatalyst in this tutorial), add a description, and Terraform for the .gitignore file:

Lastly, we need to set up the AWS environment we will use for our workflow. In the left-side navigation menu, click on CI/CD, then on Environments, and then Create environment. Add the Environment name, Description, choose your AWS account from the dropdown under AWS account connection, and click Create environment.

Dialog to create an environment in a CodeCatalyst project

Setting up a Dev Environment

To start working on our code, we need to set up a development environment, and will be using the built-in ones provided by CodeCatalyst. In the left navigation menu, click on Dev Environment under Code, then Create Dev Environment, select Cloud9 - this tutorial with use Cloud9. Select Clone a repository, select bootstrapping-terraform-automation-for-amazon-code-catalyst in the dropdown for Repository, add an Alias of TerraformBootstrap, and then click the Create button.

Dialog in CodeCatalyst to create a dev environment using a repo hosted by CodeCatalyst

It will take 1 - 2 minutes to provision your development environment, and once done, you will be presented with a welcome screen:

The version of Terraform may not be the latest, you can check which version is installed by running terraform --version. This tutorial uses version 1.3.7, to ensure you are using that version, use the following commands:

🚨 NB: If you are using a local development environment instead of one managed by CodeCatalyst, the architecture / operating system may be different, please see the downloads page to download the appropriate version of Terraform.

Lastly, we need to add AWS CLI credentials to our Dev Environment to access resource in our account. It is recommended to not use the root user, if you have not yet set up an IAM user, please do so now by following the instructions, and make sure to copy the Access key ID and Secret access key values, then run aws configure in the terminal of your dev environment (you can leave the last two default values blank, or enter values you prefer):

You can verify that access is set up correctly by running aws sts get-caller-identity in the terminal:

Bootstrapping Terraform

Next, we need to add the required infrastructure to our AWS account using Terraform. We will be creating the following resources:

IAM roles: Provides the role for our workflow to assume in the account - one for the main branch, one for any pull requests (PRs).
IAM policies: Set the boundaries of what the workflow IAM roles may do in our account - full admin access for main branch allowing creation of infrastructure, ReadOnly for the PR branches to allow validating any changes.
S3 bucket: An S3 bucket to store our Terraform state file in.
S3 bucket versioning: Allows keeping backup copies of the Terraform state file each time it changes.
DynamoDB Table: Used by Terraform to create a lock while running - this prevents multiple CI jobs making changes when run in parallel.
KMS Encryption Key: (Optional) While the state file is stored in S3, we want to encrypt it while at rest using a KMS key. For this tutorial, we will use the pre-existing aws/s3 key, if you prefer to use a different KMS key ($1/month/key), there will be a section below to describe how to make changes to do that.

To create all of the required files, you can use the following commands to create the directories, and download the files directly from the sample repository. Run the commands in the root of your cloned git repo via the dev environment terminal:

The files created will have the following content:

Once done, edit the _bootstrap/variable.tf file and update the state_file_bucket_name (S3 bucket names are globally unique), and optionally the state_file_lock_table_name variables with the values for your S3 bucket name for the state file, DynamoDB table name for locks, and optionally change the aws_region if you want to use a different region.

We will now bootstrap our infrastructure (the body of each Terraform resource from the terraform plan command has been omitted using ...):

The output should look like:

The plan command will output a list of resources to create, and you can take a look at exactly what it will create. Once you are satisfied, run terraform apply, and confirm the infrastructure creation.

Next, we will move the state file we just created with all the details of our infrastructure to our S3 bucket. To do this, we need to configure a Terraform backend using S3. Create _bootstrap/terraform.tf with the following, and update the bucket and region values with your values:

It would be easier if we could reference region and state_file_bucket variables in the Terraform backend configuration, but it does not allow any variable / local interpolation.

To migrate the state file to S3, run terraform init -migrate-state, and you should see the following output:

We are now ready to set up our workflows, but first, let's ensure we commit our changes to our git repo. Run git add ., git commit -m "Terraform bootstrapped" and git push:

Setting up workflows

In the previous section, we created two new IAM roles for our workflow, one for the main branch with permissions to create resources, and another for all pull requests with read-only permissions. We need to add these to our CodeCatalyst Space. In the top left of the page, click on the Space dropdown, and then click on your Space name. Navigate to the AWS accounts tab, click your AWS account number, and then on Manage roles from the AWS Management Console. This will open a new tab, select Add an existing role you have created in IAM, and select Main-Branch-Infrastructure from the dropdown. Click Add role:

Dialog showing configuration to add an existing IAM role to CodeCatalyst.

This will take you to a new page with a green Successfully added IAM role Main-Branch-Infrastructure. banner at the top. Click on Add IAM role, and follow the same process to add the PR-Branch-Infrastructure role. Once done, you can close this window and go back to the CodeCatalyst one.

The base infrastructure is now in place to allow us to start using our workflow for any future changes to our infrastructure. We need to create a similar Terraform backend configuration for all the resource we will create using our workflow - as mentioned, we are intentionally keeping out bootstrapping infrastructure separate from the day-to-day infrastructure. In the root of the repo, create terraform.tf, with the following content - take note that the key for the bucket is different from what we used for the bootstrapping infrastructure, and as before, replace the bucket, region, dynamodb_table, and kms_key_id with your values:

The region set in the above block indicates in which region the S3 bucket was created, not where we will create our resources. We also need to configure the AWS provider, and set the region to use. Will use a variable for this, you could also hard-code it, but it is more manageable to keep all the variables in a single variables.tf file for this purpose. Create providers.tf with the following content:

And the variables.tf file with (you can change the region here if you want to create resources in a different one):

Now we are ready to create our workflow file. First, we need to create the workflow directory and file:

Open .codecatalyst/workflows/main_branch.yml in your IDE, and add the following - remember to replace the placeholder AWS account ID 111122223333 with the value of your account, and the IAM role names if you changed them (you can choose between the standard CodeCatalyst workflow, or to use GitHub Actions with CodeCatalyst):

Let's try out our new workflow! First, we need to stage, commit, and push our changes directly to the main branch - this is needed as only workflows committed to the repo will be run by CodeCatalyst. Use the following commands:

Output:

In your browser, navigate to the CI/CD -> Workflows page. You should see the workflow running:

List of CodeCatalyst workflows, with only a single TerraformMainBranch showing

If you click on Recent runs to expand it, you will see the details of the currently running job. Click on the job ID (Run-XXXXX) to view the different stages of the build:

Visual view of the build job with different stages in CodeCatalyst workflows

Pull Request Workflow

Now that we have our main branch workflow done, it is time to set up the pull request one. The workflow will be very similar as the main branch one, with the following difference:

A different workflow name - TerraformPRBranch
We use the PR-Branch-Infrastructure IAM role to ensure we cannot make any infrastructure changes in the PR workflow
We remove the terraform apply step
The trigger for the build is for when a PR to the main branch is opened or updated (REVISION)

Create a new file for the PR workflow as .codecatalyst/workflows/pr_branch.yml, and add the following (replacing the placeholder AWS account ID of 111122223333, and the IAM role name if you changed it) - you can choose between the standard CodeCatalyst workflow, or to use GitHub Actions with CodeCatalyst:

This workflow needs to be added to the main branch before it will trigger for a new PR, so let's do that now:

This will trigger the main branch workflow as we added a change, but without adding any additional Terraform resources, it will not make any changes:

CodeCatalyst workflows dialog with the new PR workflow added, and the main branch workflow showing a second, in-progress build

We will now add an AWS resource via Terraform via a PR. First, we need to create a new branch:

Next, create a new file in the root of the project vpc.tf - we will create a VPC that has three public subnets, and the required routing tables. Add the following content to the file:

We need to commit the change, and push the branch using --set-upstream origin test-pr-workflow as the remote branch does not yet exist:

The output shows that the remote branch has been created, and we pushed changes from our local branch to that one:

This will not yet trigger a PR branch workflow as we haven't opened the pull request. In CodeCatalyst, navigate to Code, then Pull requests, and click on Create pull request. Select test-pr-workflow as the Source branch, main as the Destination branch, and add in a Pull request title and Pull request description. You can also preview the changes the PR will make on the bottom of the screen:

CodeCatalyst open new pull request to create a PR from our branch to main

Click Create, and then navigate to CI/CD -> Workflows, and select All branches from the dropdown in the top of the Workflows menu. After selecting All branches, you will see four workflows, the TerraformMainBranch and TerraformPRBranch ones, and a copy for each of the two branches main and test-pr-workflow. The TerraformMainBranch workflow will have an error with Workflow is inactive, which is expected as we limit that workflow to only run on our main branch. Click on the Recent runs under the TerraformPRBranch workflow for the test-pr-workflow branch, and then on Terraform-PR-Branch-Plan job to see the details.

CodeCatalyst view of the TerraformPRBranch workflow, with the steps listed in the side-menu on the right.

By clicking on the Terraform Plan step, you will be able to see the proposed infrastructure changes listed in the output. You can now inspect exactly which changes will be made to your infrastructure from this pull request. In you standard day-to-day operations, you would now go back to pull request to decide what action to take. If the proposed changes have been reviewed and approved, you can merge the pull request, or you can start a conversation on the PR to address any issues or concerns. We will now merge this request to roll out this infrastructure in our account by navigating to Code -> Pull requests, clicking on the Title or ID of the PR, and then the Merge button. You are presented with a choice between a Fast forward merge, or a Squash and merge option. Fast forward merge will take all the commits on the branch and add them sequentially to the main branch as if they were done there. For the Squash merge, it will combine all the commits on the test-pr-workflow branch into a single commit before merging that single commit to main. Which one you use will depend on your development approach, for this tutorial, will use the Fast forward merge one. You can also select the option to Delete the source branch after merging this pill request. Source branch: test-pr-workflow, this will help keep your repository clean from too many branches if they are no longer used. Click on Merge, and navigate to CI/CD -> Workflows to see the new VPC being created. Click on the currently running TerraformMainBranch workflow's Recent runs, then on the job ID, and then on the 2nd step to see the progress in the right-hand pane. Once the job completes, we can verify that the VPC was created by navigating to the VPC section of the AWS Console, a clicking on the VPC ID for the VPC with the name CodeCatalyst-Terraform. You should see something similar:

AWS Console displaying details of the VPC just created

Clean up

We have now reached the end of this tutorial, you can either keep the current setup and expand on it, or delete all the resources created if you are not. If you are planning to manage multiple AWS accounts, we recommend reading the Automating multiple environments with Terraform tutorial - it follows directly from this one, and you can leave the resources created in place.

To remove all the resources we created in this project, follow the following steps in your dev environment:

Make sure you are on the main branch by running git checkout main and git pull to ensure you have the latest changes, then run terraform destroy, and type yes to confirm - this will remove the VPC we created
To delete all the bootstrapping resourced, first change into the directory by running cd _bootstrap. Before we can delete everything, we need to update our S3 state file bucket. We need to change the lifecycle policy to allow the deletion, and add force_destroy = true to also delete all the objects in the bucket. Edit _bootstrap/state_file_resources.tf, and replace the first aws_s3_bucket resource with:
Run terraform apply, and accept the changes.
Now run terraform destroy, and accept the changes. This will result in two errors since we are deleting the S3 bucket where it tries to store the updated state file, and also the DynamoDB table Terraform uses to store the lock to prevent parallel runs. The output will look similar to this:
Lastly, we need to delete the project we created in CodeCatalyst. In the left-hand navigation, go to Project settings, click on Delete project, and follow the instructions to delete the project.

Conclusion

Congratulations! You've now bootstrapped Terraform with CodeCatalyst, and can deploy any infrastructure changes using a pull request workflow. If you enjoyed this tutorial, found an issues, or have feedback for us, please send it our way!

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.