Deploying AWS Infrastructure with GitOps and CloudFormation

AWS CloudFormation is AWS's service for creating and deploying infrastructure as code. It allows you to define your AWS resources in JSON or YAML templates and then consistently deploy them. With CloudFormation, you don't need to provision or configure resources manually; instead, you declare them in a template that CloudFormation manages for you. While JSON is an option, I strongly recommend using YAML. YAML is easier for humans to read and can - and should - contain comments. A good Ops person should be thinking of the people trying to read the template several years after deployment.

But deploying infrastructure isn't just about writing templates. Automating the deployment process is where GitOps comes into play.

GitOps is a modern approach to managing infrastructure using Git as the single source of truth. In a GitOps workflow, changes are made by pushing commits to a version-controlled repository (like GitHub or BitBucket), triggering an automated pipeline to deploy or update infrastructure.

With GitOps, every change to your infrastructure is represented as a commit in Git. This makes it easy to track changes, roll back if needed, and audit who made changes and when.

In this post, I'll look at several options for combining GitOps with CloudFormation for automated infrastructure deployment.

This used to be a good option for managing your infrastructure within the AWS environment. CodeCommit was never the best Git repo out there, but it did the job and was self-contained. Then, in late July, AWS quietly discontinued CodeCommit for new accounts.

It is still fully functional if you already have CodeCommit, so it's not a bad option. I recommend looking to migrate, though.

AWS CodePipeline is a CI/CD service that allows you to define pipelines to automatically deploy CloudFormation templates when changes are pushed to a Git repository. One of the good things about CodePipeline is that it can be created and deployed via CloudFormation. This makes it easy to have consistent pipelines for your environments.

I have a blog post on that topic if you want to know how to use CodeCommit and CodePipeline to deploy resources to multiple accounts. It also includes a link to my Git repo.


Link: https://www.linkedin.com/pulse/cross-account-cloudformation-via-codepipeline-jason-wood-2agoc

So, you want to move away from CodeCommit but have a bunch of CodePipeline pipelines or want to use CodePipeline with another Git repository. How do you do it?

Thankfully, the answer is pretty simple. AWS has a service called CodeConnections, which can be found in the Settings area of the Developer Tools console. This used to be called CodeStar Connections, but as AWS depreciated the CodeStar tools, it renamed the connection.

What does it do? I'm glad you asked. The connector allows you to hook up with GitHub, GitLab, or BitBucket repos and use those to trigger your CodePipeline workflows. When you commit your code to one of those repos, either directly or via a pull request and merge, it will trigger your CodePipeline.

While I encourage using IaC, this is one of those things where it's just easier to use the GUI. First, start by clicking on the Connections link in the left-hand pane. Select your repo provider, in this case, GitHub, choose a name, and finally, click the Connect to xxx button.

This will bring up a login to GitHub. Enter your details or passkey**,** and the connection will be made.

Confirm the name and click the Connect button.

If all goes well, you'll be presented with the information about your connection.

The best part about this connector is that it is helpful if you already have pipelines using CodeCommit but want to migrate. You move your code to a supported repo, create a CodeConnect connection to that new repo, and update the source block. Your pipeline will then work as before.

As you can see, the Source section is relatively similar but has a few differences. You can find more information in the following AWS documentation:

This configuration lets you use some of the most popular Git repositories while leveraging AWS-native deployment tools. Since you're using AWS CodePipeline, you retain all the advantages of AWS security, including IAM role integration and secure access management.

It also allows better security. When using CodePipeline with CodeCommit, the pipelines must all be in the same account as the repo. Using CodeConnections, you can deploy the pipelines directly into the accounts where the infrastructure will be deployed. This reduces the blast radius if someone hacks your account. You can also put manual gateways into your pipelines, so if a hacker gets to your GitHub repo, they can't mess with your Prod environment.

This setup is ideal for teams that prefer Git services like GitHub but want to maintain a streamlined, AWS-integrated CI/CD process for their CloudFormation deployments.

As much as many AWS folks want to live a Microsoft-free life, many of us work in corporations where M365, Entra ID, and other Microsoft tools are the core. It's not uncommon to find Azure DevOps also being the sanctioned source for our code repos. Have no fear; Azure DevOps Pipelines can be used to trigger deployments to AWS CloudFormation.

AWS Toolkit for Azure DevOps is a free extension that can be installed from the Azure Marketplace. You build your pipeline in Azure DevOps but connect to CloudFormation to deploy the resources.

You can find information here: https://aws.amazon.com/vsts/

This blog gives an excellent example of using the AWS Toolkit: https://aws.amazon.com/blogs/devops/use-the-aws-toolkit-for-azure-devops-to-automate-your-deployments-to-aws/

This moves away from the other options, where we use CodePipeline for our deployment. It still allows you to manage your infrastructure as code, just with Azure DevOps as both repo and pipeline.

If you want a lot more customisation, BuildKite might be your option. In this scenario, we use a third-party Git repo to manage our code, BuildKite to control our pipeline, and CloudFormation to perform the deployments. BuildKite is an Aussie company that started in Melbourne, so that's pretty cool.

Like CodeConnect, BuildKite integrates with GitHub, GitLab, and Bitbucket. It also offers a facility for using Git hooks and their REST API, so you can trigger your pipeline using any Git provider.

BuildKite also allows you to have either BuildKite-hosted or self-hosted pipelines. This is an excellent solution if you have requirements to run everything in-house. Pair that with a self-hosted Github, and you have control over your environment.

In addition to all that, BuildKite pipelines look really cool. I think they did an excellent job on the UI. Having a fun UI helps if you're doing something as necessary, but boring, as managing pipelines.

BuildKite is an excellent option for companies looking for custom CI/CD solutions that allow them to maintain control of their infrastructure rather than using fully managed services.

BuildKite docs: https://buildkite.com/docs/pipelines

Now that I've talked about a few different options, let's compare them:

^ Can also use GitHub and BitBucket

Several different pathways exist for deploying AWS CloudFormation using GitOps. Whether leveraging AWS-native tools like CodeConnections and CodePipeline or integrating with third-party CI/CD platforms like Azure DevOps and BuildKite, the key is finding the right balance of automation, control, and flexibility.

Consider your team's infrastructure, security requirements, and the ecosystems you work within. With the correct setup, you can streamline your deployments, improve collaboration, and ensure your infrastructure remains secure and scalable.

Check out the CodePipeline and Cloudformation AWS documentation: https://aws.amazon.com/codepipeline & https://aws.amazon.com/cloudformation. Experiment with Azure DevOps and BuildKite for your multi-cloud or custom CI/CD needs.

Feel free to reach out for more information or check out my previous blog on multi-account deployments with CodeCommit, CodePipeline, and CloudFormation: https://www.linkedin.com/pulse/cross-account-cloudformation-via-codepipeline-jason-wood-2agoc

Keep an eye out on my LinkedIn. I'll post updates for my Pipeline Workshop to incorporate the above options.