Manage your RDS Cluster like a Hero: Part 1

I've been an avid user of DynamoDB for the systems I've architected for some time. The ease of integrating it into apps and the seamless connection to the DynamoDB API via the AWS SDK are noteworthy. Throw in a couple of IAM roles, and you're all set.

However, every developer rides on the depends train. While designing my most recent app, I hit a snag with DynamoDB. The app had a peculiar requirement to fetch all records from the database with every request. Anyone familiar with DynamoDB would caution against using the Scan operation frequently due to its associated costs and performance issues.

So, I pivoted to an Aurora Serverless Postgres setup. Given the app's modest workload, a full-blown RDS instance might've been overkill in costs, making the serverless variant more economical.

The first thing that I needed to do was to create a new RDS Cluster. I leverage Terraform for the job to create the cluster and the serverless instances.

The setup was a breeze thanks to the AWS RDS Aurora module from terraform-aws-modules:

As you can see, it isn’t difficult to set up the cluster and it’s instances when all the complexity is hidden from us by the module. If you delve into the configuration details, take note of lines 28 and 29—they play a pivotal role in the architecture's forthcoming challenges.

However, just to point it out, when using Aurora Serverless we just pass the configuration block serverlessv2_scaling_configuration describing the amount of ACU (Aurora capacity units) that we want to use for the cluster. Each ACU enables roughly one unit of a CPU and 2 GB of RAM.

And the instances block defines how many instances we will add to the cluster, each map inside defines an instance. If we were using a normal RDS instance, inside the blocks we would define which kind of RDS instance we would use. But in this case, it was not needed since we are in the serverless world.

Once you've set up your Terraform configurations, run the plan and apply commands. And voilà! Our cluster and its endpoints are ready to go.

Now, here's where things get spicy.

The next step would be to connect to the database and:

In a quick Google search with the following input: “How to configure AWS RDS Postgres instance”, the result will contain:

Traditionally, developers have leaned on bastion hosts to interact with database instances, making configurations manually (like our Google search shows). It's 2024! Direct database configurations are somewhat passé.

The challenge was automating the RDS configuration without exposing it to my VPN. While Terraform is an asset, it unfortunately lacks an official Postgres provider. Moreover, our Atlantis pipeline (or even your localhost terraform) couldn't access the RDS instance network for configuration without a bastion host.

And even with, that challenge, I pondered the use of Ansible.

Well, that's a cliffhanger for now.

Part 2 is coming soon!

That’s all folks!