Manage your RDS Cluster like a Hero: Part 1

I've been an avid user of DynamoDB for the systems I've architected for some time. The ease of integrating it into apps and the seamless connection to the DynamoDB API via the AWS SDK are noteworthy. Throw in a couple of IAM roles, and you're all set.

However, every developer rides on the depends train. While designing my most recent app, I hit a snag with DynamoDB. The app had a peculiar requirement to fetch all records from the database with every request. Anyone familiar with DynamoDB would caution against using the Scan operation frequently due to its associated costs and performance issues.

So, I pivoted to an Aurora Serverless Postgres setup. Given the app's modest workload, a full-blown RDS instance might've been overkill in costs, making the serverless variant more economical.

The first thing that I needed to do was to create a new RDS Cluster. I leverage Terraform for the job to create the cluster and the serverless instances.

The setup was a breeze thanks to the AWS RDS Aurora module from terraform-aws-modules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
data "aws_rds_engine_version" "postgresql" {
  engine = "aurora-postgresql"

  filter {
    name   = "engine-mode"
    values = ["serverless"]
  }
}

module "aurora_postgresql_v2" {
  source  = "terraform-aws-modules/rds-aurora/aws"
  version = "~>v8.3.1"

  name                = "${local.project}-postgresqlv2"
  engine              = data.aws_rds_engine_version.postgresql.engine
  engine_mode         = "provisioned"
  engine_version      = data.aws_rds_engine_version.postgresql.version
  storage_encrypted   = true
  deletion_protection = true

  vpc_id                 = local.vpc_id
  subnets                = local.private_subnets
  create_db_subnet_group = true
  create_security_group  = true
  copy_tags_to_snapshot  = true

  master_username                     = local.aurora_master_username
  manage_master_user_password         = true
  iam_database_authentication_enabled = true
  monitoring_interval                 = 60

  serverlessv2_scaling_configuration = {
    min_capacity = 1
    max_capacity = 2
  }

  instance_class = "db.serverless"
  instances = {
    one = {}
  }
  tags = local.tags
}

As you can see, it isn’t difficult to set up the cluster and it’s instances when all the complexity is hidden from us by the module. If you delve into the configuration details, take note of lines 28 and 29—they play a pivotal role in the architecture's forthcoming challenges.

However, just to point it out, when using Aurora Serverless we just pass the configuration block serverlessv2_scaling_configuration describing the amount of ACU (Aurora capacity units) that we want to use for the cluster. Each ACU enables roughly one unit of a CPU and 2 GB of RAM.

And the instances block defines how many instances we will add to the cluster, each map inside defines an instance. If we were using a normal RDS instance, inside the blocks we would define which kind of RDS instance we would use. But in this case, it was not needed since we are in the serverless world.

Once you've set up your Terraform configurations, run the plan and apply commands. And voilà! Our cluster and its endpoints are ready to go.

Now, here's where things get spicy.

The next step would be to connect to the database and:

Setting up the application user and database
Assigning the user an admin role in the said database
Running migrations to establish the database tables

In a quick Google search with the following input: “How to configure AWS RDS Postgres instance”, the result will contain:

Traditionally, developers have leaned on bastion hosts to interact with database instances, making configurations manually (like our Google search shows). It's 2024! Direct database configurations are somewhat passé.

The challenge was automating the RDS configuration without exposing it to my VPN. While Terraform is an asset, it unfortunately lacks an official Postgres provider. Moreover, our Atlantis pipeline (or even your localhost terraform) couldn't access the RDS instance network for configuration without a bastion host.

And even with, that challenge, I pondered the use of Ansible.

But wait. Ansible works, typically using bastion hosts to jump between networks to do the configurations. How can you bypass that?

Well, that's a cliffhanger for now.

Part 2 is coming soon!

That’s all folks!

Select your cookie preferences

Site Terms, Privacy, and more.

Manage your RDS Cluster like a Hero: Part 1

Using the best of AWS, Terraform, and Ansible to manage your RDS

Comments