Confession time: I have AWS credentials 2,997 days old...

Not the most reassuring thing to see, especially if you didn't even remember those existed. Here I was thinking the video I'm working on showing how I manage my DNS with Terraform and Amazon Route53 may not be that interesting. Then I switch to one of the accounts, and bam!

Image not found

AWS Credentials so old, they are starting to use social media

That worked better than a triple espresso delivered straight with an IV. So, like any good story, let's figure out how we got here. The simple answer: good intentions and rushing things. At least, that is what I would tell you if I wanted to pretend it wasn't just a lack of paying attention. When I opened up my very first AWS account in 2010, I knew nothing, so I clicked around a bunch. Fast forward a while, and I had moved onto bash scripts, and then discovered infrastructure-as-code. You would think at this point I did some cleanup to use this new Golden Hammer? Nope. Then I discovered how to use multiple AWS accounts together with IAM role switching. Mind blown. Guess what, new Golden Hammer.

Literally years later, I finally did (what I thought at the time) a decent cleanup and migrated all my various domains to Route53 from the different registrars into a single AWS account, and then set up a 2nd AWS account to manage the DNS. All with Terraform and cross-account IAM roles. What I didn't do was check all the accounts for users.

Enough justifying things, just fix it!

So the main reason I was making the video was that I needed to update my menagerie of domains with my personal details after moving to the US. I had decided to change the OS on a 2nd hand Dell T430 server I had bought a while back. It was running TrueNAS Scale and I wanted to try out Proxmox for better VM support for Home Assistant, PiHole, and a bunch of containers I run at home. Everything is stored on a ZFS raid array, so all my data should be safe, it is easy enough to import the array on another OS. I had spent some time checking and rechecking that I had everything copied and backed up, then did the reinstall. All good so far, took about 30mins after the install finished to spin up all the Docker Compose stacks. And I had made very sure my git repo with the Terraform code for my domains was copied.

You get 10 guesses what I did not remembered to copy. That's right, my AWS config and credentials file. Usually, this wouldn't be a big issue, but with my setup, I have the following account structure in AWS Organizations:

Image not found

AWS Organizations setup

At the very top, I have an account only for billing purposes, then child accounts with specific purposes. I have IAM roles from that top account into each child account, and also from my "main" account into the others, except the billing account. The means I need to rebuild my ~/.aws/config with the different accounts and IAM roles to assume, and add the correct account's API credentials to the ~/.aws/credentials file. The config looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
[profile Billing]
region = eu-west-1
output = json

[profile Registrar]
role_arn = arn:aws:iam::111111111111:role/administrator

[profile DNS]
region = eu-west-1
output = json
role_arn = arn:aws:iam::222222222222:role/administrator

[profile BlueBattleship]
region = eu-west-1
output = json
role_arn = arn:aws:iam::333333333333:role/administrator

[profile Main-Env]
region = eu-west-1
output = json
role_arn = arn:aws:iam::444444444444:role/administrator

[profile Development-Env]
region = eu-west-1
output = json
role_arn = arn:aws:iam::555555555555:role/administrator

[profile Production-Env]
region = eu-west-1
output = json
role_arn = arn:aws:iam::666666666666:role/administrator

[profile Personal-Playground]
region = eu-west-1
output = json
role_arn = arn:aws:iam::777777777777:role/administrator

[profile OG-Personal]
region = eu-west-1
output = json
role_arn = arn:aws:iam::888888888888:role/administrator

With that out of the way, I could run terraform init and terraform plan to confirm everything was working. Success!

Avoiding this next time

While it ended up not being that hard to fix since I had access to all the accounts, this could be a big mess if it was a work project where someone had left. To make sure I don't have to deal with this again, I decided to move from IAM users/groups to AWS Identity Center. It took me a while to wrap my head around the different components and how it differs from what I had used for 10+ years. To break it down into the 4 main components:

Groups - the top most component, and the first you need to define
Users - individual people, linked to a specific group
Permission sets - where you define the IAM policy/policies and how long a session may be active for
Account Assignments - this is where you link the permission set(s) to an account(s) and also the group, think of it as "Group X may use only what MyPolicy defines in AWS Accounts A, B, and C".

The first step is to enable Identity Center via the console, and you will need to choose the AWS region to set it up in. I used us-east-1, and as part of the setup, it will ask if you want to use the AWS Organizations implementation (default), or just in this account. Since I have multiple accounts, I picked the default. Then I set the AWS access portal URL to a value I can remember - I'll use my-costomized-name for the rest of this post.

Next, I can configure access using Terraform - I based this off the iam-identity-center Terraform module using the following (I have an AWS provider alias with the appropriate profile set up for each of the AWS accounts):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
# Sample AWS providers
provider "aws" {
  region  = "eu-west-1"
  profile = "Billing"
}

provider "aws" {
  alias   = "main-identity-center"
  region  = "us-east-1"
  profile = "Billing"
}

provider "aws" {
  alias   = "dns"
  region  = "eu-west-1"
  profile = "DNS"
}

#--------------------#
# Retrieve account id 
#--------------------#
data "aws_caller_identity" "billing" {}

data "aws_caller_identity" "registrar" {
  provider = aws.registrar
}

data "aws_caller_identity" "dns" {
  provider = aws.dns
}

data "aws_caller_identity" "main_env" {
  provider = aws.main-env
}

data "aws_caller_identity" "development_env" {
  provider = aws.development-env
}

data "aws_caller_identity" "production_env" {
  provider = aws.production-env
}

data "aws_caller_identity" "personal_playground" {
  provider = aws.personal-playground
}

data "aws_caller_identity" "og_personal" {
  provider = aws.og-personal
}

locals {
  # Group definitions
  sso_groups = {
    Admin = {
      group_name        = "Admin"
      group_description = "Admin IAM Identity Center Group"
    }
  }

  # User definitions
  sso_users = {
    cobusbernard = {
      group_membership = [local.sso_groups.Admin.group_name]
      user_name        = "cobusbernard"
      given_name       = "Cobus"
      family_name      = "Bernard"
      email            = "cobus@example.com"
    }
  }

  # Permission sets definitions
  permission_sets = {
    AdministratorAccess = {
      description          = "Provides AWS full access permissions."
      session_duration     = "PT12H"
      aws_managed_policies = ["arn:aws:iam::aws:policy/AdministratorAccess"]
      tags                 = { ManagedBy = "Terraform" }
    }
  }

  # Account assignments
  account_assignments = {
    Admin = {
      principal_name  = local.sso_groups.Admin.group_name
      principal_type  = "GROUP"
      principal_idp   = "INTERNAL"
      permission_sets = ["AdministratorAccess"]
      account_ids     = [
        data.aws_caller_identity.billing.account_id,
        data.aws_caller_identity.registrar.account_id,
        data.aws_caller_identity.dns.account_id,
        data.aws_caller_identity.main_env.account_id,
        data.aws_caller_identity.development_env.account_id,
        data.aws_caller_identity.production_env.account_id,
        data.aws_caller_identity.personal_playground.account_id,
        data.aws_caller_identity.og_personal.account_id,
      ]
    }
  }
}

module "aws-iam-identity-center" {
  source = "aws-ia/iam-identity-center/aws"

  # Use local variables for groups, users, permission sets, and account assignments
  sso_groups          = local.sso_groups
  sso_users           = local.sso_users
  permission_sets     = local.permission_sets
  account_assignments = local.account_assignments
}

Since I'm the only person accessing my personal accounts, I'm assigning the AdministratorAccess role to myself for each account - you would not do this for an account at a company, unless there is a small subset of people that should have full access to an account. In the code above, you won't see any mention of a password, I found it easiest to reset my password from the login portal using the email specified.

At this point, I'm still using the API key and secret to access my AWS accounts to configure Identity Center with the new user, group, permission set, and access to the accounts.

I can now switch over to using these to access my accounts, but first, let's just back up the existing config in case something goes wrong by moving it to a backup location - I will reuse the same profile names so I don't need to update my existing Terraform configuration:

1
2
mkdir ~/aws_config_backup
mv ~/.aws/* ~/aws_config_backup

To set up the new profiles, you can run aws configure sso - after entering the first 4 values, it will ask to open up a browser window to authenticate. The session name is used to share the login session between multiple profiles, I'm using personal since this is my personal setup:

1
2
3
4
5
6
7
8
9
aws configure sso
SSO session name (Recommended): personal
SSO start URL [None]: https://my-customized-name.awsapps.com/start
SSO region [None]: us-east-1
SSO registration scopes [sso:account:access]:
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://oidc.us-east-1.amazonaws.com/authorize?response_type=code&client_id=<id>&redirect_uri=http%3A%2F%2F127.0.0.1%3A44721%2Foauth%2Fcallback&state=<guid>>&code_challenge_method=S256&scopes=sso%3Aaccount%3Aaccess&code_challenge=<challenge>

Important: Initially I used used SSH to connect to the server, but noticed it couldn't use the callback URL to complete the process. I tried again using VSCode with a remote connection to the server, which then worked. It still had the same callback URL with 127.0.0.1 in it, I'm not 100% sure why it worked this time as it should have had the same issue, but will test that out again when next I need to set it up.

Since I'm already logged in after resetting my password, I could just copy & paste the URL in my browser - if you are doing your setup on your local computer, it should open that link in a browser window for you. This is what the verification screen looks like:

Image not found

Identity Center asking permission to access the account

I'm running the configuration step on a remote server, so had to paste the URL into my browser. After clicking on "Allow Access", you'll then be presented with a list of accounts in the terminal. You can use the up/down arrows to select one, after I selected "Billing" (my top-level account), I continue to fill out the requested input:

1
2
3
4
5
6
7
8
9
10
11
There are 8 AWS accounts available to you.
Using the account ID 000000000000
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
CLI default client Region [None]: us-west-2
CLI default output format [None]: json
CLI profile name [AdministratorAccess-000000000000]: Billing

To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile Billing

Run the aws s3 ... command to confirm everything is working! Now, when I look at ~/.aws/config, I see the following:

1
2
3
4
5
6
7
8
9
10
11
[profile Billing]
sso_session = personal
sso_account_id = 00000000000000
sso_role_name = AdministratorAccess
region = us-west-2
output = json

[sso-session personal]
sso_start_url = https://my-customized-name.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

And now all I need to do is duplicate the profile for all my other ones by replacing the profile name and account ID. Oh, and add this config to the README.md in my infrastructure repo in case I even make the same mistake again.

Now that the yaks are shaved, let's update the registrar details

I've almost forgotten what it was I'm trying to do at this point, which was to update my contact details for all my domains. Right, let's get to that. The Terraform resource to use seems to be aws_route53domains_registered_domain, but it works a bit differently than normal resources, and comes with this warning:

This is an advanced resource and has special caveats to be aware of when using it. Please read this document in its entirety before using this resource.

TL;DR: It doesn't create or delete this resource, only updates it. So if you removed it from your Terraform code, nothing would happen. There is one more caveat: if you have the contact type set to a different value than what you are trying to set it with this resource, it will not work. You have to do that via the AWS Console. I had a mix of "Person" and "Company" for my contacts, so had to manually change them all to "Person".

Looking at the aws_route53domains_registered_domain resource, it has a lot of fields! And you need to specify the address, country, etc for the admin, billing, tech, and registrar contact each time. I'm lazy, and don't want to copy/paste the same details over and over, so made a local module in modules/route53_registrar with the following in main.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
resource "aws_route53domains_registered_domain" "domain" {
  domain_name        = var.domain_name
  admin_privacy      = var.contact_privacy
  auto_renew         = var.auto_renew
  billing_privacy    = var.contact_privacy
  registrant_privacy = var.contact_privacy
  tech_privacy       = var.contact_privacy

  dynamic "name_server" {
    for_each = var.name_servers
    content {
      name = name_server.value
    }
  }

  admin_contact {
    email          = var.admin_contact
    address_line_1 = var.address_line_1
    city           = var.city
    country_code   = var.country_code
    first_name     = var.first_name
    last_name      = var.last_name
    phone_number   = var.phone_number
    state          = var.state
    zip_code       = var.zip_code
  }

  billing_contact {
    email          = var.billing_contact
    address_line_1 = var.address_line_1
    city           = var.city
    country_code   = var.country_code
    first_name     = var.first_name
    last_name      = var.last_name
    phone_number   = var.phone_number
    state          = var.state
    zip_code       = var.zip_code
  }

  registrant_contact {
    email          = var.registrant_contact
    address_line_1 = var.address_line_1
    city           = var.city
    country_code   = var.country_code
    first_name     = var.first_name
    last_name      = var.last_name
    phone_number   = var.phone_number
    state          = var.state
    zip_code       = var.zip_code
  }

  tech_contact {
    email          = var.tech_contact
    address_line_1 = var.address_line_1
    city           = var.city
    country_code   = var.country_code
    first_name     = var.first_name
    last_name      = var.last_name
    phone_number   = var.phone_number
    state          = var.state
    zip_code       = var.zip_code
  }
}

And for modules/route53_registrar/variables.tf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
variable "domain_name" {
  description = "Name of the domain"
}

variable "name_servers" {
  description = "List of name servers to use for domain"
  type        = list(string)
}

variable "contact_privacy" {
  description = "(Optional) Whether to hide the contact details, defaults to true"
  default     = true
}

variable "admin_contact" {
  description = "Contact for admin"
}

variable "auto_renew" {
  description = "(Optional) Whether to auto-renew a domain, defaults to true"
  default     = true
}

variable "billing_contact" {
  description = "Contact for billing"
}

variable "registrant_contact" {
  description = "Contact for registrant"
}

variable "tech_contact" {
  description = "Contact for tech"
}

variable "transfer_lock" {
  description = "(Optional) Wherether to lock domain transfers, defaults to true"
  default     = true
}

variable "address_line_1" {
  description = "First line of address"
}

variable "address_line_2" {
  description = "(Optional) Second line of address, defaults to an empty string"
  default     = ""
}

variable "city" {
  description = "City"
}

variable "contact_type" {
  description = "Type of contact, default to PERSON - see https://docs.aws.amazon.com/Route53/latest/APIReference/API_domains_ContactDetail.html#Route53Domains-Type-domains_ContactDetail-ContactType"
  default     = "PERSON"
}

variable "country_code" {
  description = "Two-letter country code - see https://docs.aws.amazon.com/Route53/latest/APIReference/API_domains_ContactDetail.html#Route53Domains-Type-domains_ContactDetail-CountryCode"
}

variable "first_name" {
  description = "First name"
}

variable "last_name" {
  description = "Last name"
}

variable "phone_number" {
  description = "Phone number"
}

variable "state" {
  description = "State or province"
}

variable "zip_code" {
  description = "ZIP code"
}

Since the details are the same for all the domains as they are all mine, I created similar variables in the root of my project's variables.tf file, and then I could use it like this in cobus_dev.tf (manages my cobus.dev domain - I'm in the process of moving over from cobus.io to it):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
resource "aws_route53_zone" "cobus_dev" {
  provider = aws.dns
  name     = var.domain_cobus_dev
}

module "cobus_dev_registrar" {
  source = "./modules/route53_registrar"

  providers = {
    aws = aws.registrar
  }

  domain_name  = var.domain_cobus_dev
  name_servers = aws_route53_zone.cobus_dev.name_servers

  admin_contact      = var.admin_contact
  billing_contact    = var.billing_contact
  registrant_contact = var.registrant_contact
  tech_contact       = var.tech_contact
  address_line_1     = var.address_line_1
  city               = var.city
  country_code       = var.country_code
  first_name         = var.first_name
  last_name          = var.last_name
  phone_number       = var.phone_number
  state              = var.state
  zip_code           = var.zip_code
}

Rinse and repeat for the other domains. I also took this opportunity to pass in the name_servers from my Route53 zones so I don't have to manually update those if I add another domain, or make changes.

Important note: for my .co.za domains, I couldn't set the transfer_lock to true, so I didn't set that value in the module, it defaults to true, and didn't cause an error. When I did have it set, the .co.za domains would error.

After applying the changes, if you are changing the contact email for the registrar, keep an eye out for a mail asking you to verify the new email. DO NOT IGNORE THIS EMAIL. Ask me how I know... your domain will be deactivated if you don't click on the link in it to verify.

Wrapping it up

Now that I felt confident that everything was working after running terraform apply to update my details, I went through each account and deleted the IAM users to ensure I don't have this mess again next time. And pushed the commit with the ~/.aws/config in my README.md to my GitHub repo. Present-day Cobus knows how to future-proof against Cobus from IaC-Past.

I'm not completely happy with this setup as I really want to keep my registrar account isolated, but feel the convenience may be worth it in this very specific instance: me managing my personal accounts with no intention to involve anyone else.

In the next post, I'll show you the module I created to set up the DNS records for my mail servers for SPF, DMARC, and the other configs. Let me know in the comments if you've done something similar, if I made some horrible mistake, or just confess to some dark secret or mess you created for yourself...

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Select your cookie preferences

Site Terms, Privacy, and more.

Confession time: I have AWS credentials 2,997 days old...

I use Terraform to manage a number of domains I own, and accidentally lost my config. While cleaning up, I discovered some very old credentials.

Enough justifying things, just fix it!

Avoiding this next time

Now that the yaks are shaved, let's update the registrar details

Wrapping it up

Comments

Select your cookie preferences

Site Terms, Privacy, and more.

Site Terms, Privacy, and more.

Confession time: I have AWS credentials 2,997 days old...

I use Terraform to manage a number of domains I own, and accidentally lost my config. While cleaning up, I discovered some *very* old credentials.

Enough justifying things, just fix it!

Avoiding this next time

Now that the yaks are shaved, let's update the registrar details

Wrapping it up

Comments

I use Terraform to manage a number of domains I own, and accidentally lost my config. While cleaning up, I discovered some very old credentials.