Confession time: I have AWS credentials 2,997 days old...
I use Terraform to manage a number of domains I own, and accidentally lost my config. While cleaning up, I discovered some *very* old credentials.
Cobus Bernard
Amazon Employee
Published Nov 26, 2024
Not the most reassuring thing to see, especially if you didn't even remember those existed. Here I was thinking the video I'm working on showing how I manage my DNS with Terraform and Amazon Route53 may not be that interesting. Then I switch to one of the accounts, and bam!
That worked better than a triple espresso delivered straight with an IV. So, like any good story, let's figure out how we got here. The simple answer: good intentions and rushing things. At least, that is what I would tell you if I wanted to pretend it wasn't just a lack of paying attention. When I opened up my very first AWS account in 2010, I knew nothing, so I clicked around a bunch. Fast forward a while, and I had moved onto bash scripts, and then discovered infrastructure-as-code. You would think at this point I did some cleanup to use this new Golden Hammer? Nope. Then I discovered how to use multiple AWS accounts together with IAM role switching. Mind blown. Guess what, new Golden Hammer.
Literally years later, I finally did (what I thought at the time) a decent cleanup and migrated all my various domains to Route53 from the different registrars into a single AWS account, and then set up a 2nd AWS account to manage the DNS. All with Terraform and cross-account IAM roles. What I didn't do was check all the accounts for users.
So the main reason I was making the video was that I needed to update my menagerie of domains with my personal details after moving to the US. I had decided to change the OS on a 2nd hand Dell T430 server I had bought a while back. It was running TrueNAS Scale and I wanted to try out Proxmox for better VM support for Home Assistant, PiHole, and a bunch of containers I run at home. Everything is stored on a ZFS raid array, so all my data should be safe, it is easy enough to import the array on another OS. I had spent some time checking and rechecking that I had everything copied and backed up, then did the reinstall. All good so far, took about 30mins after the install finished to spin up all the Docker Compose stacks. And I had made very sure my git repo with the Terraform code for my domains was copied.
You get 10 guesses what I did not remembered to copy. That's right, my AWS config and credentials file. Usually, this wouldn't be a big issue, but with my setup, I have the following account structure in AWS Organizations:
At the very top, I have an account only for billing purposes, then child accounts with specific purposes. I have IAM roles from that top account into each child account, and also from my "main" account into the others, except the billing account. The means I need to rebuild my
~/.aws/config with the different accounts and IAM roles to assume, and add the correct account's API credentials to the ~/.aws/credentials file. The config looked something like this:With that out of the way, I could run
terraform init and terraform plan to confirm everything was working. Success!While it ended up not being that hard to fix since I had access to all the accounts, this could be a big mess if it was a work project where someone had left. To make sure I don't have to deal with this again, I decided to move from IAM users/groups to AWS Identity Center. It took me a while to wrap my head around the different components and how it differs from what I had used for 10+ years. To break it down into the 4 main components:
- Groups - the top most component, and the first you need to define
- Users - individual people, linked to a specific group
- Permission sets - where you define the IAM policy/policies and how long a session may be active for
- Account Assignments - this is where you link the permission set(s) to an account(s) and also the group, think of it as "Group X may use only what MyPolicy defines in AWS Accounts A, B, and C".
The first step is to enable Identity Center via the console, and you will need to choose the AWS region to set it up in. I used
us-east-1, and as part of the setup, it will ask if you want to use the AWS Organizations implementation (default), or just in this account. Since I have multiple accounts, I picked the default. Then I set the AWS access portal URL to a value I can remember - I'll use my-costomized-name for the rest of this post. Next, I can configure access using Terraform - I based this off the
iam-identity-center Terraform module using the following (I have an AWS provider alias with the appropriate profile set up for each of the AWS accounts):Since I'm the only person accessing my personal accounts, I'm assigning the
AdministratorAccess role to myself for each account - you would not do this for an account at a company, unless there is a small subset of people that should have full access to an account. In the code above, you won't see any mention of a password, I found it easiest to reset my password from the login portal using the email specified.At this point, I'm still using the API key and secret to access my AWS accounts to configure Identity Center with the new user, group, permission set, and access to the accounts.
I can now switch over to using these to access my accounts, but first, let's just back up the existing config in case something goes wrong by moving it to a backup location - I will reuse the same profile names so I don't need to update my existing Terraform configuration:
To set up the new profiles, you can run
aws configure sso - after entering the first 4 values, it will ask to open up a browser window to authenticate. The session name is used to share the login session between multiple profiles, I'm using personal since this is my personal setup:Important: Initially I used used SSH to connect to the server, but noticed it couldn't use the callback URL to complete the process. I tried again using VSCode with a remote connection to the server, which then worked. It still had the same callback URL with 127.0.0.1 in it, I'm not 100% sure why it worked this time as it should have had the same issue, but will test that out again when next I need to set it up.
Since I'm already logged in after resetting my password, I could just copy & paste the URL in my browser - if you are doing your setup on your local computer, it should open that link in a browser window for you. This is what the verification screen looks like:
I'm running the configuration step on a remote server, so had to paste the URL into my browser. After clicking on "Allow Access", you'll then be presented with a list of accounts in the terminal. You can use the up/down arrows to select one, after I selected "Billing" (my top-level account), I continue to fill out the requested input:
Run the
aws s3 ... command to confirm everything is working! Now, when I look at ~/.aws/config, I see the following:And now all I need to do is duplicate the profile for all my other ones by replacing the profile name and account ID. Oh, and add this config to the
README.md in my infrastructure repo in case I even make the same mistake again.I've almost forgotten what it was I'm trying to do at this point, which was to update my contact details for all my domains. Right, let's get to that. The Terraform resource to use seems to be
aws_route53domains_registered_domain, but it works a bit differently than normal resources, and comes with this warning:This is an advanced resource and has special caveats to be aware of when using it. Please read this document in its entirety before using this resource.
TL;DR: It doesn't create or delete this resource, only updates it. So if you removed it from your Terraform code, nothing would happen. There is one more caveat: if you have the contact type set to a different value than what you are trying to set it with this resource, it will not work. You have to do that via the AWS Console. I had a mix of "Person" and "Company" for my contacts, so had to manually change them all to "Person".
Looking at the
aws_route53domains_registered_domain resource, it has a lot of fields! And you need to specify the address, country, etc for the admin, billing, tech, and registrar contact each time. I'm lazy, and don't want to copy/paste the same details over and over, so made a local module in modules/route53_registrar with the following in main.tf:And for
modules/route53_registrar/variables.tf:Since the details are the same for all the domains as they are all mine, I created similar variables in the root of my project's
variables.tf file, and then I could use it like this in cobus_dev.tf (manages my cobus.dev domain - I'm in the process of moving over from cobus.io to it):Rinse and repeat for the other domains. I also took this opportunity to pass in the
name_servers from my Route53 zones so I don't have to manually update those if I add another domain, or make changes.Important note: for my.co.zadomains, I couldn't set the transfer_lock to true, so I didn't set that value in the module, it defaults totrue, and didn't cause an error. When I did have it set, the.co.zadomains would error.
After applying the changes, if you are changing the contact email for the registrar, keep an eye out for a mail asking you to verify the new email. DO NOT IGNORE THIS EMAIL. Ask me how I know... your domain will be deactivated if you don't click on the link in it to verify.
Now that I felt confident that everything was working after running
terraform apply to update my details, I went through each account and deleted the IAM users to ensure I don't have this mess again next time. And pushed the commit with the ~/.aws/config in my README.md to my GitHub repo. Present-day Cobus knows how to future-proof against Cobus from IaC-Past.I'm not completely happy with this setup as I really want to keep my
registrar account isolated, but feel the convenience may be worth it in this very specific instance: me managing my personal accounts with no intention to involve anyone else.In the next post, I'll show you the module I created to set up the DNS records for my mail servers for SPF, DMARC, and the other configs. Let me know in the comments if you've done something similar, if I made some horrible mistake, or just confess to some dark secret or mess you created for yourself...
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.