Effortless AWS EKS Cluster Autoscaling using Terraform
In this post, we’ll walk through how to use Terraform to implement autoscaling for your AWS EKS cluster, allowing you to effortlessly adjust the size of your Kubernetes infrastructure as demand changes.
Published Nov 17, 2024
Last Modified Nov 19, 2024
Introduction :
When running Kubernetes on AWS, particularly with EKS, autoscaling provides a way to dynamically adjust the number of compute resources (EC2 instances) in your cluster. Autoscaling ensures that your cluster has enough capacity during high-demand periods, while also reducing resources when demand drops—helping you avoid overprovisioning and saving costs.
By automating this process using terraform, EKS takes the manual effort out of scaling, offering:
- Better performance: Ensures that your application has the resources it needs to handle traffic spikes.
- Cost savings: Automatically reduces the number of instances when demand decreases, so you're only paying for what you use.
- Improved operational efficiency: Reduces the burden on operators to monitor and manually scale resources.
Here we are using Cluster Autoscaler is a tool that automatically adjusts the number of nodes in your EKS cluster based on the resource needs of your pods. It scales your EC2 instances up or down depending on the current usage and the pod’s resource requests.
- Scale up: If the scheduler can’t place a pod on any available node due to resource limitations, the Cluster Autoscaler will add new nodes to the cluster.
- Scale down: If there are nodes with underutilized resources and no pending pods, the Cluster Autoscaler will remove those nodes to save costs.
- Ensure you have a Terraform environment set up.
- AWS credentials are configured, typically via
~/.aws/credentialsor environment variables. - A basic understanding of Terraform and AWS services like EC2, Auto Scaling Groups (ASG), and Load Balancers.
To check connection to EKS cluster run the following command:
1- I highly recommend testing the provider first before deploying the autoscaller. It can save you a lot of time. File name is
iam-test.tf.
iam-test.tf
terraform apply
Check kubectl pods using command line:Next is to create a pod to test IAM roles for service accounts. First, we are going to omit annotations to bind the service account with the role. The way it works, you create a service account and use it in your pod spec. It can be anything, deployment, statefulset, or some jobs. Give it a name
aws-test.yaml
Then you need to apply it using
kubectl apply -f <folder/file> command.Verify via command line :
Let's add missing annotation to the service account and redeploy the pod. Don't forget to replace
XXXXXXXXX with your AWS account number.Then you need to apply it using
kubectl apply -f <folder/file> command.- Next, let's deploy the sample application and expose it using public and private load balancers. The first is a deployment object with a base nginx image. File name is
k8s/deployment.yaml.
deployment.yaml
To expose the application to the internet, you can create a Kubernetes service of a type load balancer and use annotations to configure load balancer properties. By default, Kubernetes will create a load balancer in public subnets, so you don't need to provide any additional configurations. Also, if you want a new network load balancer instead of the old classic load balancer, you can add aws-load-balancer-type equal to nlb. Call it
k8s/public-lb.yamlCreate both deployment and the service objects.
- Find load balancer in AWS console by name. Verify that LB was created in public subnets
- Sometimes if you have a large infrastructure with many different services, you have a requirement to expose the application only within your VPC. For that, you can create a private load balancer.
- To make it private, you need additional annotation: aws-load-balancer-internal and then provide the CIDR range. Usually, you use 0.0.0.0/0 to allow any services within your VPC to access it. Give it a name
private-lb.yaml.
private-lb.yaml
Verify via command line :
Finally, we got to the EKS autoscaller. We will be using OpenID connect provider to create an IAM role and bind it with the autoscaller. Let's create an IAM policy and role first. It's similar to the previous one, but autoscaller will be deployed in the kube-system namespace.
iam-autoscaler.tf
Let's apply the terraform again to create those objects.
Deploy a Cluster Autoscaler
Next, we deploy Cluster Autoscaler. To do so, you must use the Amazon Resource Names (ARN) number of the IAM role created in our earlier step.The content intended to save into a file (make sure you copy all of the content presented over the next page):Modify below two lines
- Line 8 : change IAM Role name
- Line 159 : --node-group-auto-discovery = This is used by CA to discover the Auto Scaling group ba
- sed on its tag.
Verify via command line :
terraform destroy
This basic setup will allow you to autoscale EC2 instances in AWS based on load. You can easily adjust the instance size, scaling policies, and other parameters to better fit your workload. Terraform provides a powerful way to automate and manage infrastructure as code, including setting up autoscaling groups for handling variable traffic and workloads efficiently.