AWS | Community | Amazon EKS Auto Mode ENABLED - Build your super-powered cluster

TL;DR: The Github Repo here setheliot/eks_auto_mode enables you to setup a fully functional Amazon EKS cluster (and app) with Auto Mode enabled, using Terraform.

If you're running on AWS and love Kubernetes, Amazon EKS is a natural choice. It removes the burden of managing the Kubernetes control plane and makes it easy to run Kubernetes workloads on Amazon EC2 or go serverless with AWS Fargate.

Until recently, EKS still required some burden—you had to install and maintain controllers for load balancing and auto scaling, manage IAM roles and policies, and configure IAM configurations for CSI drivers (EBS, EFS, FSx) for persistent storage.

But now, with EKS Auto Mode, announced at AWS re:Invent 2024, much of this complexity is gone! I created a GitHub repo: setheliot/eks_auto_mode for hands-on learning—feel free to dive in now and create an EKS cluster with Auto Mode enabled. Or stick around here, and this blog will take you deeper into what Auto Mode does (and doesn’t) do and how it can benefit you.

Image not found

Figure 1. Build this using the setheliot/eks_auto_mode Github repository. Give it a try!

What does Amazon EKS Auto Mode do?

EKS Auto Mode automates:

Compute: It creates new nodes when pods can't fit onto existing ones, and identifies low utilization nodes for deletion.
Networking: It configures AWS Load Balancers for Kubernetes Service and Ingress resources, to expose cluster apps to the internet.
Storage: It creates EBS Volumes to back Kubernetes storage resources.

Sounds good... What does it really do?

Glad you asked! 😁

The setheliot/eks_auto_mode Github repo uses Terraform to deploy an Amazon EKS cluster with Auto Mode enabled, including a demo app running on the cluster. Using that, I will walk you through how to deploy with Auto Mode enabled, and what Auto Mode actually does for you.

Let's start with enabling it. The following code is what sets Auto Mode to enabled.

with Auto Mode enabled

1
2
3
4
5
6
7
8
9
10
11
#
# EKS Cluster using Auto Mode
module "eks" {

  # ---removed code that is the same ---

  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose"]
  }
}

Code here.

And the following is what the code above replaces.

without Auto Mode

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#
# EKS Cluster
module "eks" {

  # ---removed code that is the same ---

  eks_managed_node_groups = {
    xyz_managed_nodes = {
      name                           = "managed-eks-nodes"
      ami_type                       = "AL2023_x86_64_STANDARD"
      use_latest_ami_release_version = true
      instance_type                  = var.instance_type

      min_size     = 1
      max_size     = 5
      desired_size = 3

      # Setup a custom launch template for the managed nodes
      # Notes these settings are the same as the defaults
      use_custom_launch_template = true
      create_launch_template     = true
    }   
  }     
}

Code here

Note: for my EKS Cluster code without Auto Mode, I am using the GitHub repos: setheliot/xyz_infra_poc and setheliot/xyz_app_poc.

To enable Auto Mode on EKS, you set enabled=true on cluster_compute_config.

But here is something that is a little tricky. Even though this is under cluster_compute_config, this is actually enabling Auto Mode for not just compute, but for everything:

compute
networking (load balancing)
storage

These three are controlled by three separate flags in the SDK for create_cluster, but you get them all when setting enabled=true using the Terraform eks module.

Compute

The above code serves two purposes. One (as just stated) is to enable Auto Mode for everything. The second is to set up how Auto Mode handles compute in our Kubernetes cluster. Let's look at what it is doing.

In the code without Auto Mode, you can see we need to setup a Node Group. This makes use of the Cluster Autoscaler which sets up an AWS Auto Scaling Group for EKS nodes. You provide instance types, AMIs, and parameters about the number of nodes.

By comparison, when using Auto Mode enabled you only specify a node pool. EKS Auto Mode uses Karpenter, an open-source Kubernetes autoscaler. By selecting the general-purpose node pool you are letting EKS Auto Mode select for you the instance size and number of nodes. If, however, you need more control over such things, you can create custom node pools.

Image not found

Figure 2. EKS Node Groups use EC2 Auto Scaling and EKS Auto Mode enabled uses Karpenter

Storage

Persistent storage in a Kubernetes cluster remains available to any container it's configured for and persists independently of the node lifecycle. In the following code we use an Amazon Elastic Block Store (EBS) volume for storage.

with Auto Mode enabled

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#
# EBS Storage Class

resource "kubernetes_storage_class" "ebs" {
  metadata {
    name = "ebs-storage-class"
  }

  storage_provisioner = "ebs.csi.eks.amazonaws.com"
  reclaim_policy      = "Delete"
  volume_binding_mode = "WaitForFirstConsumer"
  parameters = {
    type      = "gp3"
    encrypted = "true"
  }
}

#
# EBS Persistent Volume Claim

resource "kubernetes_persistent_volume_claim_v1" "ebs_pvc" {
  metadata {
    name = local.ebs_claim_name
  }

  spec {
    access_modes = ["ReadWriteOnce"]

    resources {
      requests = {
        storage = "1Gi"
      }
    }

    storage_class_name = "ebs-storage-class"
  }
  wait_until_bound = false
}

Code here

without Auto Mode

Note the extra configuration to create and attach the IAM policy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Retrieve the CSI driver  policy
data "aws_iam_policy" "csi_policy" {
  name = "AmazonEBSCSIDriverPolicy"
}

# Attach the policy to the cluster IAM role
resource "aws_iam_role_policy_attachment" "csi_policy_attachment" {
  policy_arn = data.aws_iam_policy.csi_policy.arn
  role       = local.eks_node_iam_role_name
}

#
# EBS Storage Class

resource "kubernetes_storage_class" "ebs" {
  metadata {
    name = "ebs-storage-class"
  }

  storage_provisioner = "ebs.csi.aws.com"
  reclaim_policy      = "Delete"
  volume_binding_mode = "WaitForFirstConsumer"
  parameters = {
    type   = "gp3"
    fsType = "ext4"
  }
}

#
# EBS Persistent Volume Claim

resource "kubernetes_persistent_volume_claim_v1" "ebs_pvc" {
  metadata {
    name = "ebs-volume-claim"
  }

  spec {
    access_modes = ["ReadWriteOnce"]

    resources {
      requests = {
        storage = "1Gi"
      }
    }

    storage_class_name = "ebs-storage-class"
  }
  wait_until_bound = false
}

Code here

Ok. there are a few things going on here.

With Auto Mode enabled, you don’t need to add IAM permissions for the CSI (Container Storage Interface) driver (AmazonEBSCSIDriverPolicy). Auto Mode has its own CSI driver and permissions are already set up.

Note: EKS purists may notice I took a shortcut when setting up CSI permissions in the "without Auto Mode" case. I should be using IRSA (IAM roles for service accounts), but am not. In the new repo "with Auto Mode enabled" I do indeed use IRSA when needed later (to give my application necessary permissions).

Setting up StorageClass looks pretty similar. However note the different storage_provisioner values.

ebs.csi.eks.amazonaws.com with Auto Mode enabled
ebs.csi.amazonaws.com using the older CSI driver

This can be a little tricky, so make sure you include that eks. It also shows how with Auto Mode enabled, it is using a new CSI driver, specific to Auto Mode.

And from here, the PersistentVolumeClaim setup is the same. Also, later, when configuring your pods, you would reference the PersistentVolumeClaim the same way to create the PersistentVolumes. (Here if you are curious).

Load balancing (networking)

Load balancing enables you to route requests to the pods in your cluster. Let's take a look at how Auto Mode enabled helps with this.

We will break this into two parts

Load balancing part 1 - Install the AWS Load Balancer Controller

with Auto Mode enabled

Nothing 🚫; Zilch 0️⃣; Nada 🙅‍♂️

without Auto Mode

1
2
3
4
5
6
7
#
# Create  policy to give EKS nodes necessary permissions to run the LBC
resource "aws_iam_policy" "alb_controller_custom" {
  name        = "AWSLoadBalancerControllerIAMPolicy"
  description = "IAM policy for AWS Load Balancer Controller"
  policy      = file("${path.module}/policies/iam_policy.json")
}

Code here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#
# AWS Load Balancer Controller
# Retrieve the LBC IAM policy
data "aws_iam_policy" "lbc_policy" {
  name = "AWSLoadBalancerControllerIAMPolicy"
}
# Attach the policy (existing or newly created) to the node IAM role
resource "aws_iam_role_policy_attachment" "alb_policy_node" {
  policy_arn = data.aws_iam_policy.lbc_policy.arn
  role       = local.eks_node_iam_role_name
}
#
# Create the K8s Service Account that will be used by Helm
resource "kubernetes_service_account" "alb_controller" {
  metadata {
    name      = "aws-load-balancer-controller"
    namespace = "kube-system"
  }
}
resource "helm_release" "aws_load_balancer_controller" {
  name       = "aws-load-balancer-controller"
  namespace  = "kube-system"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  set {
    name  = "clusterName"
    value = local.cluster_name
  }
  set {
    name  = "serviceAccount.create"
    value = "false"
  }
  set {
    name  = "serviceAccount.name"
    value = kubernetes_service_account.alb_controller.metadata[0].name
  }
  set {
    name  = "region"
    value = local.region
  }
  set {
    name  = "vpcId"
    value = local.vpc_id
  }
}

Code here

And here we can see the big win for Auto Mode enabled. It is great that AWS Load Balancer Controller (LBC) exists, but installing it and its IAM roles was no fun. Amazon EKS with Auto Mode enabled eliminates this chore. It provides its own controller for creating load balancers, including all permissions already configured to use it.

Load balancing part 2 - Deploy the `Ingress` to create the ALB

Here is the code to create the Application Load Balancer (ALB).

with Auto Mode enabled

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
# Kubernetes Ingress Resource for ALB via AWS Auto Mode 
resource "kubernetes_ingress_v1" "ingress_alb" {
  metadata {
    name      = "${var.prefix_env}-ingress-alb"
    namespace = "default"
    annotations = {
      "alb.ingress.kubernetes.io/scheme" = "internet-facing"
    }
  }
  spec {
    ingress_class_name = "${var.prefix_env}-ingressclass-alb"
    rule {
      http {
        path {
          path      = "/"
          path_type = "Prefix"
          backend {
            service {
              name = kubernetes_service_v1.service_alb.metadata[0].name
              port {
                number = 8080
              }
            }
          }
        }
      }
    }
  }
}

# Kubernetes Service for the App
resource "kubernetes_service_v1" "service_alb" {
  metadata {
    name      = "${var.prefix_env}-service-alb"
    namespace = "default"
    labels = {
      app = var.app_name
    }
  }
  spec {
    selector = {
      app = var.app_name
    }
    port {
      port        = 8080
      target_port = 8080
    }
    type = "ClusterIP"
  }
}

resource "kubernetes_ingress_class_v1" "ingressclass_alb" {
  depends_on   = [null_resource.apply_ingressclassparams_manifest]
  metadata {
    name = "${var.prefix_env}-ingressclass-alb"
  }
  spec {
    # Configures the IngressClass to use EKS Auto Mode
    controller = "eks.amazonaws.com/alb"
    parameters {
      api_group = "eks.amazonaws.com"
      kind      = "IngressClassParams"
      name = "${var.prefix_env}-ingressclassparams-alb"
    }
  }
}

resource "null_resource" "apply_ingressclassparams_manifest" {
  provisioner "local-exec" {
    command = <<EOT
    aws eks --region ${var.aws_region} update-kubeconfig --name ${var.cluster_name}
    kubectl apply -f - <<EOF
apiVersion: eks.amazonaws.com/v1
kind: IngressClassParams
metadata:
  name: "${var.prefix_env}-ingressclassparams-alb"
spec:
  scheme: internet-facing
EOF
    EOT
  }
}

Code here

without Auto Mode

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# This is the IngressClass created when Helm installed the AWS LBC
locals {
    ingress_class_name = "alb"
}

# Kubernetes Ingress Resource for ALB via AWS Load Balancer Controller
resource "kubernetes_ingress_v1" "xyz_ingress_alb" {
  metadata {
    name      = "xyz-ingress-alb-${var.env_name}"
    namespace = "default"
    annotations = {
      "alb.ingress.kubernetes.io/scheme"                   = "internet-facing"
      "alb.ingress.kubernetes.io/target-type"              = "ip"
      "alb.ingress.kubernetes.io/listen-ports"             = "[{\"HTTP\": 80}]"
      "alb.ingress.kubernetes.io/load-balancer-attributes" = "idle_timeout.timeout_seconds=60"
    }
  }
  spec {
    ingress_class_name = local.ingress_class_name
    rule {
      http {
        path {
          path      = "/"
          path_type = "Prefix"
          backend {
            service {
              name = kubernetes_service_v1.xyz_service_alb.metadata[0].name
              port {
                number = 8080
              }
            }
          }
        }
      }
    }
  }
}

# Kubernetes Service for the App
resource "kubernetes_service_v1" "xyz_service_alb" {
  metadata {
    name      = "xyz-service-alb-${var.env_name}"
    namespace = "default"
    labels = {
      app = var.app_name
    }
  }
  spec {
    selector = {
      app = var.app_name
    }
    port {
      port        = 8080
      target_port = 8080
    }
    type = "ClusterIP"
  }
}

Code here

At first, it looks like Auto Mode adds extra work here. Let's look more closely.

IngressClass

Using Auto Mode requires we create an IngressClass. If you look at the "without Auto Mode" code there is no such resource, but you can see it referenced:

ingress_class_name = local.ingress_class_name

The IngressClass referenced was automatically created by Helm when installing the AWS Load Balancer Controller (LBC).

IngressClassParams

IngressClassParams is optional when using LBC, but mandatory for Auto Mode.

Here’s where Terraform presents a challenge:

IngressClassParams is not a standard Kubernetes resource; it is a custom resource managed inside the cluster.
Terraform does not have a built-in resource to create it, so we must use a workaround—executing the AWS CLI via Terraform to apply it.

Controller specification

The configuration that determines whether to use LBC (without Auto Mode) or use the Auto Mode controller is in the IngressClass. With Auto Mode enabled you can see in the code above we use:

controller = "eks.amazonaws.com/alb"

This explicitly tells EKS Auto Mode to manage the ALB.

But without Auto Mode (using LBC), this information is in the IngressClass that was created by Helm. It looks like this:

controller = ingress.k8s.aws/alb

Application-Level Authorization

Earlier I mentioned IRSA (IAM Roles for Service Accounts), which is the best way to grant your pods access to the AWS resources they need. And while having IRSA is more convenient than assigning broad permissions to node IAM roles or manually managing static credentials in Secrets, it still requires setting up an OIDC provider and configuring roles properly.

Here is the code that uses IRSA to give the application pods authorization to access an Amazon DynamoDB table. Auto Mode enabled does not help at all with this. I would like to see it help to set up trust relationships with OIDC, which are going to be common across IAM roles used with IRSA. Hopefully, the EKS team is working on this!

And that's it

With Auto Mode enabled, Amazon EKS simplifies compute, storage, and networking setup—making Kubernetes on AWS easier than ever. Ready to try it? Check out the setheliot/eks_auto_mode repo and deploy your first Auto Mode cluster today!"

Seth Eliot is Principal Resilience Architect at Arpio, where he helps AWS customers implement Disaster Recovery and Cyber-Resilience strategies to protect their workloads. Before that, he was Principal Engineer at AWS and Microsoft.

https://linktr.ee/setheliot

Select your cookie preferences

Site Terms, Privacy, and more.

Amazon EKS Auto Mode ENABLED - Build your super-powered cluster

Deploy a Fully Functional Amazon EKS Cluster with Auto Mode Using Terraform – See How It Simplifies Operations

What does Amazon EKS Auto Mode do?

Sounds good... What does it really do?

Compute

Storage

Load balancing (networking)

Load balancing part 1 - Install the AWS Load Balancer Controller

Load balancing part 2 - Deploy the `Ingress` to create the ALB

Application-Level Authorization

And that's it

Comments

Select your cookie preferences

Site Terms, Privacy, and more.

Site Terms, Privacy, and more.

Amazon EKS Auto Mode ENABLED - Build your super-powered cluster

Deploy a Fully Functional Amazon EKS Cluster with Auto Mode Using Terraform – See How It Simplifies Operations

What does Amazon EKS Auto Mode do?

Sounds good... What does it really do?

Compute

Storage

Load balancing (networking)

Load balancing part 1 - Install the AWS Load Balancer Controller

Load balancing part 2 - Deploy the Ingress to create the ALB

Application-Level Authorization

And that's it

Comments

Load balancing part 2 - Deploy the `Ingress` to create the ALB