logo
Menu

Cluster Autoscaler configure on AWS EKS -1.25.

Cluster Autoscaler plays a vital role in a Kubernetes cluster by ensuring adequate computing resources are available by adding nodes to a cluster and keeping infrastructure costs down by removing nodes.

Published Mar 30, 2024

Introduction :

The Kubernetes Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail or are rescheduled onto other nodes. The Cluster Autoscaler uses Auto Scaling groups. For more information, see Cluster Autoscaler on AWS.

Step 2: Verify how many nodes and pods are running

Node :
1
2
3
4
[root@ip-172-31-88-31 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-2-24.ec2.internal Ready <none> 22m v1.25.16-eks-5e0fdde
ip-192-168-47-180.ec2.internal Ready <none> 22m v1.25.16-eks-5e0fdde
Pods:
1
2
3
4
5
6
7
8
[root@ip-172-31-88-31 ~]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-8qc9n 2/2 Running 0 23m
kube-system aws-node-f7rkv 2/2 Running 0 23m
kube-system coredns-5b676c58cf-67rnt 1/1 Running 0 29m
kube-system coredns-5b676c58cf-sr5l5 1/1 Running 0 29m
kube-system kube-proxy-mbnwt 1/1 Running 0 23m
kube-system kube-proxy-rb7vn 1/1 Running 0 23m

Step 3: Create a IAM OIDC Provider

IAM OIDC is used for authorizing the Cluster Autoscaler to launch or terminate instances under an Auto Scaling group. Open EKS Dashboard and copy a OpenID Connect Provider link
  • Open a IAM Providers
  • Click “Add provider,” select “OpenID Connect,” and click “Get thumbprint” as shown below:
Then enter the “Audience” (sts.amazonaws.com in our example pointing to the AWS STS, also known as the Security Token Service) and click the add provider.
  • Adding identity information on identity providers
Step 4: Create IAM Policy
Create a Policy with necessary permission.
  • To create the policy with the necessary permissions, save the below file as AmazonEKSClusterAutoscalerPolic
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeScalingActivities",
"autoscaling:DescribeTags",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplateVersions"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeImages",
"ec2:GetInstanceTypesFromInstanceRequirements",
"eks:DescribeNodegroup"
],
"Resource": [
"*"
]
}
]
}
Review and create a policy

Step 5 : Create a IAM Role for the provider.

  • Create role
Select the web identity
  • Select identity provide and audience click next.
Add Policy AmazonEKSClusterAutoscalerPolicy
Click Next and provide Role Name : EKS_Autoscaler
verify the IAM role and make sure the policy is attached.
Edit the “Trust relationships.”
Before Edit “Trust relationships.”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::256050093938:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/41FF760102BA37A6D1AF25984B291ABE"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/41FF760102BA37A6D1AF25984B291ABE:aud": "sts.amazonaws.com"
}
}
}
]
}
After Edit “Trust relationships.”
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::256050093938:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/41FF760102BA37A6D1AF25984B291ABE"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/41FF760102BA37A6D1AF25984B291ABE:aud": "sts.amazonaws.com",
"oidc.eks.us-east-1.amazonaws.com/id/41FF760102BA37A6D1AF25984B291ABE:sub": "system:serviceaccount:kube-system:cluster-autoscaler"
}
}
}
]
}
Then click “Update Trust Policy” to save it.

Step 6 : Deploy a Cluster Autoscaler

Next, we deploy Cluster Autoscaler. To do so, you must use the Amazon Resource Names (ARN) number of the IAM role created in our earlier step.The content intended to save into a file (make sure you copy all of the content presented over the next page):
Modify below two lines
  • Line 8 : change IAM Role name
  • Line 159 : --node-group-auto-discovery = This is used by CA to discover the Auto Scaling group based on its tag.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::256050093938:role/EKS_Autoscaler
name: cluster-autoscaler
namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["events", "endpoints"]
verbs: ["create", "patch"]
- apiGroups: [""]
resources: ["pods/eviction"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/status"]
verbs: ["update"]
- apiGroups: [""]
resources: ["endpoints"]
resourceNames: ["cluster-autoscaler"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["watch", "list", "get", "update"]
- apiGroups: [""]
resources:
- "pods"
- "services"
- "replicationcontrollers"
- "persistentvolumeclaims"
- "persistentvolumes"
verbs: ["watch", "list", "get"]
- apiGroups: ["extensions"]
resources: ["replicasets", "daemonsets"]
verbs: ["watch", "list", "get"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["watch", "list"]
- apiGroups: ["apps"]
resources: ["statefulsets", "replicasets", "daemonsets"]
verbs: ["watch", "list", "get"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses", "csinodes"]
verbs: ["watch", "list", "get"]
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["create"]
- apiGroups: ["coordination.k8s.io"]
resourceNames: ["cluster-autoscaler"]
resources: ["leases"]
verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create","list","watch"]
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
verbs: ["delete", "get", "update", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
app: cluster-autoscaler
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
labels:
app: cluster-autoscaler
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'
spec:
serviceAccountName: cluster-autoscaler
containers:
- image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.2
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 500Mi
requests:
cpu: 100m
memory: 500Mi
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/ashish
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt #/etc/ssl/certs/ca-bundle.crt for Amazon Linux Worker Nodes
readOnly: true
imagePullPolicy: "Always"
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs/ca-bundle.crt"
To deploy CA, save the following content presented after the command below in a file and run this provided command:
1
2
3
4
5
6
7
[root@ip-172-31-88-31 ~]# kubectl apply -f cluster-autoscaler.yaml
serviceaccount/cluster-autoscaler created
clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created
role.rbac.authorization.k8s.io/cluster-autoscaler created
clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
deployment.apps/cluster-autoscaler created
The expected results are displayed below.
1
2
3
4
5
6
7
8
9
[root@ip-172-31-88-31 ~]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-8qc9n 2/2 Running 0 74m
kube-system aws-node-f7rkv 2/2 Running 0 74m
kube-system cluster-autoscaler-68d5bd4958-lb7cw 1/1 Running 0 49s
kube-system coredns-5b676c58cf-67rnt 1/1 Running 0 80m
kube-system coredns-5b676c58cf-sr5l5 1/1 Running 0 80m
kube-system kube-proxy-mbnwt 1/1 Running 0 74m
kube-system kube-proxy-rb7vn 1/1 Running 0 74m
Configure metrics-server plugins :
URL : kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@ip-172-31-88-31 ~]# kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
[root@ip-172-31-88-31 ~]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-8qc9n 2/2 Running 0 83m
kube-system aws-node-f7rkv 2/2 Running 0 83m
kube-system cluster-autoscaler-68d5bd4958-lb7cw 1/1 Running 0 10m
kube-system coredns-5b676c58cf-67rnt 1/1 Running 0 89m
kube-system coredns-5b676c58cf-sr5l5 1/1 Running 0 89m
kube-system kube-proxy-mbnwt 1/1 Running 0 83m
kube-system kube-proxy-rb7vn 1/1 Running 0 83m
kube-system metrics-server-5bc776bf9b-4zwff 0/1 Running 0 9s

Troubleshoot :

verify the logs by issuing this command:
1
kubectl logs -l app=cluster-autoscaler -n kubesystem -f

Conclusion :

Cluster Autoscaler plays a vital role in a Kubernetes cluster by ensuring adequate computing resources are available by adding the nodes to a cluster and keeping infrastructure costs down by removing nodes