Amazon EKS Upgrade release notes (v1.26 to v1.27)

As one of the largest and most popular open-source projects for building cloud-native applications, the Kubernetes project is continually integrating new features, design requests, and bug fixes through version upgrades. New version updates are available on average every three months.

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes platform provided by AWS to enable customers to deploy, manage, and scale Kubernetes clusters on the AWS Cloud. As EKS is based on open-source Kubernetes, AWS constantly updates EKS to ensure compatibility with the latest version of Kubernetes, while providing backward compatibility for older versions.

At AWS, Containers Specialists work closely with customers daily to help them with the migration and upgrades of large-scale EKS deployments. In this simple blog post, we have consolidated a list of key items to take note of as you go through the EKS upgrade from v1.26 to v1.27.

Kubernetes v1.27 has removed the alpha seccomp annotations. These annotations, specifically seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io, were deprecated in Kubernetes v1.19 and have now been completely removed. It's crucial to update your configurations to use the securityContext.seccompProfile field for Pods or containers to configure seccomp profiles.

The —container-runtime command line argument for the kubelet was removed. Amazon EKS has been using containerd as the default container runtime since version v1.24, making this argument obsolete. It's essential to remove this argument from all your node creation workflows and build scripts to prevent potential errors during the node bootstrap process.

Kubernetes v1.27 also brings improvements to kubelet API query handling. The default kubeAPIQPS has been increased from 50, and the default kubeAPIBurst has been raised from to 100. These enhancements allow the kubelet to handle a higher volume of API queries, resulting in improved response times and overall performance - Pod launches are quciker and cluster operations are more effective. While no immediate action is required, it's worth being aware of these changes as they may impact API server load.

The new version introduces enhanced Pod topology spread policies. New features such as minDomain, nodeAffinityPolicy, and nodeTaintPolicy provide finer control over Pod distribution. Additionally, the matchLabelKeys field in topologySpreadConstraints allows for more precise selection of Pods for spreading calculations during rolling upgrades. If these features align with your use case, you may want to review and update your Pod specifications accordingly.

Kubernetes v1.27 has promoted to beta a new policy mechanism for StatefulSets that controls the lifetime of their PersistentVolumeClaims (PVCs). This new PVC retention policy allows you to specify whether PVCs should be automatically deleted or retained when a StatefulSet is deleted or when replicas in the StatefulSet are scaled down. It's advisable to review your StatefulSet configurations and consider implementing these new PVC retention policies if they would be beneficial for your workloads.

Amazon EKS version v1.27 has enabled the goaway-chance option in the Kubernetes API server. This feature helps prevent HTTP/2 client connections from being stuck on a single API server instance by randomly closing connections. If your workloads running on the Amazon EKS cluster use clients that are not compatible with HTTP GOAWAY, it's recommended to update these clients to handle GOAWAY by reconnecting on connection termination.

Please refer to the following reference links for add-on versions, pre-requisites and update instructions

Apart from the above mentioned core add-ons, you might also be running other add-ons like AWS Load Balancer controller, EBS CSI driver, Cluster Autoscaler, Karpenter, Prometheus, etc. These add-ons will also need to be validated individually for version compatibility.

For more best practices on EKS Cluster Upgrades, refer to the official AWS EKS documentation here.

Gladwin Neo is a Containers Solutions Architect at AWS. He is a tech enthusiast with a passion for containers. He is now focusing on helping customers from a wide range of industries to modernize their workloads through the use of Containers technologies in AWS which includes Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS).