Building an Amazon EKS Cluster Preconfigured to Run Asynchronous Batch Tasks
Use scalable data storage solutions with eksctl 'quickstart' template.
Ahmad Tariq
Amazon Employee
Published Oct 2, 2023
Last Modified Apr 19, 2024
In the diverse ecosystem of cloud-native applications, there are times when real-time interactivity or immediate responses aren't your prime directive. Instead, the crux lies in executing asynchronous background tasks with precision and efficiency. Whether you're transferring large files, harmonizing disparate datasets, or orchestrating complex job schedules, you need an environment built to handle workload fluctuations and optimized for resource utilization. With Amazon EKS, the EFS CSI Driver for scalable storage, and the Cluster Autoscaler for automated scaling policies, you can effortlessly establish an EKS cluster that is not only optimized for asynchronous operations but also adapts intelligently to workload demands.
This tutorial shows you how to create a managed node groups-based Amazon EKS cluster using an eksctl “quickstart” template. The cluster is designed to integrate seamlessly with Amazon SQS for decoupled job orchestration and Amazon EFS for persistent, scalable storage, setting the stage for the advanced use-cases explored in the next part of this series.
| About | |
|---|---|
| ✅ AWS experience | 200 - Intermediate |
| ⏱ Time to complete | 30 minutes |
| 🧩 Prerequisites | - AWS Account |
| 📢 Feedback | Any feedback, issues, or just a 👍 / 👎 ? |
| ⏰ Last Updated | 2023-10-02 |
Before you begin this tutorial, you need to:
- Install the latest version of kubectl. To check your version, run:
kubectl version --short. - Install the latest version of eksctl. To check your version, run:
eksctl info.
This tutorial is the first installment in a series focused on optimizing Amazon EKS for batch processing workloads. It aims to guide you through the preconfiguration of a cluster equipped to handle small- to medium-scale asynchronous background tasks, using an eksctl 'quickstart' template. This template not only facilitates the execution of batch jobs but also adds an extra layer of security, aligning with best practices for production deployments. The template configures the following components:
- Autoscaling: The managed node groups in this setup use a "c5a.xlarge" instance type, ideal for compute-bound applications that benefit from high performance processors and well-suited for batch processing workloads. With a minimum size of "3" and a maximum size of "6", these node groups can dynamically adapt to workload demands. The subnet tags allow the Kubernetes Cluster Autoscaler (CA) to dynamically scale your cluster on demand.
- Authentication: Necessary IAM Roles for Service Accounts (IRSAs) mappings to enable communication between Kubernetes pods and AWS services. This includes the Kubernetes Cluster Autoscaler (CA) for dynamic scaling, Amazon ECR for private repository access of container images, essential for batch workloads, the Amazon EBS CSI Driver for block-level persistent data storage, Amazon EFS CSI Driver for shared file system storage across multiple nodes. Additionally, an OpenID Connect (OIDC) endpoint enables seamless and secure communication.
- Add-ons: The template includes the installation of the EFS CSI Driver add-on to facilitate shared file systems for complex batch jobs.
- Private Networking: Managed node groups utilize private networking and a NAT gateway to bolster security by limiting direct internet access.
- Monitoring: An Amazon CloudWatch IAM policy is attached to the IAM Role for Service Account (IRSA), aiding optional components like CloudWatch Container Insights to collect and summarize metrics and logs.
In this section, you will configure the Amazon EKS cluster to meet the specific demands of high-traffic microservice applications. By creating this
cluster-config.yaml file, you'll define the settings for IAM roles, scalable resources, private networking, and monitoring. These configurations are essential for ensuring that the cluster is robust, scalable, and secure, with optimized performance for dynamic scalability and data persistence.To create the cluster config:
- Create a
cluster-config.yamlfile and paste the following contents into it. Replace theregion.
Now we're ready to create our Amazon EKS cluster. This process takes several minutes to complete. If you'd like to monitor the status, see the AWS CloudFormation console.
- Create the EKS cluster by running the following command:
Note: if you receive an “Error: checking AWS STS access” in the response, be sure to check that you’re using the right user identity for the current shell session. Depending on how you configured the AWS CLI, you may also need to specify a named profile (for example,
--profile clusteradmin).Upon completion, you should see the following response output:
When the previous command completes, verify that all of your nodes have reached the
Ready state with the following command:The expected output should look like this:
Check whether the EFS CSI driver add-on is installed on your cluster. Replace the sample value for
region.The expected output should look like this:
Verify all the service accounts created in the cluster with the following command:
The expected output should look like this:
Congratulations! You've successfully set up the foundational infrastructure of your Amazon EKS cluster, laying the essential groundwork for deploying data-intensive batch processes. This setup provides you with fully operational environment needed to begin deploying batch jobs and job queues.
Now, you're ready to orchestrate batch processing workloads in your Kubernetes cluster, leveraging Amazon SQS as a job queue and Amazon EFS for data persistence. For a comprehensive walkthrough, refer to Managing Asynchronous Tasks with SQS and EFS Persistent Storage in Amazon EKS. This tutorial guides you through the process of setting up Amazon SQS for job queue management and integrating Amazon EFS for persistent storage. Lastly, you can optionally configure the EFS CSI Driver Add-On to handle complex batch jobs that require a shared file system across multiple nodes.
To avoid incurring future charges, you should delete the resources created during this tutorial. You can delete the EKS cluster with the following command. Replace the
region.Upon the successful completion of this tutorial, you've effectively set up an Amazon EKS cluster tailored for batch processing tasks. By choosing compute-optimized EC2 instances and implementing key IAM policies, you've engineered a cluster that's both agile and robust. This tutorial has guided you through the creation and deployment of your Amazon EKS cluster, and the verification of node and pod health. To extend your cluster's capabilities, consider setting up the Cluster Autoscaler to dynamically adjust the number of worker nodes based on resource requirements, ensuring optimal resource utilization and cost-efficiency, or Amazon EFS for shared, persistent storage across multiple nodes or even across different availability zones. These additional configurations will furnish you with a versatile, fully operational environment, ready for deploying both stateless and data-intensive batch jobs.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.