A Simple Config to Reduce Cross-AZ traffic of EKS Workloads
In this post, we will explore how "Traffic Distribution" helps in reducing the cross-AZ data transfer of EKS workloads.
Xiangyan Wang
Amazon Employee
Published Dec 17, 2024
It is a best practice to deploy your EKS workloads across multiple availability zones to improve your applications' high availability. However, this incurs cross-AZ data transfer, which leads to additional costs.
There have been some explorations on how to maintain the HA of your applications while reducing cross-AZ traffic. Please refer to the following AWS blogs for more details:
Exploring the effect of Topology Aware Hints on network traffic in Amazon Elastic Kubernetes Service
In Kubernetes v1.31, the “Traffic distribution” feature of Service became beta and is enabled by default. It allows you to express preferences (such as routing to topologically closer endpoints). This can help optimize performance, cost, or reliability.
In this post, we will explore how "Traffic Distribution" helps in reducing the cross-AZ data transfer of EKS workloads.
Environment setup, create:
- An EKS cluster;
- Three worker nodes, distributed to three AZs evenly;
- An AWS Distro for OpenTelemetry collector to ingest tracing data to AWS X-Ray;
- A sample retail store application;
Check the worker nodes and make sure that they are in different AZs.
We should be able to see that each node is in a different AZ. In other words, each node represents one AZ.
The following diagram is the architecture of the sample retail store application. Users visit this store through ELB to the UI service. Traffic from the UI service to the Catalog service is load balanced by Kubernetes Service.
At the beginning, pods of each service are spread to different nodes, meaning that each pod is in a unique AZ.
| AZ | # of UI pods | # of Catalog pods |
|---|---|---|
| AZ1 | 1 | 1 |
| AZ2 | 1 | 1 |
| AZ3 | 1 | 1 |
Start a pod to generate traffic to the UI service and then check the Trace Map on the X-Ray console. In the default setup, traffic from the UI service is load balanced to all catalog pods in different AZs. This incurs cross-AZ data transfer.
Update the configuration of the catalog service to configure
trafficDistribution.Get
endpointslices information of the catalog service.We can see a
hint of AZ information was added to each endpoint.Start a pod to generate traffic to the UI service again.
Check the Trace Map on the X-Ray console (You may need to refresh the whole web page.). Now we can see that each UI pod sends requests to only one catalog pod, since its traffic is routed to the same AZ only.
Conclusion: After configuring
trafficDistribution, traffic from UI are only routed to Catalog endpoint in the same AZ.
In this case, we will simulate a situation where pods of UI service are evenly distributed to all AZs, but pods of Catalog service are NOT.
| AZ | # of UI pods | # of Catalog pods |
|---|---|---|
| AZ1 | 1 | 0 |
| AZ2 | 1 | 2 |
| AZ3 | 1 | 1 |
Cordon one of the worker nodes.
Delete the catalog pod on this node.
Check the pods again. Now two out of three catalog pods are running on the same node. That means one of three AZs has no catalog pod.
Start a pod to generate traffic to the UI service again.
Check the Trace Map on the X-Ray console. We can see that:
- A1 has no target endpoint in the same AZ, so its traffic is routed to all AZs;
- A2 has two target endpoints in the same AZ, so its traffic is load balanced to B1 and B2 in the same AZ;
- A3 has one target endpoint in the same AZ, so its traffic is routed to the B3 in the same AZ;
Conclusion: When there is no target endpoint in the same AZ of the UI pod, its traffic will be routed to all other AZs. But traffic of UI pods in the other two AZs still stick to the same AZ.
In this case, we will simulate a situation where pods of Catalog service are evenly distributed to all AZs, but pods of UI service are NOT.
| AZ | # of UI pods | # of Catalog pods |
|---|---|---|
| AZ1 | 1 | 1 |
| AZ2 | 2 | 1 |
| AZ3 | 0 | 1 |
Uncordon the node.
Restart pods of Catalog service and make sure they are distributed to three AZs evenly.
Cordon one of the worker nodes.
Delete the UI pod on this node.
Check the pods again. Now two out of three UI pods are running on the same node. That means one of three AZs has no UI pod.
Start a pod to generate traffic to the UI service again.
Check the Trace Map on the X-Ray console. We can see that:
- A1 has one target endpoint in the same AZ, so its traffic is routed to B1 the same AZs;
- A2 and A3 are in the same AZ, their traffic are routed to B2 in the same AZ;
- B3 is not in this Trace Map at all, because there is no traffic to B3;
Conclusion: If no source pod in the same AZ, the destination pod will not have any incoming traffic. But other destination pods cab be overloaded.
From the experiment above, we are able to avoid or reduce cross-AZ network traffic of EKS workloads by simply enabling the trafficDistribution of the destination service. But we also see some drawbacks if the source or destination services are not spread to all AZs.
| Case # | UI pods | Catalog pods | Result |
|---|---|---|---|
| Case 1 | 1:1:1 | 1:1:1 | Load balanced and no cross-AZ traffic. |
| Case 2 | 1:1:1 | 0:2:1 | Traffic from AZ1 is routed to other AZs; Load NOT balanced. |
| Case 3 | 1:2:0 | 1:1:1 | Catalog in AZ3 has no incoming traffic; Load NOT balanced. |
Therefore, it is a best practice to spread endpoints of both the source and destination services to each AZ as evenly as possible (e.g., by TopologySpreadConstraints). This will help us to:
- Avoid traffic imbalance and potential overload;
- Keep data transfer within the same AZ.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.