A Simple Config to Reduce Cross-AZ traffic of EKS Workloads

Introduction

It is a best practice to deploy your EKS workloads across multiple availability zones to improve your applications' high availability. However, this incurs cross-AZ data transfer, which leads to additional costs.

There have been some explorations on how to maintain the HA of your applications while reducing cross-AZ traffic. Please refer to the following AWS blogs for more details:

Exploring the effect of Topology Aware Hints on network traffic in Amazon Elastic Kubernetes Service

Optimize AZ traffic costs using Amazon EKS, Karpenter, and Istio

Kubernetes New Feature

In Kubernetes v1.31, the “Traffic distribution” feature of Service became beta and is enabled by default. It allows you to express preferences (such as routing to topologically closer endpoints). This can help optimize performance, cost, or reliability.

Exploration

In this post, we will explore how "Traffic Distribution" helps in reducing the cross-AZ data transfer of EKS workloads.

Environment setup, create:

An EKS cluster;
Three worker nodes, distributed to three AZs evenly;
An AWS Distro for OpenTelemetry collector to ingest tracing data to AWS X-Ray;
A sample retail store application;

Check the worker nodes and make sure that they are in different AZs.

1
kubectl get node -ocustom-columns="Name:.metadata.name,Zone:.metadata.labels.topology\.kubernetes\.io/zone"

We should be able to see that each node is in a different AZ. In other words, each node represents one AZ.

Image not found

The following diagram is the architecture of the sample retail store application. Users visit this store through ELB to the UI service. Traffic from the UI service to the Catalog service is load balanced by Kubernetes Service.

Image not found

Case 1: Pods of UI and Catalog are evenly distributed

At the beginning, pods of each service are spread to different nodes, meaning that each pod is in a unique AZ.

AZ	# of UI pods	# of Catalog pods
AZ1	1	1
AZ2	1	1
AZ3	1	1

Image not found

Start a pod to generate traffic to the UI service and then check the Trace Map on the X-Ray console. In the default setup, traffic from the UI service is load balanced to all catalog pods in different AZs. This incurs cross-AZ data transfer.

Image not found

Update the configuration of the catalog service to configure trafficDistribution.

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: Service
metadata:
  name: catalog
  namespace: catalog
spec:
  trafficDistribution: PreferClose
  ports:
  - name: http
  ...

Get endpointslices information of the catalog service.

1
kubectl get endpointslices -l kubernetes.io/service-name=catalog -n catalog -o yaml

We can see a hint of AZ information was added to each endpoint.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: v1
items:
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 10.0.12.162
    conditions:
      ready: true
      serving: true
      terminating: false
    hints:
      forZones:
      - name: ap-northeast-1d
    nodeName: ip-10-0-12-191.ap-northeast-1.compute.internal
    ...

Start a pod to generate traffic to the UI service again.

Check the Trace Map on the X-Ray console (You may need to refresh the whole web page.). Now we can see that each UI pod sends requests to only one catalog pod, since its traffic is routed to the same AZ only.

Conclusion: After configuring trafficDistribution, traffic from UI are only routed to Catalog endpoint in the same AZ.

Image not found

Case 2: Pods of Catalog are NOT evenly distributed

In this case, we will simulate a situation where pods of UI service are evenly distributed to all AZs, but pods of Catalog service are NOT.

AZ	# of UI pods	# of Catalog pods
AZ1	1	0
AZ2	1	2
AZ3	1	1

Cordon one of the worker nodes.

1
2
export NODE=<node name>
kubectl cordon $NODE

Delete the catalog pod on this node.

1
2
3
kubectl -n catalog delete pod \
  $(kubectl -n catalog get po -owide -l app.kubernetes.io/component=service \
  | grep $NODE | awk '{print $1}')

Check the pods again. Now two out of three catalog pods are running on the same node. That means one of three AZs has no catalog pod.

1
2
kubectl -n ui get po -owide -l app.kubernetes.io/component=service
kubectl -n catalog get po -owide -l app.kubernetes.io/component=service

Image not found

Start a pod to generate traffic to the UI service again.

Check the Trace Map on the X-Ray console. We can see that:

A1 has no target endpoint in the same AZ, so its traffic is routed to all AZs;
A2 has two target endpoints in the same AZ, so its traffic is load balanced to B1 and B2 in the same AZ;
A3 has one target endpoint in the same AZ, so its traffic is routed to the B3 in the same AZ;

Conclusion: When there is no target endpoint in the same AZ of the UI pod, its traffic will be routed to all other AZs. But traffic of UI pods in the other two AZs still stick to the same AZ.

Image not found

Case 3: Pods of UI are NOT evenly distributed

In this case, we will simulate a situation where pods of Catalog service are evenly distributed to all AZs, but pods of UI service are NOT.

AZ	# of UI pods	# of Catalog pods
AZ1	1	1
AZ2	2	1
AZ3	0	1

Uncordon the node.

1
kubectl uncordon $NODE

Restart pods of Catalog service and make sure they are distributed to three AZs evenly.

Cordon one of the worker nodes.

1
2
export NODE=<node name>
kubectl cordon $NODE

Delete the UI pod on this node.

1
2
3
kubectl -n ui delete pod \
  $(kubectl -n ui get po -owide -l app.kubernetes.io/component=service \
  | grep $NODE | awk '{print $1}')

Check the pods again. Now two out of three UI pods are running on the same node. That means one of three AZs has no UI pod.

1
2
kubectl -n ui get po -owide -l app.kubernetes.io/component=service
kubectl -n catalog get po -owide -l app.kubernetes.io/component=service

Image not found

Start a pod to generate traffic to the UI service again.

Check the Trace Map on the X-Ray console. We can see that:

A1 has one target endpoint in the same AZ, so its traffic is routed to B1 the same AZs;
A2 and A3 are in the same AZ, their traffic are routed to B2 in the same AZ;
B3 is not in this Trace Map at all, because there is no traffic to B3;

Conclusion: If no source pod in the same AZ, the destination pod will not have any incoming traffic. But other destination pods cab be overloaded.

Image not found

Conclusion

From the experiment above, we are able to avoid or reduce cross-AZ network traffic of EKS workloads by simply enabling the trafficDistribution of the destination service. But we also see some drawbacks if the source or destination services are not spread to all AZs.

Case #	UI pods	Catalog pods	Result
Case 1	1:1:1	1:1:1	Load balanced and no cross-AZ traffic.
Case 2	1:1:1	0:2:1	Traffic from AZ1 is routed to other AZs; Load NOT balanced.
Case 3	1:2:0	1:1:1	Catalog in AZ3 has no incoming traffic; Load NOT balanced.

Therefore, it is a best practice to spread endpoints of both the source and destination services to each AZ as evenly as possible (e.g., by TopologySpreadConstraints). This will help us to:

Avoid traffic imbalance and potential overload;
Keep data transfer within the same AZ.

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Select your cookie preferences

Site Terms, Privacy, and more.

A Simple Config to Reduce Cross-AZ traffic of EKS Workloads

In this post, we will explore how "Traffic Distribution" helps in reducing the cross-AZ data transfer of EKS workloads.

Introduction

Kubernetes New Feature

Exploration

Case 1: Pods of UI and Catalog are evenly distributed

Case 2: Pods of Catalog are NOT evenly distributed

Image not found

Case 3: Pods of UI are NOT evenly distributed

Conclusion

Comments