EKS Abnormal Notification Received After Deleting Subnet

One evening, I received the following email from AWS:

Title: [Action required] Resolve Amazon EKS cluster health issues

Content: The following is a list of affected clusters with their cluster arns, cluster health status, and corresponding cluster health issues. The health of an EKS cluster is a shared responsibility between AWS and customers. You must resolve these issues to maintain operational stability for your EKS cluster.

At the time, I skimmed through the email and thought that there might be a problem with the EKS infrastructure. However, the next day, I took a closer look and saw that there was an abnormality with EKS health. When I checked the health of the affected cluster, I found that the associated subnet had been deleted.

The deletion of the subnet itself was a normal operation, as it was an intentional one. However, we were not aware that it was being used by EKS. Fortunately, this EKS cluster was a test environment, not a production environment, so it did not cause any problems.

We reviewed the timeline between the subnet deletion and the notification:

16:52 Subnet deleted

17:01 EKS cluster abnormality notification

The notification arrived within 9 minutes. The official documentation states that it can take up to 3 hours, so we cannot say for sure that the notification will always arrive within this timeframe.

https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#cluster-health-status

However, even if a similar problem occurred in a production environment, the notification could provide a clue to the problem. (Of course, the application itself would likely detect the problem before then.)

This story illustrates that you may receive a notification from AWS if you perform an operation that affects EKS cluster health. I was impressed by AWS's attention to detail and the mechanisms in place to enhance service reliability.