Easily Monitor Containerized Applications with Amazon CloudWatch Container Insights
How to collect, aggregate, and analyze metrics from your containerized applications using Amazon CloudWatch Container Insights.
Olawale Olaleye
Amazon Employee
Published Oct 2, 2023
Last Modified Mar 18, 2024
Step 1: Set up Container Insights on Amazon EKS
Step 2: Deploy a Container Application in the Cluster
Step 3: Use CloudWatch Logs Insights Query to search and analyze container logs
To Run a CloudWatch Logs Insights Sample Query:
Step 4: Monitor Performance of the Application with Container Insights
View Container Insights Dashboard Metrics
View Additional Amazon EKS and Kubernetes Container Insights Metrics
Monitoring containerized applications requires precision and efficiency. One way to handle the complexities of collecting and summarizing metrics from your applications is to use Amazon CloudWatch Container Insights. As the performance metrics of your containers change, Container Insights offers real-time data, enabling you to maintain consistent application performance through informed decisions.
Building on the Amazon EKS cluster from part 1 of our series, this tutorial dives into setting up Amazon CloudWatch Container Insights. Included in the cluster configuration for the previous tutorial is the Amazon CloudWatch IAM policy attached to the IAM Role for Service Account (IRSA) and the OpenID Connect (OIDC) endpoint. For part one of this series, see Building an Amazon EKS Cluster Preconfigured to Run High Traffic Microservices. Alternatively, to set up an existing cluster with the components required for this tutorial, use the instructions in the verify prerequisites section of EKS official documentation.
In this tutorial, you will configure your Amazon EKS cluster, deploy containerized applications, and monitor the application's performance using Container Insights. Container Insights can handle lightweight applications like microservices as well as more complex systems like databases or user authentication systems, providing seamless monitoring.
Note: If you're within your inaugural 12-month phase, be advised that Amazon CloudWatch Container Insights falls outside the AWS free tier, hence usage could result in additional charges.
| About | |
|---|---|
| ✅ AWS experience | 200 - Intermediate |
| ⏱ Time to complete | 30 minutes |
| 🧩 Prerequisites | - AWS Account |
| 📢 Feedback | Any feedback, issues, or just a 👍 / 👎 ? |
| ⏰ Last Updated | 2023-10-02 |
Before you begin this tutorial, you need to:
- Install the latest version of kubectl. To check your version, run:
kubectl version --short. - Install the latest version of eksctl. To check your version, run:
eksctl info.
For CloudWatch Container Insights to collect, aggregate, and summarize metrics and logs from your containerized applications and microservices on Amazon Elastic Kubernetes Service (Amazon EKS), some setup steps need to be performed. Container Insights supports both Amazon EKS EC2 and Fargate. There are a few ways you can set up Container Insights on an Amazon EKS cluster: using the CloudWatch agent, a “quick start” setup, or through a manual setup approach. Below, you will find the steps required for the “quick start” method.
First, set and configure the following environment variables, ensuring consistency for
ClusterName and RegionName. In the following example, my-cluster is the name of your Amazon EKS cluster, and us-east-2 is the region where the logs are published. You should replace these values with your own values. It's advisable to specify the same region where your cluster is located to minimize AWS outbound data transfer costs. Additionally, FluentBitHttpPort is given a value of '2020' because this port is commonly used for monitoring purposes and allows for integration with existing tools, and FluentBitReadFromHead is given a value of 'Off' to ensure that the logs are read from the end, not the beginning, which can be essential for managing large log files and optimizing performance.Set the following environment variable to ensure that logs are read either from the head or the tail, but not both.
Next, set the following environment variable to control whether the HTTP server for Fluent Bit is enabled or disabled. In this command, the
FluentBitHttpServer for monitoring plugin metrics is on by default.Download and review the content of the Fluent Bit Daemonset by running the following command:
Use an IAM role for service accounts for the cluster, and attach the policy to this role. The command below creates an IAM Role and Service Account pair for fluent-bit:
Deploy the Fluent Bit Daemonset to the cluster by running the following command:
Validate that the agent is deployed by running the following command:
In this step, you will deploy a comprehensive containerized application environment within your Kubernetes cluster using a manifest file named
workload.yaml.- Create a Kubernetes manifest called workload.yaml and paste the following contents into it.
- Deploy the Kubernetes resources in
workload.yaml.
The expected output should look like this:
- Use the following command to check the status of the deployed Nginx containers and ensure that they are running:
The expected output should look like this:
- Use the following command to view the real-time logs of the "load" Pod, which is continuously making requests to the Nginx service. Use Ctrl+C to stop.
The expected output should look like this:
You can use CloudWatch Logs Insights Query to interactively search and analyze the container logs of the application in Amazon CloudWatch Logs. Fluent Bit sends logs from your containers in the cluster to CloudWatch Logs. In Step 1 above, we’ve set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs. Fluent Bit creates the log group below if it doesn't already exist:
/aws/containerinsights/Cluster_Name/application which contains all log files in /var/log/containers on each worker node in the cluster.- Open the CloudWatch console.
- In the navigation pane, choose Logs, and then choose Log groups.
- Click the log group
/aws/containerinsights/CLUSTER_NAME/application. Where CLUSTER_NAME is the actual name of your EKS cluster. - Under the log details (top-right), click View in Logs Insights.
- Delete the default query in the CloudWatch Log Insight Query Editor. Then, enter the following command and select Run query:
- Use the time interval selector to select a time period that you want to query. For example:
Monitoring the performance of your containerized application is essential for maintaining optimal functionality, identifying potential issues, and understanding the behavior of the system. In this step, you'll leverage AWS CloudWatch's Container Insights to gain detailed visibility into your container's performance.
In this section, you will learn how to access the Container Insights Dashboard Metrics within AWS CloudWatch. This dashboard provides a centralized view of your Amazon EKS and Kubernetes clusters' performance, offering real-time insights into various metrics such as CPU utilization, memory usage, and network activity. By following these steps, you can quickly navigate to the specific cluster and resources you wish to monitor, enabling you to keep a close eye on the health and performance of your containerized applications.
- Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
- In the left navigation pane, open the Insights dropdown menu, and then choose Container Insights.
- Under “Container Insights” (top), select Performance Monitoring from the dropdown menu.
- In the “EKS Clusters” dropdown field, select the name of your cluster.
- Use the additional dropdown menus to filter resources, such as “EKS Clusters” and “EKS Pods.” For example:
In this section, you will explore how to access a broader set of metrics specific to Amazon EKS and Kubernetes within AWS CloudWatch. These additional metrics, like pod_cpu_utilization_over_pod_limit, provide deeper insights into the performance and behavior of your containerized applications and the underlying infrastructure. Whether you are looking to analyze CPU utilization, Memory usage, or Network metrics, this process allows you to customize your view and focus on the aspects most relevant to your needs.
- Create a Kubernetes manifest called
geo-api.yamlwith the content below to deploy a simple backend application called geo-api with the following command:
The Deployment creates 1 Pod that has 1 container. The container is defined with a request for 0.125 CPU and 10MiB of memory. The container has a limit of 0.25 CPU and 12MiB of memory.
- Deploy the application using the command below:
- Create a load for the web server by running a container.
- Verify the Pods status:
The expected output should look like this:
The output shows that the geo-api Pod status is Running and sometimes getting OOMKilled.
Get a more detailed view of the Container status:
The output shows that the Container was killed because it is out of memory (OOM):
- Let’s view the Container Insights metrics of this pod:
- Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
- In the navigation pane, choose Metrics, and then choose All metrics.
- Select the ContainerInsights metric namespace. Select the ClusterName, Namespace, and PodName, in the search bar, copy and paste PodName="geo-api".
- You can view the percentage of CPU units being used by the pod relative to the pod limit and the percentage of memory that is being used by pods relative to the pod limit by selecting the metrics below:
pod_cpu_utilization_over_pod_limitpod_memory_utilization_over_pod_limit
The graph shows that the container in the pod has completely utilized its CPU and memory limit and you will need to specify enough resources to prevent the container in the pod from being terminated.
To avoid incurring future charges, you should delete the resources created during this tutorial. You can delete the container application and also delete the CloudWatch agent and Fluent bit for Container Insights with the following command:
By following this tutorial, you've successfully set up CloudWatch agent and Fluent Bit for Container Insights to monitor sample containerized workloads in an Amazon EKS cluster. With these instructions, you'll have a robust monitoring and logging solution to help you monitor the performance of application deployments in the cluster. If you want to explore more tutorials, check out Navigating Amazon EKS.
This article was co-authored with Ahmad Tariq.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.