AWS Logging, Monitoring & Auditing with AWS CloudWatch and CloudTrail!
CloudWatch and CloudTrail provide the tools necessary to implement the observability, empowering you to monitor, troubleshoot, and audit your applications in the cloud. Let’s explore each of these services and some of the valuable features they offer.
Published Nov 7, 2024
In this article, I’ll discuss two essential AWS services that should be integral to your cloud migration and operational strategy: CloudWatch and CloudTrail.
Observability, a key component of operational excellence as defined in the AWS Well-Architected Framework, is crucial for supporting and managing cloud operations effectively.
CloudWatch and CloudTrail provide the tools necessary to implement this observability, empowering you to monitor, troubleshoot, and audit your applications in the cloud.
Let’s explore each of these services and some of the valuable features they offer.
In AWS, CloudWatch and CloudTrail are often discussed together or even confused with one another, but each service has a distinct role in managing and securing cloud resources. Contrary to the idea of "CloudWatch vs. CloudTrail," it's more accurate to think of them as complementary tools that help you achieve observability and compliance across your AWS environment.
In simplest terms:
- CloudWatch is a logging, monitoring, and alerting service.
- CloudTrail is an auditing service that records events within your AWS accounts.
Understanding the unique functions of each can help you design a robust, well-monitored, and compliant cloud infrastructure.
Let's look at the architecture diagram!
AWS CloudWatch is designed to provide logging, monitoring, and alerting capabilities. It is instrumental in ensuring the observability of your AWS resources, applications, and services.
- Primary Purpose: Logging, Monitoring and Alarming
- Common Use Cases:
- Setting up alarms for critical metrics, like CPU utilization, memory, or API latency.
- Visualizing metrics over time using CloudWatch dashboards.
- Aggregating logs for troubleshooting and analysis using CloudWatch Logs.
Core Features:
- Log Collection and Management: CloudWatch Log Groups store and organize logs from various AWS services and custom application logs. For example, you can create a log group to track application logs or system-level metrics from EC2 instances.
- Alarms and Notifications: CloudWatch can trigger alarms based on metric thresholds, enabling you to monitor key performance indicators (KPIs) and take automated actions, such as scaling an application or sending an alert if a service experiences issues.
- CloudWatch Logs Insights: This interactive query tool helps you analyze log data to identify trends and diagnose problems, offering queries by source, region, IP, and more for in-depth visibility.
Setting Up and Using CloudWatch Log Groups
Now we understand the use cases for CloudWatch. We will use an example to create a CloudWatch Log groups, use it and then delete it so that we don't incur cost by leaving the log group forever in our account.
- From AWS Console, navigate to the CloudWatch service and select Log Groups
- Select Create Log Group and then specify a name for the Log group, example: /aws/lambda/customLogGroupGB
- Configure retention setting to specify how long the logs should be stored.
- You have option from 1 day to Never expire (many options in days, weeks, months and years!)
- Update Lambda function config to use this custom log group
- You don't need to use custom log group as Lambda will create a default log group /aws/lambda/functionName format.
- Review the log
- Delete the log group ( to avoid incurring cost )
Example of CloudWatch Log Group
AWS CloudTrail is an auditing service designed to record events within an AWS account. It automatically logs management events for AWS resources, tracking any actions taken via the AWS Management Console, SDKs, or CLI. These records provide insight into "who did what, where, and when" within an AWS environment, making CloudTrail a critical component for security , compliance and operational transparency.
- Primary Purpose: Auditing and compliance
- Common Use Cases:
- Capturing account activity for compliance and security audits.
- Monitoring user actions to detect unusual activity.
- Storing logs in Amazon S3 for long-term retention or analysis with other tools.
Core Features:
- Logging of Management Events and Data Events: CloudTrail distinguishes between management events (such as creating or modifying resources) and data events (specific actions taken on resources, like accessing an S3 bucket).
- Example: Creating a Lambda function is logged as a management event, while invoking that function is considered a data event.
- Default Logging for Management Events: By default, CloudTrail retains management events for the past 90 days.
- You can also create a trail to log these events to an S3 bucket for long-term storage and set up alerts on specific events.
Example of CloudTrail Events
Since management event are active by default, you don't need to take any specific steps to configure it.
Here is an example of how management event in CloudTrail will show up:
In summary, both CloudWatch and CloudTrail are indispensable AWS services that contribute to the observability and security pillars of the AWS Well-Architected Framework. While CloudWatch is essential for real-time monitoring and alerting, CloudTrail provides a comprehensive log of user activity and API calls, making it invaluable for auditing and compliance.
For a robust cloud solution, leverage both CloudWatch and CloudTrail to gain comprehensive insights into your AWS environment’s performance, security, and compliance. Logging, monitoring, and auditing should be foundational components in any cloud migration or architecture to ensure resilience, security, and regulatory compliance.
By leveraging CloudWatch and CloudTrail, you build a foundation for observability and governance that aligns with AWS best practices and keeps your cloud environment secure and compliant.
Thank you for reading!
Click here to watch the YouTube video for this article:
https://www.youtube.com/watch?v=sCaPrOCcz8k
𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶
𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦
𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵