AWS CloudWatch Logging and Live Tail!

In the previous articles, I explained the use of CloudWatch and CloudTrail for logging, monitoring, alarming, and auditing. I also covered how to create aggregated logs in CloudWatch using a custom log group.

If you missed it, be sure to check out the articles linked below before continuing.

Link: AWS Logging, Monitoring & Auditing with AWS CloudWatch and CloudTrail!

Link: AWS Lambda Log Aggregation Using CloudWatch Custom Log Group & Logs Insights!

In this article, I’ll introduce another useful feature: CloudWatch Live Tail, which lets you view live logs as they’re generated!

If you’re a Linux user, you’re likely familiar with the tail command for viewing live logs. CloudWatch’s live tail feature works similarly, allowing you to monitor logs in real time as your functions are executed.

In this example, I’ll demonstrate how to set up live tail logging in AWS CloudWatch. We’ll invoke a Lambda function and analyze its logs live as they’re published to the log stream.

This feature is incredibly helpful for various use cases, including debugging issues, monitoring deployments, command center oversight, identifying performance issues, and more.

Let's look at the architecture diagram!

Let's see the steps to create this solution using AWS CloudWatch live tail and Lambda logging.

CloudWatch Live Tail provides options to refine and target specific log data:

Note: CloudWatch Live Tail is charged on a per-minute basis during the session, making it a flexible, on-demand solution for troubleshooting and monitoring.

For this CloudWatch Logs live tail demonstration, I will use the log group /aws/lambda/MeetingNotesSummarizer. This log group is already in place for an existing Lambda function, and all logs generated by this function are stored here. This setup allows us to monitor and analyze the log data produced by this Lambda function in real time.

The Lambda function we’re using is pre-configured with logging capabilities. This function supports a Generative AI use case and includes guardrails to ensure data security and compliance. For example, if Personally Identifiable Information (PII) is detected in the data processed by this function, a guardrail log entry will be generated in the CloudWatch log group to flag the occurrence. This helps maintain data safety by providing insights into sensitive data handling during AI operations.

Let's configure a live tail. I will use the GUARDRAIL_INTERVENED as the highlighted term.

I will also filter on 'GUARDRAIL_INTERVENED' to get the live tail whenever guardrail is applied for the Generative AI use case in this function.

Apply the filter and start the live tail session!

To validate the policy configuration, Let's invoke a lambda function to write to the logs.

I am using a GenAI app created by me to invoke the lambda in the backend. This App uses multiple AWS services including API Gateway, Lambda, Bedrock and others.

When lambda function is invoked, logs will be created and live tail will show it as well!

If you don't need the log group, make sure to delete it and end the live tail session.

In this article, I have demonstrated how to implement a live tail for CloudWatch logs. Logging and monitoring are essential components of observability, and mastering different techniques can greatly enhance your skills in this area.

With this setup, you are now equipped to monitor live tails from CloudWatch log groups effectively!

Thank you for reading!

Click here to watch the YouTube video for this solution:

𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶

𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦

𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵