AWS CloudWatch Logging and Live Tail using AWS CLI!
Implement a CloudWatch live log tail using AWS CLI and validate it with a Lambda function integrated into an example GenAI summarization application!
Published Nov 27, 2024
In the previous articles, I explained how to set up live tail logging in AWS CloudWatch. The setup of log group live tail logging was completed using AWS Console. Subsequently, I also explained how to use a programmatic approach to start the log live tail using Python and Boto3.
In this article, I will demonstrate another way to get live tail for the selected cloud log group using AWS CLI. I will use AWS CLI to write a command that will publish the live tail based on the cloud watch group and filter pattern provided.
If you’re a Linux user, you’re likely familiar with the tail command for viewing live logs. CloudWatch’s live tail feature works similarly, allowing you to monitor logs in real time as your functions are executed.
This feature is incredibly helpful for various use cases, including debugging issues, monitoring deployments, command center oversight, identifying performance issues, and more.
If you are interested in prior articles on the cloud watch logs topics, Please refer to the links below:
Let's look at the architecture diagram!
Let's see the steps to create this solution using AWS CloudWatch live tail and Lambda logging.
Few points about CloudWatch Live Tail
- CloudWatch Live Tail provides options to refine and target specific log data:
- Use Highlight to flag events that match your specified terms (up to 5 terms).
- Use Filter to limit the events displayed in the session.
- You can select up to 10 log groups to include in the live tail session.
- AWS may update this limit so please refer to AWS documentation for the latest limit.
Note: CloudWatch Live Tail is charged on a per-minute basis during the session, making it a flexible, on-demand solution for troubleshooting and monitoring.
Review the CloudWatch Log Group
For this CloudWatch Logs live tail demonstration, I will use the log group /aws/lambda/MeetingNotesSummarizer. This log group is already in place for an existing Lambda function, and all logs generated by this function are stored here. This setup allows us to monitor and analyze the log data produced by this Lambda function in real time.
Review the Lambda Function Description
The Lambda function we’re using is pre-configured with logging capabilities. This function supports a Generative AI use case and includes guardrails to ensure data security and compliance. For example, if Personally Identifiable Information (PII) is detected in the data processed by this function, a guardrail log entry will be generated in the CloudWatch log group to flag the occurrence. This helps maintain data safety by providing insights into sensitive data handling during AI operations.
Review the AWS CLI Command
This command uses AWS CLI. Once executed, the command will monitors the AWS CloudWatch logs in real time (live tail) with the option to filter the logs.
We will also be able to highlight desired term in the log using the interactive mode.
You will need to ensure that you have AWS CLI 2.16.0 or higher installed.
You can run aws --version command to check the AWS CLI version installed.
Press 'c' to clean highlighted term
You can stop the program by pressing CTRL+C.
Run the AWS CLI command and start the live tail session!
Invoke the Lambda Function
To validate the policy configuration, Let's invoke a lambda function to write to the logs.
I am using a GenAI app created by me to invoke the lambda in the backend. This App uses multiple AWS services including API Gateway, Lambda, Bedrock and others.
Monitor the Live tail for highlighted and Filtered logs
When lambda function is invoked, logs will be created and live tail will show it as well!
Invoke the lambda function again and monitor the tail.
Notice that this term is now highlighted in different color:
Cleanup - Delete Log Group, End the live tail session
If you don't need the log group, make sure to delete it and end the live tail session.
Conclusion
In this article, we successfully demonstrated how to implement a live tail for CloudWatch logs using AWS CLI. If you watched the previous workshop on setting up a live tail using the AWS Console, you now know an alternative method to achieve the same functionality using AWS CLI.
Logging and monitoring are essential components of observability, and mastering different techniques can greatly enhance your skills in this area.
Thank you for reading!
Click here to watch the YouTube video for this solution:
𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶
𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦
𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵