AWS CloudWatch Logging and Live Tail using Python/Boto3 SDK!
Implement a CloudWatch live log tail using Python/Boto3 and validate it with a Lambda function integrated into an example GenAI summarization application!
Published Nov 23, 2024
In the previous article, I explained how to set up live tail logging in AWS CloudWatch. The setup of log group live tail logging was completed using AWS Console.
In this article, I will demonstrate another way to get live tail for the selected cloud log group using a programmatic approach as oppose to using the AWS Console. I will use Python and Boto3 to write a command line python program that will publish the live tail based on the cloud watch log group and filter pattern provided.
If you’re a Linux user, you’re likely familiar with the tail command for viewing live logs. CloudWatch’s live tail feature works similarly, allowing you to monitor logs in real time as your functions are executed.
This feature is incredibly helpful for various use cases, including debugging issues, monitoring deployments, command center oversight, identifying performance issues, and more.
If you are interested in prior articles on the cloud watch logs topics, Please refer to the links below:
Let's look at the architecture diagram!
Let's see the steps to create this solution using AWS CloudWatch live tail and Lambda logging.
Few points about CloudWatch Live Tail
- CloudWatch Live Tail provides options to refine and target specific log data:
- Use Highlight to flag events that match your specified terms (up to 5 terms).
- Use Filter to limit the events displayed in the session.
- You can select up to 10 log groups to include in the live tail session.
- AWS may update this limit so please refer to AWS documentation for the latest limit.
Note: CloudWatch Live Tail is charged on a per-minute basis during the session, making it a flexible, on-demand solution for troubleshooting and monitoring.
Review the CloudWatch Log Group
For this CloudWatch Logs live tail demonstration, I will use the log group /aws/lambda/MeetingNotesSummarizer. This log group is already in place for an existing Lambda function, and all logs generated by this function are stored here. This setup allows us to monitor and analyze the log data produced by this Lambda function in real time.
Review the Lambda Function Description
The Lambda function we’re using is pre-configured with logging capabilities. This function supports a Generative AI use case and includes guardrails to ensure data security and compliance. For example, if Personally Identifiable Information (PII) is detected in the data processed by this function, a guardrail log entry will be generated in the CloudWatch log group to flag the occurrence. This helps maintain data safety by providing insights into sensitive data handling during AI operations.
Review the Python Program
This python program is for a command line utility that monitors the AWS CloudWatch logs in real time (live tail) with the option to filter the logs. Here’s a simple breakdown of its main parts:
The script uses a config.json file to get the following settings:
- log_group: The CloudWatch log group you want to monitor.
- filter_pattern: The text used to filter log messages.
The functions in this program are as below:
- read_config(): Reads the settings from the config.json file.
- get_log_events(): Fetches log entries from a specific log group and log stream.
- get_log_streams(): Retrieves log streams and sorts them by the most recent ones.
Example function:
In order to run this program successfully, you need to ensure that your environment is configured. Ensure that you have AWS credential configured, boto3 library is installed and Config.JSON has the name of the log group and filter pattern.
You can stop the program by pressing CTRL+C.
Review Config.JSON
Run the Python Program and start the live tail session!
Invoke the Lambda Function
To validate the policy configuration, Let's invoke a lambda function to write to the logs.
I am using a GenAI app created by me to invoke the lambda in the backend. This App uses multiple AWS services including API Gateway, Lambda, Bedrock and others.
Monitor the Live tail for Filtered logs
When lambda function is invoked, logs will be created and live tail will show it as well!
Cleanup - Delete Log Group, End the live tail session
If you don't need the log group, make sure to delete it and end the live tail session.
Conclusion
In this article, I have demonstrated how to implement a live tail for CloudWatch logs using Python and Boto3. If you have reviewed previous articles on setting up a live tail using the AWS Console, you now know an alternative method to achieve the same functionality using Python. In the next article, I will explain another approach to setting up a live tail using the AWS CLI.
Logging and monitoring are essential components of observability, and mastering different techniques can greatly enhance your skills in this area.
Thank you for reading!
Click here to watch the YouTube video for this solution:
https://www.youtube.com/watch?v=JkW7sg4AhLI
𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶
𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦
𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵