AWS Lambda Log Aggregation Using CloudWatch Custom Log Group & Logs Insights!
Implement a custom log group using CloudWatch and configure multiple lambda functions to publish logs to this log group. Use Logs insights to analyze the logs.
Published Nov 14, 2024
In the previous article, I explained the use of CloudWatch and CloudTrail for logging, monitoring, alarming, and auditing.
If you missed it, be sure to check out the article linked below before continuing.
In this article, I’ll discuss how to aggregate logs for multiple lambda functions in one CloudWatch custom log group.
By default, Lambda functions create individual log groups. This works fine for isolated functions, but if you have multiple Lambda functions supporting a feature, it may be desirable to have a single log group where all these functions log their output, simplifying monitoring and troubleshooting.
In this article, we’ll walk through how to set up and configure two AWS Lambda functions (HelloWorld1 and HelloWorld2) to publish their logs to a custom CloudWatch log group, enabling centralized log aggregation.
We’ll then use Logs Insights to review logs in one place, making it easier to analyze function behaviors and events together.
Let's look at the architecture diagram!
Let's see the steps to create this solution using AWS Lambda and CloudWatch custom log group.
Create a Custom CloudWatch Log Group
First, I will create a custom CloudWatch log group. This is the log group that will be consumed by the two sample Lambda functions.
- Navigate to CloudWatch Console: In the AWS Management Console, open CloudWatch.
- Create a Log Group:
- In the Log groups section, choose Create log group.
- Name the log group, e.g., /aws/lambda/customLogGroupGB.
- Set Retention: Configure the retention time to 5 days to automatically delete older logs, ensuring manageable log storage.
- Save: Confirm the settings to create the log group.
Create the First Lambda Function (HelloWorld1)
Next, I will create the first lambda function and will name it as HelloWorld1. I am going to use python as the runtime but you can select any runtime of your choice from the supported runtime options.
- Create Lambda Function:
- Go to the Lambda console, select Create function, and name it HelloWorld1.
- Select Runtime as Python.
- Define Function Code:
- Add code that generates a log statement example below:
Here is the sample code for the lambda function.
Configure Logging to Custom Log Group
Since I want this lambda function to use the custom log group for logging, I will update the function configuration.
- Go to Monitoring and operations tools > Edit in the function configuration.
- In the Log groups field, set the custom log group name /aws/lambda/customLogGroupGB
- Save and Deploy the function.
Test HelloWorld1 and Validate Log Publication
Let's invoke the function and validate that logs are getting published to the custom log group.
- Invoke the Function:
- Test the function using the Test button.
- Verify Logs in CloudWatch:
- Go to the CloudWatch console and navigate to /aws/lambda/customLogGroupGB
- Confirm that the log “Hello from HelloWorld1!” appears.
Create the Second Lambda Function (HelloWorld2)
I will create another lambda function now and will repeat the process to use the same custom log group.
- Follow the same process as above, naming this function HelloWorld2.
- Code and Log Setup
- Add similar code to HelloWorld1, modifying the log message to identify this function:
HelloWorld2 Lambda Function
- Configure HelloWorld2 to publish logs to the same custom log group /aws/lambda/customLogGroupGB
Custom Log Group - Test HelloWorld2 and Confirm Log Aggregation
- Invoke HelloWorld2 and validate logs.
- Check the custom log group in CloudWatch to confirm logs from both HelloWorld1 and HelloWorld2 are appearing.
- Logs Insights to analyze the logs
- Now that both functions are configured to log into the same group, use CloudWatch Logs Insights to analyze the logs effectively.
- Open Logs Insights:
- In the CloudWatch console, select Logs Insights.
- Select Log Group:
- Choose /aws/lambda/customLogGroupGB
- Run Queries:
- Example query to filter logs by function:
Logs Insights 
Aggregated Logs
View Results
Observe timestamps and log messages from each function, which can aid in tracking events across functions.
Conclusion
In this article, I’ve demonstrated how to aggregate logs from multiple Lambda functions into a single CloudWatch custom log group. By using a shared log group and querying capabilities in Logs Insights, you can simplify log management and gain insights across functions in a streamlined way. This approach is particularly valuable when managing micro-services or event-driven architectures, as it allows centralized access and analysis for better monitoring and troubleshooting.
With this setup, you’re ready to monitor multiple Lambda functions efficiently in AWS!
Thank you for reading!
Click here to watch the YouTube video for this solution:
𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶
𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦
𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵