Leveraging Amazon Q for Developer and QuickSight for Lightning-Fast CloudOps Data Visualization
Learn how to streamline cloud operations by combining Amazon Q for Developer and QuickSight. This solution shows how to quickly use these GenAI tools to build a serverless application for automated resource inventory management and monitoring.
Cloud platform engineers frequently need to develop lightweight solutions for automating CloudOps tasks, with resource inventory management being a critical component that helps infrastructure teams gain rapid insights into their Cloud environment. Typically, Cloud infrastructure teams face daily challenges such as understanding the distribution of EC2 instance types and operating system, or quickly identifying which specific EC2 instance is associated with a particular network address.
Traditionally, organizations have invested significant time and resources in researching and developing custom solutions to build automated solutions. However, with the advent of Amazon Q for Developer, Cloud platform engineers can now rapidly design and implement comprehensive solutions in just a few hours.
Scenario
As organizations run hundreds or thousands of EC2 and RDS instances in their AWS environment, and application development teams continuously deploy new resources, the Cloud platform team must maintain vigilant monitoring of newly deployed resources. This monitoring is crucial for quickly identifying the source instances when suspicious network activity is detected.
The Cloud platform team decides to build a serverless solution that leverages multiple AWS services, including AWS Config Advanced queries, EventBridge, Lambda, Athena, S3, and QuickSight. This integrated solution starts with a daily triggered EventBridge rule that trigger a Lambda function to run against a pre-defined AWS Config advanced query, which returns all running EC2 instance inventory information. The returned data will be parsed and stored as a csv file in an S3 bucket. The Cloud platform team can then create an Athena query, from which Amazon QuickSight is connected to allow them to ask any natural human questions about their Cloud inventory data.
Walkthrough
How I use Amazon Q for Developer to Expedite the Solution Development
Use ‘/dev’ prefix to prompt Amazon Q for creating project structure We'll enhance our development speed by implementing AWS SAM (Serverless Application Model). This framework specializes in Infrastructure as Code (IaC) for serverless applications, making resource development and deployment more efficient. Though AWS SAM offers command-line optlike 'sam init' for creating project templates, I'll be using Amazon Q's “/dev” command to generate our project structure.
"I would like to use AWS SAM to deploy a serverless solution called "chat-with-your-cloud-resources", please create the project folder structure for me. Here's the requirement: 1. There an EventBridge rule triggering a Lambda Function (written in python 3.12). 2. The Lambda Function would run an AWS Config Advanced query and export the returned result of the query into an S3 bucket called "ChatOp-demo" as a csv file. 3. Make sure the Lambda function has the necessary permissions to run the AWS Config Advanced query and write the exported csv file into the S3 bucket."
Dev Agent response
Using Amazon Q for Developer, I was able to automatically generate a standardized folder structure for the project. The generated template.yaml file within the project also includes the majority of essential components required for deployment.
Generated code
Use Inline Chat feature to refine the IaC code I noticed Amazon Q's assumption about an existing S3 bucket, so I utilized the 'Inline Chat' feature to request Amazon Q to generate code for creating a new S3 bucket. I also needed to establish a Lambda execution role that would enable the Lambda function to perform daily AWS Config advanced queries and store the results in the S3 bucket. Amazon Q displayed the proposed code modifications for review, and I approved them by clicking the 'Accept' button.
I can optionally perform any ad-hoc changes to refine the code. E.g., I am prompting “Update teh Eventbridge rule with a cronjob expression that triggers the Lambda function at 9 am HKT timezone each day." to refine the EventBridge rule.
For the next step, I'll leverage Amazon Q for Developer to assist in writing the Lambda function code. A key requirement is customizing the data output format, and Amazon Q will be particularly helpful in achieving this. I'll utilize Q Chat to develop the Lambda function's core logic. To begin, I submitted a prompt to Amazon Q requesting help with implementing the AWS Config Query functionality. Prompt:
"I need to run an advanced AWS Config query for a Config Aggregator called "my-aggregator". The query should return the fields as the following fields. "Resource ID, Account ID, Instance Type, Core Count, Threads Per Core, Architecture, Platform Details Volume ID, Private IP Public IP, Subnet ID, VPC ID, Creation Time, Availability Zone, Region, Tags""
Next, I prompted Amazon Q with the AWS Config SQL query's CSV output as a reference sample data to write code for parsing the data. This allowed Amazon Q to help me extract the key value associated with the 'Name' tag from the EC2 instance tags. Since EC2 instances don't have a built-in name field, I'm using the value of the 'Name' tag to identify each instance. Prompt:
"Here's the sample data from the exported query result. Please help: 1. write AWS Config query result into a csv file and store it in the S3 bucket. Note, you need to handle pagination. 2. Extract the value of the 'Name' key in the "Tags" field and store it in a new column called "Instance Name”."
Once I am fine with all the tuning on the code, I can ask Amazon Q for Developer to guide me to deploy the stack. Following the command lines, you can quickly complete the deployment of the serverless stack.
How I use Amazon Q for Quicksight to Quickly Understand My Cloud Environment
With the AWS Config Advanced query results now exported as CSV data, I can proceed to create an external table using Amazon Athena. This will enable me to visualize the data through Amazon QuickSight for better analysis and insights.
Create a new dataset in Amazon QuickSight that connects to the previously created Athena table. (Remember to configure proper permissions for QuickSight to access the S3 bucket containing the CSV file)
Here are some common questions that Cloud operations teams frequently encounter. Amazon Q for QuickSight offers a powerful way to explore your data through natural language queries, allowing you to gain deeper insights about your cloud environment by simply asking questions.
Check all EC2 instances details
Quickly locate EC2 instance with certain IP address
Count Number of EC2 instances in a Certain AWS Account
Show Instance Type Distribution
Conclusion
This solution demonstrates how the combination of Amazon Q for Developer and Amazon Q for QuickSight can significantly transform Cloud operations analytics. By leveraging Amazon Q for Developer, Cloud platform engineers can rapidly develop serverless solutions for Cloud operation, reducing development time from weeks to hours. The automated solution, built using AWS SAM and integrating services like AWS Config, EventBridge, Lambda, Athena, and S3, provides a robust foundation for continuous cloud resource inventory monitoring.
Furthermore, Amazon Q for QuickSight revolutionizes how teams interact with their Cloud infrastructure data. Through natural language queries, operations teams can instantly access critical information about their EC2 instances, IP addresses, and resource distributions, eliminating the need for complex manual queries or custom reporting tools.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.