Better Threat Detection with CloudWatch and Generative AI

In the previous parts of this series, I set up a base architecture for AWS Threat Detection and Management, deployed AWS WAF, AWS Shield Advanced, GuardDuty, Inspector, and AWS Config. I also briefly touched upon Amazon EventBridge. As we wrap up this series, I want to emphasize the importance of CloudWatch Logs and how Generative AI can be leveraged to enhance our threat detection and management capabilities.

CloudWatch is a fully managed service that enables you to centralize, store, and analyze log data from various AWS resources, applications, and services. In our case, we are sending logs from AWS WAF, which allows us to monitor and analyze traffic patterns, detect potential threats, and take appropriate actions.

While CloudWatch Logs provide invaluable data, manually analyzing and interpreting logs can be a daunting and time-consuming task, especially when dealing with large volumes of log data. This is where Generative AI comes into play. By leveraging the power of Large Language Models (LLMs), I can streamline the analysis process and gain valuable insights from our log data.

Here's the updated section of the blog post with the latest code:

To demonstrate the integration of CloudWatch Logs and Generative AI, I have developed a sample application using Streamlit and Python. Let's break down the code and understand how it works:

The application starts by importing the necessary libraries: streamlit for creating the user interface, boto3 for interacting with AWS services, json for handling JSON data, and botocore.exceptions for catching AWS-related exceptions. The title "CloudWatch Log Analyzer" is set for the Streamlit application, and a session client for CloudWatch Logs is created.

The application retrieves the list of available regions where the current AWS credentials have access to CloudWatch Logs. If there's an error retrieving the available regions, an error message is displayed. The default region is set to us-east-1 if available, or the first available region if us-east-1 is not accessible. A dropdown menu is created for selecting the region, with the default region preselected if available.

The application retrieves the list of log groups in the selected region, handling any errors that may occur due to insufficient permissions. If log groups are available, a dropdown menu is created for selecting the log group. Similarly, the application retrieves the list of log streams in the selected log group, handling any errors that may occur. If log streams are available, a dropdown menu is created for selecting the log stream.

The application retrieves the log events from the selected log stream, handling any errors that may occur. If log events are retrieved successfully, the log event messages are concatenated into a single string called prompt_data. A client for the Bedrock Runtime service is created to interact with the LLM.

The summarize_article function takes the prompt_data as input and constructs a prompt configuration object with instructions for the LLM. The log data is included in the prompt, and the LLM is tasked with analyzing the data and providing recommendations for AWS WAF rules. The invoke_model method is used to send the prompt configuration to the Bedrock Runtime API. The LLM's response is received and parsed to extract the summary.

Finally, the application displays the log data and the LLM's recommendations on the Streamlit page. If there are no regions available with CloudWatch Logs access, a warning message is displayed.

With this application, you can select a region, log group, and log stream, and the application will retrieve the log data, send it to the LLM for analysis, and display the LLM's recommendations on the Streamlit page.

To illustrate the application's capabilities, let's refer to the provided images:

Figure 1 shows the initial screen of the application, where the AWS region "us-east-1" has been selected.

In Figure 2 you can see the details of the log data being analyzed by the LLM.

Figure 3 is a continuation of Figure 2, that shows the LLM's summary of what happened. In this case, it indicates that the request was blocked due to the URI /.env being identified as a potential exploitation attempt by the ExploitablePaths_URIPATH rule in the AWS-AWSManagedRulesKnownBadInputsRuleSet rule group.

The final figure, Figure 4, displays the LLM's recommendations based on the log data analysis.

This application demonstrates how CloudWatch Logs can be effectively utilized in conjunction with Generative AI to enhance threat detection capabilities. By leveraging the LLM's natural language understanding and analysis capabilities, we can quickly interpret and gain valuable insights from log data. The LLM can identify potential threats, provide explanations, and recommend appropriate actions or rule configurations based on the log data.

Instead of manually sifting through vast amounts of log data, security analysts can rely on the LLM to streamline the process and highlight potential issues or areas of concern. This approach not only saves time but also reduces the risk of overlooking critical information hidden within the logs.

By integrating CloudWatch Logs and Generative AI, I can significantly improve our threat detection and management capabilities. The sample application showcases how this integration can be achieved using Streamlit, Python, and the Bedrock Runtime LLM. The ability to quickly analyze log data and receive actionable recommendations from the LLM empoIrs security teams to respond promptly and effectively to potential threats.

I encourage you to explore this application and experiment with it using your own log data. Feel free to modify and enhance the code to suit your specific requirements. Happy Labbing!