Centralizing Generative AI operations: distributed logging using Amazon Bedrock and LangChain

In the world of cloud computing, some developers and architects take segmentation to the extreme by creating separate AWS accounts for each customer or organization. This approach isn’t just about organization — it’s a strategic move to enhance security, simplify compliance, and streamline cost allocation. By isolating workloads at the account level, companies can enforce strict security boundaries, prevent cross-customer data leaks, and comply with industry regulations like HIPAA or GDPR with minimal risk. It also enables precise cost tracking, allowing SaaS providers to allocate expenses per customer effortlessly.

So, why do multi-tenant architectures exist? Should we be applying multi-account strategies in every privacy-conscious product? That may not be a great idea. The price to pay for the enhanced security and compliance is added complexity in managing operations. Think of, for example, enabling Amazon Bedrock to allow applications running on separate accounts to run generative AI-powered applications. The software provider would have the responsibility to request and manage Model Access, and to manage throughput quotas to maintain the application operational. This process could be annoying to do for a few customers; and a total nightmare for thousands of customers.

There is, however, a middle ground. We can implement a multi-account deployment for most components of the infrastructure while centralizing only the generative AI components in one account. This blog introduces an architecture that centralizes Amazon Bedrock operations in one account while preserving total segmentation of logs, and ensuring that none of the data/logs from the customer will ever get logged in an environment outside of their account, to preserve the privacy principles that motivated the use of a multi-account deployment in the first place.

Before we discuss the solution at hand, we will go over two other options of multi-account invocation logging, depending on the desired degree of privacy needed.

By default, Amazon Bedrock does not log invocations in AWS CloudWatch. A user can, however, enable invocation logging on the AWS console. This can be done as simply as switching a toggle on the AWS console. By turning Bedrock logging on, all invocations metrics will be logged in CloudWatch, S3 (or both) in the Operations Account. This method is great if due to the workload, we are okay with all of the logs, from multiple customer accounts being stored in the operations account. There is a caveat to this method - the customer accounts invoking Bedrock have no visibility into the logs. We also need to consider the case where the customer does not want the operations account to be able to view the logs either, and a greater degree of privacy is needed. The straightforward process to enable Amazon Bedrock logging on the the AWS console is shown below.

In this second solution we will discuss how CloudWatch can be used for cross-account logging. This method builds on top of the first method. Here just like the first solution, we will first save all of the Bedrock invocation logs to the Operations Account. After the logs are neatly in the Operations Account, we will use tags and some custom logic to filter different customer invocations and send them to the CloudWatch logs of their specific account. After that is done, the logs will be deleted from the Operations Account. This solution allows for each customer to have their logs in their own accounts, while not having any persisting logs in the Operations Account. This solution is acceptable, if what we are concerned about is the end state of the logs in their respective accounts.

Here is a high level overview of how something like this would look like:

The Lambda functions here will contain the customer logic. If you want a deeper dive into this process, here is an AWS blog that goes into the intricacies.

Both methods above offer varying degrees of invocation logging privacy, which can be appropriate sometimes. A major factor here is that the logs still having to pass through the Operations account, which in scenarios where a great degree of privacy is needed, we would not want.

Below we will get into the details of how we can log Bedrock invocations in the Customer Account only, while bypassing the need for any logs or visibility in the Operations Account. This will be done by using Langchain. In specific Langchain Callbacks, which is a utility that allows you to hook into multiple stages of an LLM application for the purpose of for logging, monitoring, streaming, and other tasks. We will use this to hook in during the invocation from the Customer Account to log the metadata into the Customer Account only. With this architecture, even though the Operations Account is creating the Bedrock Client, and Bedrock is being used from this account, it has no way to capture metadata of the Customer’s Bedrock invocations as long as the automatic logging feature is kept turned off in the Operations Account.

We will now dive into the architecture of what we are trying to accomplish.

Lets break down the Diagram above:

1. First we have an IAM Role in the Customer Account, this role will assume the role in the Source Account allowing it to access the Amazon Bedrock service in the source account. When assuming the role from the Customer Account, the Source Account will generate temporary credentials for the Customer Account Role to use. We denote Customer “N” in the diagram to show that this may scale up to as many Customers required for your workflow.

Here is some example code for cross-account role assumption:

In the code above we are creating an “assume_role” function to generate some temporary credentials, we are then declaring the role from the Operations Account with “assumed_role_arn” and passing it into the function. The “SagemakerSession“ is being passed as the session because this example code was tested in Amazon Sagemaker. This is not required for other IDE’s you write/test this code in.

2. The IAM Role in the source account will create an Amazon Bedrock client that will allow for the invocation of Large Language Models (LLM’s), this Amazon Bedrock client will ultimately be used for invocations from the Customer Account.

Here is some example code for creating the Bedrock client:

Depending on how much flexibility you want to give the customer, the “model_id“ can be hardcoded to allow the Customer Account to use a specific model, or you may give the Customer Account the choice of model by allowing them to select the ”model_id”. Make sure to take note that the Model ID chosen, must pertain to a model that is accessible by the Operations Account.

3. The Amazon Bedrock Client will be invoked by the Customer Account IAM Role allowing a prompt to get sent to the Bedrock service, and a response to come back to the Customer Account. The Source Account quota is the one that will be drawn upon during this call.

In this piece of code we are using Langchain HumanMessage to help construct our message. the prompt, and model id are being passed here. We are also passing in a “custom_handler“ which takes a user_id and application_id. This "CustomCallbackHandler" is where the Langchain Callbacks hooks into the invocation step, we will explore it in the next step below.

4. Once the response comes back to the Customer Account, Cloudwatch Logs can be written for any data needed to be captured from the Bedrock invocation (tokens in/tokens out, total latency, prompts/responses etc.) with Langchain Callbacks that are hooked into the invocation.

Here is the custom logic within the "CustomCallbackHandler", as you can see we are using this function to write relevant information to Cloudwatch, in particular for this example: Application Id, Input tokens, Output tokens, and Invoked model. You are not limited to this, you may even save the prompt/response pairs if you so choose. Take note the Cloudwatch Log Group with the name “/aws/bedrock/langchain/logs” and a Log Stream named after the user id (1234) need to be created for successful logging. These are the names for this example, and can change based on yours.

The architecture presented in this blog demonstrates that organizations don't have to choose between robust security isolation and operational efficiency when implementing generative AI capabilities. By centralizing Amazon Bedrock operations while maintaining strict data boundaries using tools like LangChain, we achieve the best of both worlds: the simplified management of a centralized deployment with the security benefits of multi-account isolation.

While multi-account strategies remain a powerful tool for security and compliance, strategic centralization of specific components like Amazon Bedrock can significantly reduce operational complexity without compromising the core benefits of account isolation. As organizations continue to adopt generative AI capabilities, this pattern provides a blueprint for balancing security, compliance, and operational efficiency. Amazon Bedrock is flexible enough to get varying degrees of privacy as required by the workflow.

Remember that this architecture can be adapted to similar scenarios where centralized service management makes sense, while still maintaining strict data boundaries. The key is identifying which components truly benefit from centralization versus those that require strict isolation, or need a delicate combination of both.