Tracing Amazon Bedrock Agents
Sending Bedrock trace data to Amazon X-Ray
Randy D
Amazon Employee
Published Feb 20, 2025
In an agentic workflow, an LLM interacts with other systems to tackle a problem. The sequence of steps could be long and involve back and forth as the LLM plans an approach, invokes a tool, works with another agent, and so on. The diagram below shows the type of interactions that can happen in an autonomous agent workflow, where the LLM does the planning and then accesses other tools and systems to help it achieve a goal. Notice that the LLM might operate in a loop, taking steps, reflecting on the results, then deciding on a next step.
In order to understand what's happening in an agentic system, we need tracing data. We want to capture the flow of information between the different parts of the system. That way, we can understand where time is being spent (latency) and how the agent is 'thinking'. We can also see where things are breaking down if the agent isn't working as expected.
Bedrock agents can provide tracing data as part of the agent response. In this article, I'll describe a crude approach to capturing that data and relaying it to AWS X-Ray, a tracing service that is part of the AWS observability suite. By relaying the data to X-Ray, we make it more visible outside of Bedrock, and we can also pull in tracing data from other systems that we use in an agentic workflow. X-Ray natively supports several AWS services.
In order to get started, we'll follow Module 1 of this workshop, up through Exercise 7. That gets us to the point of having a multi-agent Bedrock agent example, where a supervisor agent can call another agent to help respond to a task. Then we'll borrow some code from this example notebook so we can invoke the agent programmatically.
This code block invokes the agent:
Next, we'll capture the trace events from the output.
There are 20 trace events captured in this example. Here's an abbreviated version of one of those events. This event shows the first input to the supervisor agent's LLM.
We want to convert these trace events into X-Ray segments. Each segment has, at a minimum, the name of the service, a unique segment ID, a trace ID that groups together related segments, a start and end time, and optional data. We'll use the optional annotations field to capture additional detail, as well as the
parent_id field to indicate the order of the segments in the overall execution.I made some assumptions about how to group Bedrock trace events into logical X-Ray segments, and then gave my coding assistant some instructions to generate Python code to do the translation. Here's the relevant prompt.
The resulting method converts the list of Bedrock traces to a list of X-Ray segments. Then we can just send them to X-Ray.
Here's the X-Ray trace flow from our multi-agent example. Each node shows an agent, either the 'Supervisor' or the 'Worker', performing a single unique task, like a model invocation, a guardrail check, or invoking another agent.
The segments view shows a more traditional "call chain" flow.
We captured a lot of the additional detail in the segment annotations. For example, the caller chain shows that this segment is performed by an agent collaborating with a supervisor agent.
This is a very rough first approach. Be aware of these limitations:
- At the time of writing, the traces do not contain the start and end times, so we just use the current timestamp for the start of the first segment, and set the end to the start plus 5 seconds. Each subsequent segment should have a start time equal to the previous segment's end time plus 1 second. We can fix this once Bedrock provides the start and end times.
- We are not yet handling all the possible types of traces.
- We didn't enable tracing in the Lambda functions called as tools in the action groups.
- Using the higher-level X-Ray SDK would let us capture more segments from other AWS services as part of a single logical trace.
Some of my colleagues are working on an approach to capture Bedrock traces more systematically using OpenTelemetry. I'll provide a follow-on post when we have more to share.
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.