Speak Up, AI: Voicing Your Agents with Amazon Connect, Lex, and Bedrock

Ever felt like your customer service system was playing a game of high-stakes telephone?

"Your call is important to us" on repeat isn't exactly the pinnacle of efficiency. Well, get ready to turn your silent digital helpers into smooth-talking problem solvers that'll make even the most sophisticated AI assistants sound like they're still learning their ABCs!

In this article, I'll show you how to create a customer service system so intelligent and versatile, it'll make your Swiss Army knife look like a butter spreader. We're talking about a system that can book your flight, check the weather at your destination, and give you jet lag advice faster than you can say "Is this seat supposed to recline this far?"

Here's what I'll cover in this article:

The architecture of our voice-enabled AI system and why it's cooler than a polar bear's toenails
How to integrate Amazon Connect, Lex, and Bedrock for a seamless voice experience
Leveraging the Multi-Agent Orchestrator for context-aware conversations
Creating specialized agents that switch tasks faster than your GPS recalculates after a wrong turn
Putting it all together for a customer service experience so smooth, your customers will think they're talking to JARVIS's more capable cousin

Let's dive into the system architecture that makes this AI magic happen!

System Architecture: Where the Magic Happens

Picture this: a customer service system so seamless, your customers won't know if they're talking to an AI or Tony Stark himself. Here's how our sophisticated solution works:The Grand Entrance: It all starts with a customer reaching out to your Amazon Connect contact center. They could be calling in or sliding into your DMs via web chat.

Lex Bot: The Smooth Talker: Enter our Lex bot, the linguistic acrobat of our system. Its job? To turn your customer's words into text faster than you can say "Can you repeat that?"
The Orchestrator: The Mastermind: At the heart of our system lies the Multi-Agent Orchestrator. Think of it as the air traffic controller for your AI agents, making sure every query lands in the right place.
Specialized Agents: The Dream Team: We've got a squad of AI superheroes ready to tackle any question:

Airline Bot: For when your customers need to fly the friendly skies
Weather Agent: Because nobody wants to pack flip-flops for a blizzard
Health Agent: Dispensing advice on jet lag, travel vaccines, and more
Tech Agent: For all those "Is it plugged in?" moments

Context is King: Our Orchestrator is like that friend who remembers every detail of your life. It keeps track of the full conversation history, ensuring your AI never suffers from short-term memory loss.
Seamless Experience: To your customer, it'll feel like one smooth conversation, even if behind the scenes we're playing AI hot potato.

This system can handle complex, multi-topic interactions so smoothly, it'll make your old IVR system look like it's still using dial-up internet.

Demo: The Magic in Action

Behind The Scene

For this example, here is how I have set up our Multi Agent Orchestrator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
import {
  MultiAgentOrchestrator,
  BedrockLLMAgent,
  DynamoDbChatStorage,
  LexBotAgent,
} from "multi-agent-orchestrator";

const storage = new DynamoDbChatStorage(
  process.env.HISTORY_TABLE_NAME!,
  process.env.AWS_REGION!,
  process.env.HISTORY_TABLE_TTL_KEY_NAME,
  Number(process.env.HISTORY_TABLE_TTL_DURATION),
);

const orchestrator = new MultiAgentOrchestrator({
  storage: storage,
  config: {
    LOG_AGENT_CHAT: true,
    LOG_CLASSIFIER_CHAT: true,
    LOG_CLASSIFIER_RAW_OUTPUT: true,
    LOG_CLASSIFIER_OUTPUT: true,
    LOG_EXECUTION_TIMES: true,
  }
});

const techAgent = new BedrockLLMAgent({
    name: "Tech Agent",
    description:
      "Specializes in technology areas including software development, hardware, AI, \
      cybersecurity, blockchain, cloud computing, emerging tech innovations\
      and pricing/costs related to technology products and services. \
      Be brief in the response",
    streaming: false,
    inferenceConfig: {
      temperature: 0.1,
    },
    customSystemPrompt:{
      template:
      `You are a Tech Agent. Specializes in technology areas including software development, hardware, AI,
      cybersecurity, blockchain, cloud computing, emerging tech innovations
      and pricing/costs related to technology products and services.
        and medical terminology or concepts. Provide helpful and accurate information based on your expertise.
        You will engage in an open-ended conversation, providing helpful and accurate information based on your expertise.
        The conversation will proceed as follows:
        - The human may ask an initial question or provide a prompt on any topic.
        - You will provide a relevant and informative response.
        - The human may then follow up with additional questions or prompts related to your previous response, allowing for a multi-turn dialogue on that topic.
        - Or, the human may switch to a completely new and unrelated topic at any point.
        - You will seamlessly shift your focus to the new topic, providing thoughtful and coherent responses based on your broad knowledge base.
        Throughout the conversation, you should aim to:
        - Understand the context and intent behind each new question or prompt.
        - Draw insights and connections from your extensive knowledge when appropriate.
        - Ask for clarification if any part of the question or prompt is ambiguous.
        - Maintain a consistent, respectful, and engaging tone tailored to the human's communication style.
        - Seamlessly transition between topics as the human introduces new subjects.

       Respond with concise way in less than 100 words.`,
    }});

const healthAgent = new BedrockLLMAgent({
      name: "Health Agent",
      description:
        "Focuses on health and medical topics such as general wellness, nutrition, \
        diseases, treatments, mental health, fitness, healthcare systems \
        and medical terminology or concepts. \
        Be brief in the response",
      inferenceConfig: {
        temperature: 0.1,
      },
      customSystemPrompt:{
        template:
        `You are a Health Agent. Focuses on health and medical topics such as general wellness, nutrition,
        diseases, treatments, mental health, fitness, healthcare systems
        and medical terminology or concepts. Provide helpful and accurate information based on your expertise.
        You will engage in an open-ended conversation, providing helpful and accurate information based on your expertise.
        The conversation will proceed as follows:
        - The human may ask an initial question or provide a prompt on any topic.
        - You will provide a relevant and informative response.
        - The human may then follow up with additional questions or prompts related to your previous response, allowing for a multi-turn dialogue on that topic.
        - Or, the human may switch to a completely new and unrelated topic at any point.
        - You will seamlessly shift your focus to the new topic, providing thoughtful and coherent responses based on your broad knowledge base.
        Throughout the conversation, you should aim to:
        - Understand the context and intent behind each new question or prompt.
        - Draw insights and connections from your extensive knowledge when appropriate.
        - Ask for clarification if any part of the question or prompt is ambiguous.
        - Maintain a consistent, respectful, and engaging tone tailored to the human's communication style.
        - Seamlessly transition between topics as the human introduces new subjects.

        Respond with concise way in less than 100 words.`,
      }
    });

orchestrator.addAgent(techAgent);
orchestrator.addAgent(healthAgent);
  

For the weather Agent I need to retrieve weather data information via API Call. This is how I defined our tool that I can use with Amazon Bedrock:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
import { ConversationMessage, ParticipantRole } from "multi-agent-orchestrator";

export const  weatherToolDescription = [
    {
      toolSpec: {
            name: "Weather_Tool",
            description: "Get the current weather for a given location, based on its WGS84 coordinates.",
            inputSchema: {
                json: {
                    type: "object",
                    properties: {
                        latitude: {
                            type: "string",
                            description: "Geographical WGS84 latitude of the location.",
                        },
                        longitude: {
                            type: "string",
                            description: "Geographical WGS84 longitude of the location.",
                        },
                    },
                    required: ["latitude", "longitude"],
                }
            },
        }
    }
];

export const WEATHER_PROMPT = `
You are a weather assistant that provides current weather data for user-specified locations using only
the Weather_Tool, which expects latitude and longitude. Infer the coordinates from the location yourself.
If the user provides coordinates, infer the approximate location and refer to it in your response.
To use the tool, you strictly apply the provided tool specification.

- Explain your step-by-step process, and give brief updates before each step.
- Only use the Weather_Tool for data. Never guess or make up information. 
- Repeat the tool use for subsequent requests if necessary.
- If the tool errors, apologize, explain weather is unavailable, and suggest other options.
- Report temperatures in °C (°F) and wind in km/h (mph). Keep weather reports concise. Sparingly use
  emojis where appropriate.
- Only respond to weather queries. Remind off-topic users of your purpose. 
- Never claim to search online, access external data, or use tools besides Weather_Tool.
- Complete the entire process until you have all required data before sending the complete response.
`

interface InputData {
    latitude: number;
    longitude: number;
}

interface WeatherData {
    weather_data?: any;
    error?: string;
    message?: string;
}

export async function weatherToolHanlder(response:ConversationMessage, conversation: ConversationMessage[]):Promise<any>{

    const responseContentBlocks = response.content as any[];
      
    // Initialize an empty list of tool results
    let toolResults:any = []
  
    if (!responseContentBlocks) {
      throw new Error("No content blocks in response");
    } 
    for (const contentBlock of responseContentBlocks) {
        if ("text" in contentBlock) {
        }
        if ("toolUse" in contentBlock) {
            const toolUseBlock = contentBlock.toolUse;
            const toolUseName = toolUseBlock.name;
    
            if (toolUseName === "Weather_Tool") {
                const response = await fetchWeatherData({latitude: toolUseBlock.input.latitude, longitude: toolUseBlock.input.longitude});
                toolResults.push({
                    "toolResult": {
                        "toolUseId": toolUseBlock.toolUseId,
                        "content": [{ json: { result: response } }],
                    }
                });
            }      
        }
    }
    // Embed the tool results in a new user message
    const message:ConversationMessage = {role: ParticipantRole.USER, content: toolResults};
    
    return message;
}
  
async function fetchWeatherData(inputData: InputData): Promise<WeatherData> {
    const endpoint = "https://api.open-meteo.com/v1/forecast";
    const { latitude, longitude } = inputData;
    const params = new URLSearchParams({
      latitude: latitude.toString(),
      longitude: longitude?.toString() || "",
      current_weather: "true",
    });
  
    try {
      const response = await fetch(`${endpoint}?${params}`);
      const data = await response.json() as any;  
      if (!response.ok) {
        return { error: 'Request failed', message: data.message || 'An error occurred' };
      }
  
      return { weather_data: data };
    } catch (error: any) {
      return { error: error.name, message: error.message };
    }
  }

We can now add it to our orchestrator

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import { weatherToolDescription, WEATHER_PROMPT, weatherToolHanlder } from './weather_tool'

const weatherAgent = new BedrockLLMAgent({
    name: "Weather Agent",
    description: "Specialized agent for giving weather condition from a city.",
    streaming: false,
    inferenceConfig: {
      temperature: 0.0,
    },
    toolConfig:{
      useToolHandler: weatherToolHanlder,
      tool: weatherToolDescription,
      toolMaxRecursions: 5
    }
  });
  
weatherAgent.setSystemPrompt(WEATHER_PROMPT);
orchestrator.addAgent(weatherAgent);

For the Airlines bot, we have reused an existing sample from AWS blog post. Once deployed we can also add it to our agent's dream team:

1
2
3
4
5
6
7
8
9
orchestrator.addAgent(
    new LexBotAgent({
      name: "AirlinesBot",
      description: "Helps users book and manage their flight reservation",
      botId: "replace with your bot id",
      botAliasId: "replace with your bot alias id",
      localeId: "en_US",
    })
  );

Amazon Lex

We utilize Amazon Lex, an AWS service designed for building conversational interfaces, to transcribe user voice input into text. This powerful tool enables seamless integration of voice and text capabilities into our application.

Once the voice input is transcribed, the resulting text is processed by our custom bot. The primary function of this bot is to act as an intermediary, efficiently transferring the transcribed text to our team of agents through a multi-agent orchestrator.

This workflow enables efficient processing of voice inputs, leveraging the strengths of Amazon Lex for accurate transcription and our custom multi-agent system for comprehensive and context-aware responses.

Here is the visual of the Fallback intent:

Image not found

Fullfillment Lambda for Fallback Intent

Amazon Connect Flow

With that in place, it is now time to enable voice for our Ai Agents. I use Amazon Connect, an AI-powered application that provides one seamless experience for your contact center customers and users. In my case, I created a new Amazon Connect instance and created a flow to interact with my fleet of Ai Agents. Here is how the flow looks like:

Image not found

Amazon Connect Flow

Benefits: Why This Integration is the Bee's Knees

🕰️ 24/7 Availability: These AI agents never sleep, unlike that barista who keeps spelling your name wrong.
📈 Scalability: Handle call spikes easier than a porcupine handles its quills.
🎯 Consistency: Get responses so consistent, they'll make a metronome jealous.
📊 Data-Driven Insights: Gather more valuable data than a squirrel gathers nuts for winter.
🔄 Seamless Escalation: Transition to human agents smoother than a buttered slide.

Challenges and Considerations: Because Nothing's Perfect (Except Maybe This System)

🎙️ Voice Recognition Accuracy: Ensure your Lex bot can understand accents better than your aunt who claims she speaks five languages.
🤔 Handling Complex Queries: Some questions might still stump your AI. We're working on world peace next.
🔒 Data Privacy: Protect customer info like it's the secret recipe for your grandma's apple pie.
💪 Continuous Improvement: Keep refining your system. Think of it as sending your AI to gym class.

Conclusion: Your Customer Service's New Best Friend

By integrating AI agents with Amazon Connect through Amazon Lex, we're not just improving customer service; we're revolutionizing it.

This system combines the efficiency of AI with the personal touch of voice interactions, creating a customer service experience smoother than a freshly waxed bowling lane.

Remember, the goal isn't to replace your human agents with robots (we'll save the robot uprising for another day). Instead, we're giving your human agents superpowers, freeing them up to handle the really tricky stuff – like explaining why the in-flight movie selection doesn't include "Snakes on a Plane."

As you implement this system, keep refining and improving. With some careful tweaking, you'll have a customer service operation so slick, even the most grumpy customers will crack a smile.

Now go forth and let your AI speak! Just remember, with great power comes great responsibility... and possibly a few confused customers who think they've been transported to the future.

Explore more

Ready to dive deeper? Check out these resources to get the most out of the Multi-Agent Orchestrator and related AWS services:

Multi Agent Orchestrator

📚 Documentation: Get comprehensive guidance and details on how to use the toolkit effectively.
🛠️ GitHub Repository: Access the source code, contribute, or browse issues and updates.
📦 NPM Repository
📦 PyPi Repository

If you find this framework helpful, please consider giving us a star on GitHub. We'd love to hear your thoughts, so feel free to leave a comment below. And if you have ideas for new features or improvements, don't hesitate to create a feature request on our GitHub repository. Who knows, your idea might be the next big thing in AI customer service!

Image not found

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Select your cookie preferences

Site Terms, Privacy, and more.

Speak Up, AI: Voicing Your Agents with Amazon Connect, Lex, and Bedrock

Here's what I'll cover in this article:

System Architecture: Where the Magic Happens

Demo: The Magic in Action

Behind The Scene

Amazon Lex

Amazon Connect Flow

Benefits: Why This Integration is the Bee's Knees

Challenges and Considerations: Because Nothing's Perfect (Except Maybe This System)

Conclusion: Your Customer Service's New Best Friend

Explore more

Multi Agent Orchestrator

Comments