Enhancing IVS Customer Success with Generative AI: Leveraging Amazon Q with Slack
a journey on how an internal chatbot helps to tap into our vast knowledge-base with Amazon Q and Slack
Sasha
Amazon Employee
Published Jun 26, 2024
Customers often discover unique uses for any technology, leading to integration-related inquiries. On IVS Customer Success, we work directly with Amazon IVS customers to find solutions to their questions, which results in a significant buildup of internal knowledge. To manage this knowledge, we capture interactions in customer engagement documentation, and use a ticketing system to track customer details and provide continuity. These artifacts often contain remedy information in the form of a fix or workaround to a problem. The problem is providing rapid access to these gems to our front line customer success personnel and to other cross team members for knowledge sharing.
In it’s default form, much of this knowledge too specific to be included in the public user guides. Instead, it is stored internally and made accessible to frontline team members who mentally index this data. While searching is possible, it requires searching many internal repositories and documents with not always great search capabilities. And like any pre-GenAI based search it requires that you know what you’re looking for. To help unite these repositories and make searching more “intelligent” why not leverage a Generative AI that taps into this knowledge layer?
Parallel to our internal efforts to build a bespoke generative AI tool, this presented an opportunity to explore Amazon Q to compare and evaluate its retrieval augmentation (RAG) and inference capabilities. Given that our team already uses Slack daily, my first thought was to use Slack as the interface for interactivity.
Here is a high level view of the application architecture:
Image not found
- An existing support Slack Bot is expanded to work with Amazon Q to facilitate the look up past conversations and access a large knowledge base for IVS questions.
- Using an Amazon Q native retrieval augmentation pipeline to create embedding to run inference on 2000+ redacted/sanitized tickets, documents and public user guides.
- The bot provides attribution to the both internal and external source document URLs.
Image not found
First I extracted ticket data from our internal tools. Each file was tagged with a unique identifier (UUID) for clarity and traceability, allowing us to trace content back to the original artifact or customer interaction. The next step was sanitizing and redacting any sensitive information in these records.
Following the initial extraction of the knowledge data, the next step is sanitization and reduction of context. This involves filtering out sensitive data from over 40+ permutations of patterns, including personally identifiable information, confidential details, and temporal references.
Some example regexs:
1
2
3
4
5
6
7
8
9
10
11
12
func sanitizeWord(word string, customerNames []string) string {
// Regular expressions trying to filter text by ines and then each line by words
stageArnRegex := regexp.MustCompile(`arn:aws:ivs:[\w-]+:\d+:stage/[A-Za-z0-9]+`)
starfruitRegex := regexp.MustCompile(`\bStarfruit\b|AWSStarfruit`)
rtmpsSupportUrlRegex := regexp.MustCompile(`rtmps://[\w-]+\.support\.contribute\.live-video\.net(:\d+)?/[\w-]*/`)
tinyAmazonUrlRegex := regexp.MustCompile(`https:\/\/tiny\.amazon\.com\/[\w]+`)
dateRegex := regexp.MustCompile(`\d{4}-\d{2}-\d{2}`)
emailRegex := regexp.MustCompile(`\S+@\S+`)
accountIdRegex := regexp.MustCompile(`\b\d{12}\b`)
caseIdRegex := regexp.MustCompile(`\b\d{11}\b`)
arnRegex := regexp.MustCompile(`(arn:aws:|)\b[\w-]+:[\w-]+:\d+:[\w-/]+`)
urlRegex := regexp.MustCompile(`https?:\/\/\S*\.(mp4|m3u8|ts)`)The goal is to seek out to replace any identifiers, stream/channel details, amazon-internal domains, confidential data, and other sensitive elements by replacing these with placeholders.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// Apply regex patterns
switch {
case stageArnRegex.MatchString(word):
fmt.Println("Matched Stage ARN Regex:", word)
return "[Statge ARN]"
case starfruitRegex.MatchString(word):
fmt.Println("Matched starfruitRegex:", word)
return "[Starfruit]"
case rtmpsSupportUrlRegex.MatchString(word):
fmt.Println("Matched rtmpsSupportUrlRegex:", word)
return "[RTMPSSupportURL]"
...
case accountIdRegex.MatchString(word):
fmt.Println("Matched accountIdRegex:", word)
return "[AccountId]"
...By doing so, we effectively prevent the accidental leakage of such information during the retrieval augmentations operations that follow. Once redacted, this data is ready to be ingested into a retrieval augmentation pipeline of Amazon Q to generate embeddings.
Amazon Q provides the ability to feed the data artifacts via integrated services, simplifying the more involving part of “prompt engineering” typically required of implementing a custom retrieval augmentation (RAG) orchestration.
The flow to create an app goes something like this:
Image not found
Amazon Q console view in the first steps of creating an application:
Image not found
The next step of ingestion involves selecting a retriever.
Amazon Q allows you to choose between native or existing retrievers, each offering varying functionalities (see screenshot below). Additionally, you need to specify an index provisioning. For our purposes, we use the Native retriever with Starter Index provisioning, as it meets the criteria for our proof of concept needs and development.
Image not found
To build a retriever, first you must connect data sources.
Amazon Q integrates with a number of AWS and partnered services for retrieving large data sets, providing a wide selection of integrations:
Image not found
The setup process involves specifying your S3 bucket details and defining the parameters for how Amazon Q should process and retrieve information from your dataset. In our case, we set up a couple of S3 buckets containing pre-processed ticket artifacts, web crawler data, and Quip documents.
We also specify Sync scopes and Sync mode as part of S3 ingestion. After pointing to a bucket containing the artifacts, have the Retriever sync the database and index the sources. This can take tens of minutes, depending on the dataset size. Additionally, both the sync for the S3 and the Web crawler (that can go through web pages — in our case IVS public user guides) can be scheduled to run at specific intervals. In our case, we set the crawler to run weekly and the S3 sync to run upon detecting changes in the bucket.
Image not found
Retrieval is a computationally expensive process and will incur costs. An initial full sync is advised when indexing new buckets or crawling through ser guides. However, for any subsequent runs, the “New, Modified, or Deleted Content Sync” mode is preferred. This mode can help avoid unexpected costs that can arise from repeatedly executing the retrieval step at frequent intervals.
Image not found
Once the setup is complete, running the inference step is straightforward. At this point we add our Slack integration.
Note: Amazon Q provides a UI component to the application the users can interact with, however we won’t cover this topic here.
So once we’ve put this together, how do we get Slack to interact our Q?
One challenge was concurrency. Since running inference takes up to 20 seconds, and Slack times out after 3000 ms, we needed a mechanism to queue the request. For this we chose Amazon Simple Queue Service. SQS allows us to buffer incoming messages and process them asynchronously when they complete. This is also useful when dealing with high volumes of incoming queries.
Example of queuing request using Golang using a FIFO queue:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
func queueEventForProcessing(eventDetails EventDetails) error {
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatalf("Unable to load SDK config, %v", err)
}
sqsClient := sqs.NewFromConfig(cfg)
queueURL := "https://sqs.us-west-2.amazonaws.com/<accountID>/q-bot-queue.fifo"
// serealizing eventDetails to JSON
messageBody, err := json.Marshal(eventDetails)
if err != nil {
return fmt.Errorf("error marshalling event details: %w", err)
}
// sending the message to the SQS queue
_, err = sqsClient.SendMessage(context.TODO(), &sqs.SendMessageInput{
QueueUrl: aws.String(queueURL),
MessageBody: aws.String(string(messageBody)), // the message we are sending
MessageGroupId: aws.String("Group1337"), // some unique identifier for the group of messages
})
if err != nil {
return fmt.Errorf("error sending message to SQS: %w", err)
}
return nil
}The request flow starts with receiving an event from the SQS queue, which contains the user’s message and metadata such as the channel ID and thread timestamp. The bot then makes an API request to Amazon Q. This involves creating a `ChatSyncInput` object with the relevant application ID, user ID and message. The request is sent using the `qbusiness.ChatSync` method, and the response is logged and processed. Here’s an example of how the API request may look like:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
func handleQBusinessQueryWithAttribution(ctx context.Context, applicationID, userMessage string, wg *sync.WaitGroup, responses map[string]string, label string) {
defer wg.Done()
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
log.Printf("Unable to load SDK config: %v", err)
responses[label] = fmt.Sprintf("%s: Error loading SDK config", err)
return
}
qClient := qbusiness.NewFromConfig(cfg)
input := &qbusiness.ChatSyncInput{
ApplicationId: aws.String(applicationID),
UserId: aws.String("<accountID>"),
UserMessage: aws.String(userMessage),
}
resp, err := qClient.ChatSync(ctx, input)
if err != nil {
log.Printf("Failed to send message to qClient: %v", err)
responses[label] = fmt.Sprintf("%s: Failed to query Qbusiness service", err)
return
}
log.Printf("Received response from Amazon Q (%s): %+v", label, resp)
responseMessage := formatResponseAttribution(resp)
responses[label] = fmt.Sprintf("%s: %s", label, responseMessage)
}When it comes to sending the Amazon Q response back to Slack, the process involves capturing the inferred response and any associated attributions. The named `sendMessageToSlack` function handles this by taking the user’s input, the generated response message, the thread timestamp, and the user’s channel ID. Typically, the response from Amazon Q includes the inferred response along with a couple of source attributions, which are formatted and included in the message sent back to Slack.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// sendMessageToSlack posts a message back to Slack. Ensure this uses the correct access token.
func sendMessageToSlack(userInput, messageText, threadTimestamp, userChannelID string) error {
slackToken := os.Getenv("SLACK_TOKEN")
api := slack.New(slackToken)
// Log the message details before sending to Slack
log.Printf("Sending message to Slack: ChannelID=%s, ThreadTimestamp=%s, Message=%s", userChannelID, threadTimestamp, messageText)
_, _, err := api.PostMessage(
userChannelID,
slack.MsgOptionText(messageText, false),
slack.MsgOptionPostMessageParameters(
slack.PostMessageParameters{
UnfurlLinks: true,
ThreadTimestamp: threadTimestamp,
},
),
)
if err != nil {
log.Printf("Failed to send message to Slack: %s", err.Error())
return errors.New("Failed to send message to Slack: " + err.Error())
}Additionally, to provide proper attribution for the information retrieved, I needed to regenerate internal URLs based on the UUID so that a user can jump to the internal source ticket or a document.
This involves extracting specific patterns from the titles of the response attributions. For instance, titles matching the pattern “([a-zA-Z0–9-]+)_sanitized(\.json|_claude\.txt)” are transformed into internal URLs like https://subdomain.amazon.com/{alphanumericName}.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// Append attribution information if available
if len(resp.SourceAttributions) > 0 {
message += "\n\n*Source Attributions*:\n"
re := regexp.MustCompile(`([a-zA-Z0-9-]+)_sanitized(\.json|_claude\.txt)`)
for _, attribution := range resp.SourceAttributions {
// Check if the title matches the pattern
match := re.FindStringSubmatch(*attribution.Title)
if len(match) > 1 {
// If the title matches the regex, generate a new URL based on the match
alphanumericName := match[1]
newURL := "https://a2z.amazon.com/" + alphanumericName
cleanSnippet := cleanSnippetText(*attribution.Snippet) // HTML Unescape string formatting
message += fmt.Sprintf("\nTitle: %s\n\nURL: %s\n\nSnippet: %s\n\n", *attribution.Title, newURL, cleanSnippet)
} else {
// If the title does not match the regex, use the original URL
originalURL := "URL not available"
if attribution.Url != nil {
originalURL = *attribution.Url
}
cleanSnippet := cleanSnippetText(*attribution.Snippet) // this helps with unescape HMTL
message += fmt.Sprintf("\nTitle: %s\n\nURL: %s\n\nSnippet: %s\n\n", *attribution.Title, originalURL, cleanSnippet)
}
}
} else {
message += "\nNo source attributions provided."
}So far, it’s been useful; I use the bot on the weekly and at times daily to help me look up information that I think may be floating in the knowledge base but I don’t know the keywords for. Not only does the bot locate the information, but it also provides me with reasonably phrased language that I can quickly repurpose for my correspondence. Additionally, it offers attributes and links to the sources where it found the information, helping me to identify and dig into more specifics if needed. The bot also provides snippets of the original articles or language used in the correspondence.
When compared to a ClaudeV3-based retrieval that we’ve built internally for wider customer use and the initial yet withstanding Amazon Q implementation, we’ve found that Amazon Q’s tone is well-suited for our internal needs, allowing us to look up information with more specifics quickly.
Here are a couple of simple examples:
Image not found
Image not found
Next, I plan to set up automation to provider Support with proposed responses upon support request creation. An IVS colleague can then review these proposals and, with some adjustments, provide speedy and informed replies to the inquirer, likely saving tenth of minutes of search and writing time.
## Special Thanks to Mike Gaffney aka "Gaffo", an L8 Principal Engineer Amazon/Twitch colleague (no big deal!) for helping me to revise this post!
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.
Comments
Log in to comment