Using Amazon Bedrock to compare Retrieval Augmented Generation (RAG) based Generative AI (GenAI) application between Amazon Nova Pro and Anthropic Claude 3.5 Sonnet

ABSTRACT

The main objective of this article is to share a quick and easy way to build a Retrieval Augmented Generation (RAG) based Generative AI (GenAI) application using Amazon Bedrock. We are using information from few HTML/PDF documents with general descriptions in the areas of Environmental, Social and Governance (ESG). These documents are embedded and form our knowledge base on the subject. This example approach can be easily adapted to use different high-performing Foundation Models (FMs) with advanced RAG to evaluate the LLM responses based on the required tasks.

GEN-AI RAG PIPELINE OVERVIEW

The high level task pipeline for the GenAI-powered application :

Create a Knowledge Base from the documents to be used as contexts for FM queries.
Select the secure, reliable, accurate, efficient and cost-effective FM.
Based on the user queries and document embeddings (Cohere Embed English), retrieve the similar document chunks from vector store with FAISS engine.
For improved relevancy and accuracy, rerank the retrieved similar document chunks.
Augment user query with the reranked document chunks.
Re-write user query and perform Prompt Construction for the selected FM (Amazon Nova Pro / Claude 3.5 Sonnet).
Use Amazon Bedrock Guardrails to filter harmful contents and topics on both the user inputs and FM responses.
Perform FM responses with streaming for responsiveness.
Format the FM responses accordingly based on the use case.
Collect user feedback on the responses for potential model improvements.
Save the Queries and Responses for model evaluations, model fine-tuning and/or continued pre-training.

Python Libraries

The following libraries are required for the GenAI RAG application prototype.

Create AWS clients for making inference requests

Create a boto3 agent runtime client to connect programmatically for making retrieval from Amazon Bedrock Knowledge Base (OpenSearch Serverless).
Create a boto3 agent runtime client to connect programmatically for making inference requests from Large Language Models (LLM) hosted in Amazon Bedrock (eg. Amazon Nova Pro).

Amazon Bedrock Agent Retrieve and Generate

Retrieve embedded chunks (Cohere Embed English) from OpenSearch Serverless (OSS) vector store with semantic search.
Rerank document chunks with Cohere Rerank 3.5 model.
Stop harmful content in models using Amazon Bedrock Guardrails.
Generate LLM response in streams.

Amazon Bedrock LLM responses in streams

Perform multi-turn conversations with sessionId.
Obtain LLM response in streams after knowledge retrieval, rerank and employing Amazon Guardrails.

Display external information from RAG with citations and retrieved document chunks based on the numberOfResults parameter.

Reinforcement Learning Human Feedback (RLHF)

Collect user feedback based on the LLM responses to user queries.
Save these user feedback in JSON formatted output file and DynamoDB table.
These collected user data can be used for model evaluations with Amazon Bedrock evaluations.
This data can also be used as a source for Reinforcement Learning Human Feedback (RLHF) which can be used in fine-tuning of the Foundational Models.

DynamoDB Table - GenAI-Rerank-RAG-Chat — GenAI RAG and Rerank Chatbot Information stored in DynamoDB Table

MODEL EVALUATIONS AND IMPROVEMENTS

In the model evaluation, we can prepare a simple evaluation table (with 👍, 👎) to measure relevancy and accuracy of the application responses to the user queries. This table could also be used as a source for Reinforcement Learning Human Feedback (RLHF) which can be used in fine-tuning of the Foundational Model. Moreover, during the solution building, we can use the Amazon Bedrock Chat playground with different models to compare the prompt outputs with the application responses of the same user queries.

GenAI Chatbot with RAG User Interface

Using Streamlit to build the User interface for the LLM Chatbot with RAG.

Conversational AI with RAG and Rerank using Amazon Bedrock

REFERENCES 📚

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Select your cookie preferences

Site Terms, Privacy, and more.