AWS Logo
Menu

Envoy AI Gateway with Amazon Bedrock - Getting Started Guide

Learn how to integrate Envoy AI Gateway with Amazon bedrock and 3rd party model providers. This tutorial guides through the setup process with a practical example.

amazonbedrockgenaiopsgenaigateway
Swagat Kulkarni
Amazon Employee
Published Mar 11, 2025

Introduction

The integration of Artificial Intelligence (AI) into various applications is rapidly transforming industries. However, managing AI service traffic, especially when leveraging multiple AI providers, can be complex. This article explores how to use Envoy AI Gateway in conjunction with Amazon Bedrock to simplify and enhance AI application management.

What is an AI Gateway?

An AI Gateway is a specialized middleware platform that facilitates the integration, management, and deployment of artificial intelligence (AI) models and services within an organization's IT infrastructure. It acts as a bridge between AI systems and end-user applications, providing a unified endpoint for connecting AI infrastructure and services.
Envoy AI Gateway is an open-source project for using Envoy Gateway to handle request traffic from application client to GenAI services. It provides capabilities such as:
  • Routing: Directing traffic to specific AI models or providers based on defined rules.
  • Authentication: Securing access to AI services.
  • Security Policies: Implementing security measures to protect AI applications.
In this article, we will guide you through a sample notebook that demonstrates deploying Envoy AI Gateway on an Amazon EKS cluster and invoking large language models hosted on Amazon Bedrock and third-party model providers.
The sample code is available at the repository here.

Prerequisites

Before diving into the integration, ensure you have the following tools installed:
  • kubectl: Kubernetes command-line tool.
  • helm: Kubernetes package manager.
  • curl: For testing API endpoints.
  • AWS CLI: AWS command-line interface.

Setting Up the Environment

The following steps outline the setup process:
1. Kubernetes Cluster on EKS
  • Create a Kubernetes cluster on Amazon EKS (Elastic Kubernetes Service). A detailed guide can be found in the EKS Workshop.
  • Note the name of your newly created EKS cluster.
2. Update Kubeconfig File
  • Update your kubeconfig file using the AWS CLI, replacing <YOUR_AWS_REGION> and <YOUR_EKS_CLUSTER_NAME> with your specific details:
 3. Install Envoy Gateway
  • Install the Envoy Gateway system using Helm:
4. Install Envoy AI Gateway
  • Install the Envoy AI Gateway using Helm:
5. Configure Envoy Gateway
  • Apply the necessary YAML configurations:
  1. Verify Pods
  • Ensure that the pods are running in the envoy-ai-gateway-system and envoy-gateway-system namespaces:
Now, we will apply the basic configuration for Envoy AI Gateway as described in the basic.yaml file.
At a high level, the basic.yaml file defines the configuration for an Envoy AI Gateway, setting up routing, security policies, and backend connections to AI services like OpenAI and Amazon Bedrock within a Kubernetes environment.
Wait for the pods to be ready:

Configure Gateway URL

Let's verify the assigned external IP from the envoy gateway system and set the GATEWAY_URL environment variable.
Set the GATEWAY_URL environment variable:
Now, with this setup in place, we are ready to test the model backend. We will demonstrate invoking both Amazon Bedrock model and an OpenAI model via the gateway URL

Testing with Model Backend

To test the integration with Amazon Bedrock and OpenAI:
  1. Update AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the basic.yaml file.
  2. Enable access to "amazon.nova-lite-v1:0" in the us-east-1 region.
  3. Update OPENAI_API_KEY in the basic.yaml file.
  4. Update x-ai-eg-model value to us.amazon.nova-lite-v1:0 from from us.meta.llama3-2-1b-instruct-v1:0 in basic.yaml file.
Note - For production use, consider use of AWS secrets manager to store credentials. You can run retrieve_secrets.py file under utils folder to dynamically access the secrets from AWS Secrets Manager and update the values.
Apply the updated configuration:

Example: Text Summarization and Sentiment Analysis

The example demonstrates how to use Amazon Bedrock and OpenAI models via Envoy AI Gateway:
  1. Summarization: Invoke the Amazon Nova Lite model on Bedrock to generate a summary of a customer review.
  2. Sentiment Analysis: Pass the output of the first LLM call to OpenAI gpt-4o-mini model to determine the overall sentiment of the customer review.
Here's the example Bedrock payload:
Generated summary from Bedrock:
We then pass this summary to OpenAI. Here is the payload:
And the generated response:

Conclusion

In this article, we demonstrated the integration of Envoy AI Gateway, an open source project built on Envoy Proxy , with various LLM providers including Amazon Bedrock and OpenAI to solve for an example use case.
AI Gateways are essential for organizations looking to integrate diverse AI service while maintaining granular control over routing, security and costs. Envoy AI Gateway abstracts the complexities of integration and reduces the operational overhead.
 

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Comments