Deploying Langfuse on Amazon ECS with AWS Fargate: A Step-by-Step Guide Using AWS CDK
Explore Langfuse, an open-source LLM engineering platform that enhances collaboration in debugging and improving AI applications. Learn to deploy it with AWS CDK, ECR, ECS, and Fargate.
In the rapidly evolving field of artificial intelligence, particularly in the development and deployment of large language models (LLMs), Langfuse is emerging as an innovative platform. As an open-source LLM engineering platform, Langfuse assists development teams in collaboratively debugging, analyzing, and iteratively improving LLM applications. This blog post will guide you through deploying Langfuse using the AWS Cloud Development Kit (CDK) alongside Amazon Elastic Container Registry (ECR), Amazon Elastic Container Service (ECS), and AWS Fargate.
Key Features of Langfuse
Langfuse addresses several critical issues in the development and operation of LLM applications:
Observability and Debugging: Provides comprehensive tracing tools to capture all relevant data points and interactions within LLM applications, helping identify root causes in complex, iterative, chained, or agent-based calls. It offers integrations with popular frameworks like Python and JavaScript/TypeScript asynchronous SDKs, OpenAI SDK, LangChain, and LlamaIndex.
Prompt Management: Allows seamless management, versioning, and deployment of prompts. It provides an LLM playground for testing and iterating prompts to improve performance.
Analytics and Cost Optimization: Tracks key metrics such as LLM costs, latency, and quality. It offers deep analysis through intuitive dashboards and data exports to identify resource-intensive areas of LLM applications and target cost-saving efforts.
Quality Monitoring and Evaluation: Enables scoring of production traces to measure quality more closely, supporting model-based evaluation, user feedback collection, and manual annotation.
Experimentation and Testing: Allows tracking and testing of application behavior before deploying new versions. It supports datasets for testing expected input-output pairs and benchmarking performance.
User Behavior Analysis: Helps understand various inputs and user intents in LLM applications by classifying inputs and conducting in-depth analysis of user behavior.
Deploying Langfuse to Amazon ECS
AWS CDK is an Infrastructure as Code (IaC) tool that allows you to manage infrastructure by codifying it, enabling you to define and provision cloud resources programmatically. Amazon ECS is a fully managed container orchestration service that helps you easily deploy, manage, and scale container applications.
Here, I will explain how to configure the architecture for deploying Langfuse to Amazon ECS using AWS CDK.
The deployment sequence for the CDK project is as follows:
Setting up the CDK project environment
Configuring the CDK context (cdk.context.json)
Deployment
1. Setting up the CDK Project Environment
The Langfuse CDK project repository is structured as a standard Python project. The CDK project includes a configuration file, cdk.json, which instructs how to run the app using CDK commands.
To run the CDK Python app, set up a Python virtual environment (the .venv directory).
For macOS and Linux:
For Windows:
Install the required Python packages:
2. Configuring the CDK Context
The cdk.context.json file stores context information used by the CDK application in a key-value format. Save the necessary settings for Langfuse deployment in the cdk.context.json file as shown in the example below.
To explain the key properties included in the cdk.context.json file:
db_cluster_name: Specifies the name of the PostgreSQL database cluster used by the Langfuse application.
ecs_cluster_name: Defines the name of the Amazon ECS cluster, which is used to host the Langfuse application.
ecs_service_name: Specifies the name of the Amazon ECS Service, which is used with an Application Load Balancer (ALB) to run the Langfuse application.
image_version: Specifies the version of the Langfuse Docker image to be deployed. "latest" means using the latest version.
langfuse_env: This configuration object defines the environment variables for the Langfuse application.
NODE_ENV: Indicates that the application is running in production mode (can have values like development or production).
NEXTAUTH_SECRET: The secret key used for NextAuth.js authentication.
SALT: A random data value used for data encryption or hashing.
TELEMETRY_ENABLED: Enables telemetry data collection.
NEXT_PUBLIC_SIGN_UP_DISABLED: Enables the user sign-up feature.
LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: Enables experimental features of Langfuse.
These settings play a crucial role in configuring the deployment and execution environment of the Langfuse application. Particularly, the environment variables within the langfuse_env object are used to control the application's behavior and manage security settings.
For security reasons, important values like NEXTAUTH_SECRET and SALT should be generated and managed securely during actual deployment. It is recommended to use the openssl rand -base64 32 command to generate these values.
Once Langfuse is deployed, you can use its trace functionality to track and analyze the execution of LLM applications. A trace in Langfuse typically represents a single request or task and includes metadata about the request as well as the complete input and output of the function.
1. Accessing Langfuse URL
To access the Langfuse web application, use the following AWS CLI command to find the URL of Langfuse:
2. Creating a New Project in Langfuse
Langfuse organizes the development, debugging, analysis, and optimization of LLM applications into units called projects. When you first access Langfuse, you can create a new project by following these steps:(Step2) Create a New Project: After logging in, click the "New" button on the home screen to create a new project.
(Step 1) Create a Langfuse Account: Sign up or log in to the Langfuse website.
(Step2) Create a New Project: After logging in, click the "New" button on the home screen to create a new project.
(Step 3) Generate API Credentials: In the project settings, click "Create API Keys" to generate new API credentials. These credentials are necessary for integrating LLM applications with Langfuse.
3. Logging Amazon Bedrock LLM Application Calls in Langfuse
ℹ️ LANGFUSE_SECRET_KEY and LANGFUSE_PUBLIC_KEY are the API Keys created when setting up the Langfuse project.
Initialize Langfuse Callback Handler:
Initialize Amazon Bedrock LLM Model:
Example LLM Application Code:
Now, when you run the LLM application code, you can view the generated trace in the Langfuse web console.
4. Analyzing Traces
By analyzing the traces generated in Langfuse, you can understand the performance and behavior of your LLM application.Click on the trace of the LLM application you want to analyze to see the execution information stored in the trace.
Click on the Traces menu in the Langfuse web console to view the full list of traces.
Click on the trace of the LLM application you want to analyze to see the execution information stored in the trace.
A trace includes information such as:
Input and Output: Complete input and output data of the function.
Metadata: Additional information like user, session, and tags.
Observations: Records of individual steps in the execution process, including events, spans, and logs of AI model generations.
Conclusion
In this blog post, we introduced how to efficiently deploy Langfuse using AWS CDK, Amazon ECR, ECS, and AWS Fargate. Now, you can leverage the powerful features of Langfuse to debug, analyze, and optimize LLM applications.
Deploying Langfuse on Amazon ECS Fargate offers several benefits that enhance the overall efficiency, security, and scalability of LLM applications:
Simplified Infrastructure Management: AWS Fargate abstracts the underlying infrastructure, allowing you to focus on building and deploying applications without worrying about server provisioning, scaling, or operating system management.
Enhanced Security: AWS Fargate provides built-in security features such as isolation between containers, secure communication, and integration with AWS IAM for granular access control.
Scalability: AWS Fargate supports auto-scaling by dynamically adjusting the number of active containers to handle varying loads without manual intervention.
Cost Efficiency: AWS Fargate follows a pay-as-you-go pricing model, charging only for the vCPU and memory resources used, helping avoid costs associated with over-provisioning.
Flexibility: AWS Fargate allows you to specify CPU and memory resources for each container and supports both Linux and Windows containers, seamlessly integrating with various containerization technologies like Docker.
Easy Integration with Other AWS Services: AWS Fargate integrates seamlessly with other AWS services such as Amazon Elastic Container Registry (ECR), CloudWatch, and AWS IAM, providing a comprehensive solution for container management.
Logging and Visibility: Integration with Amazon CloudWatch in AWS Fargate provides robust logging and visibility capabilities, enabling real-time monitoring and quick identification and resolution of issues.
Cleaning Up Resources
If you no longer need Langfuse, you can delete all CloudFormation stacks using the following CDK command:
By following this guide, you can effectively deploy and manage Langfuse, allowing you to focus on enhancing your LLM applications' capabilities and performance.
