Solving Problems You Can't See: How AWS X-Ray and CloudWatch Provide User-Level Observability in Serverless Microservices Applications
Avoid blind spots by tracking users on apps built on AWS.
AWS Admin
Amazon Employee
Published Sep 26, 2023
Last Modified Jun 21, 2024
Step 1: Deploy the aws-serverless-shopping-cart Sample Application
Step 2: Explore the Tracing Capabilities Already Implemented in the Sample Application
A Brief Tour of the Frontend Code
From Your Browser to API Gateway
AWS X-Ray - Your Application’s Tracing Companion
Step 3: Update the Application to Capture the Cognito User ID in AWS X-Ray
Step 4: Group Application Requests by Registered and Anonymous Users
Step 5: View and Diagnose Issues by Registered and Anonymous Users in the AWS X-Ray Console
Imagine you’re a cloud engineer for a website built using AWS serverless technologies, such as AWS Lambda. You receive a call from a customer insisting they’re unable to open their shopping cart on your website. You open the same website and log in with your ID and try the same operation, and it works fine. What do you do?
If you’re using AWS X-Ray and Amazon CloudWatch, there are plenty of things you can do to understand what is happening from this user’s or other users’ perspectives.
In this tutorial, I’ll walk you through ways AWS X-Ray and Amazon CloudWatch help you improve end-user observability. You'll use the aws-serverless-shopping-cart sample application as the starting point and enhance its AWS X-Ray usage and implementation to include an AWS X-Ray group for registered and anonymous users, along with related CloudWatch Metrics and CloudWatch Alarms. You'll also learn how to use the AWS X-Ray SDK and AWS X-Ray constructs such as segments, subsegments, and annotations. Throughout the tutorial, I’ll point out some potential gotchas and how to work around them.
- How to configure and implement AWS X-Ray in a serverless application that uses Amazon Cognito, Amazon API Gateway, and AWS Lambda
- How to follow and observe your user requests as they flow through Amazon Cognito, Amazon API Gateway, and AWS Lambda
- How to use the AWS X-Ray SDK for python to instrument your python applications with AWS X-Ray
- How to use AWS X-Ray groups to identify and diagnose applications based on user segmentation
- How to use AWS X-Ray CloudWatch metrics to alarm on AWS X-Ray groups
Before starting this tutorial, you will need the following:
| About | |
|---|---|
| ✅ AWS Level | Advanced - 300 |
| ⏱ Time to complete | 120 minutes |
| 💰 Cost to complete | Free when using the AWS Free Tier |
| 🧩 Prerequisites | - AWS Account - Git client |
| 💻 Code Sample | aws-serverless-shopping-cart |
| 📢 Feedback | Any feedback, issues, or just a 👍 / 👎 ? |
| ⏰ Last Updated | 2023-09-26 |
The aws-serverless-shopping-cart application allows us to quickly deploy a serverless application that uses multiple AWS Lambda functions configured with AWS X-Ray distributed tracing turned on. This application also integrates with Amazon Cognito, which we will leverage to identify and demonstrate user based tracing.
Clone the github repository and deploy the sample application using the first deployment option. This will provide you with more control when we make changes, and allow you to develop and test the frontend interface more easily. Make sure you set the environment variable AWS_DEFAULT_REGION when you initiate the deployment, otherwise you may receive an error.
The sample application is already instrumented with AWS X-Ray to provide request tracing for the application. You will be enhancing this capability. Open the backend/shopping-cart-service/list_cart.py file in your cloned GitHub repository. This is the python code that gets executed when a user lists items in their shopping cart. You will notice that the code is already using aws lambda powertools. This development toolkit makes it easy for you to instrument your applications with AWS X-Ray. The sample application includes annotations provided by the Tracer utility class from the aws_lambda_powertools python package. The annotation
@tracer.capture_lambda_handler is all that is needed to create an AWS X-Ray annotation named ColdStart as well as an annotation identifying your service in X-Ray traces. It will also capture exceptions generated by your lambda function as AWS X-Ray metadata.Now make sure your frontend is started and running locally using the first deployment option. Load the frontend user interface by visiting http://localhost:8080. Add a few products to your shopping cart. This will generate some requests to your backend AWS lambda functions named aws-serverless-shopping-cart-shop-ListCartFunction-<hash> and aws-serverless-shopping-cart-GetProductsFunction-<hash>.
You will first manually trace the user requests you just made when you accessed the application locally from your code, all the way through to the AWS services that were used to retrieve the product data displayed in your browser. The sample application is written using the Vue.js frontend framework. This tutorial skips the details of the overall application functionality and provides some basic high level information for you to orient yourself with the frontend application code so you can focus on the AWS X-Ray integration points.
This script creates the Vue application and loads the Vue components that make up the application. You can see that each component is loaded from a separate file in the frontend/src/components directory. You will focus on the shopping cart (CartDrawer) and product listing (Product) components.
This file provides the layout for the application. You can see that the main product content for the application is being loaded from the router in
<router-view />. You can also see that the cart drawer (<cart-drawer />) remains constant, even if the path and route for the application changes.App.vue code snippet
When the page is first loaded (
mounted()), the shopping cart for the user is retrieved:The
fetchCart() call is defined as an action in frontend/src/store/actions.js:This makes a call to
getCart() in frontend/src/backend/api.js:The
getCart() function initiates the call to retrieve the cart contents from the aws-serverless-shopping-cart-shop-ListCartFunction<hash> AWS Lambda function through API Gateway using the Amplify JavaScript SDK.This file defines what views should be loaded based on the URL path. You can see that the base URL is loading the Home view which displays the product list. The router determines what will be loaded into the layout defined in your App.js
<router-view></router-view> position.This file initiates the call to retrieve Products from the AWS lambda function named aws-serverless-shopping-cart-shop-ListCartFunction-<hash> via API Gateway and the Amplify JavaScript SDK.
You can see that when the component is created, it initiates a call to fetchProducts and stores the results in the products object. It then iterates through the returned products and lays them out using Vue components.
The fetchProducts() call is defined as an action in frontend/src/store/actions.js:
This makes a call to getProducts() in frontend/src/backend/api.js:
At this point, you may have already experienced an overflow in your mental call stack. Don’t worry! All you really need to remember is that the
getProducts() and getCart() calls from our frontend code are what initiate our call to retrieve our products and cart contents from our sample applications AWS Lambda functions using API Gateway with the Amplify JavaScript SDK.If you are using Google Chrome, you can open the Developer Tools console (View -> Developer Tools) and go to the Network tab after the sample application loads. Click on the Fetch/XHR option to filter out all the network requests to only those fetching data.
Here we can see the API Gateway endpoints that were called using
API.get() from the Amplify SDK. These endpoints are defined in the frontend/src/aws-exports.js file:When you deployed the aws-serverless-shopping-cart application, the deployment script updated the frontend/.env.local environment variables referenced in this file with the endpoints for your deployment. The use of environment variables allows your endpoints to change between environments while your code stays the same.
Now that you are leaving the confines of your local computer for the AWS cloud, you will journey through your request for product and cart data on the backend. Proceed to the AWS console where you deployed the AWS Lambda function and open the API Gateway management console. You should see two APIs listed corresponding to the product and cart APIs used by your frontend:
Open the aws-serverless-shopping-cart-shoppingcart-service API and select the GET method. This method retrieves your shopping cart contents from the AWS lambda function named aws-serverless-shopping-cart-shop-ListCartFunction<hash>:
API Gateway makes it easy to integrate with AWS Lambda functions using the LAMBDA_PROXY integration type. Next, click on the aws-serverless-shopping-cart-shop-ListCartFunction<hash> link vertically displayed to open the function.
This lambda function is using AWS Lambda function aliases. Aliases enable you to shift traffic between versions of your Lambda function. The sample application is simply pointing the live alias to the latest version of the lambda function.
Click on the function name to return to the main menu for this function and then make your way to the Code tab:
You’ve made it! You are now at the backend code that lists the contents in the shopping cart. You can now see the code that lists the shopping cart contents in list_cart.py.
Focus on the code related to AWS X-Ray:
Your lambda function handler is instrumented to create AWS X-Ray traces for calls to this AWS Lambda function with the
@tracer.capture_lambda_handler annotation.Now that you’ve taken the journey of manually tracing the application call to list shopping cart contents, you’ll see how AWS X-Ray visualizes this journey.
Proceed to the AWS X-Ray console. The AWS X-Ray console is now a part of the CloudWatch console. Select Service map under the X-Ray traces menu to see a visual representation of our X-ray traces:
If you don’t see any data, select the 1h timeframe or longer depending on how long ago you opened and used the front end service for the application in Step 1. Your visualization should look something like this:
If your account has more services instrumented with AWS X-Ray, your service map may be larger. For this sample application, you can see that there are three different client calls to three different AWS services depending on the interactions you performed. These calls were initiated by you when you used the frontend interface.
For this tutorial, you are going to focus on the shopping cart. Select the node corresponding to the AWS Lambda function execution for aws-serverless-shopping-cart-shop-ListCartFunction-<hash>. Every AWS Lambda function call in X-Ray is represented by two nodes. The first node represents the call made to AWS Lambda to execute the function (context / entry node) and the second node represents the function execution itself.
The node view enables you to quickly focus on the metrics for an X-Ray instrumented service in the context of other AWS services that it interacts with. This can help you identify and troubleshoot performance bottlenecks or issues in a particular user experience (such as listing shopping cart items).
Click View Traces to view the traces that X-Ray has captured that include this AWS Lambda Function execution. Now scroll down and click on one the traces captured by AWS X-Ray that includes this function call. You may have more than one depending on how many times you used the front end user interface and initiated calls:
The trace provides the end-to-end X-Ray instrumented view of your request. The nodes represent each service that participated in the request:
Click on the Lambda Function node to view its trace details and then click on the ## lambda_handler subsegment in the Segments timeline. Function calls are prefixed with “##”. Click the Annotations tab. The aws lambda powertools integration with X-Ray automatically identifies and annotates AWS Lambda function calls that required a cold start (ColdStart: true / false). You can see the impact of the cold start in the Initialization time before your lambda handler was executed (~1s):
X-Ray annotations are searchable. As a result, you can search and retrieve all traces that experienced a cold start. Click on the Metadata tab. Here you can see that the response from the lambda function is also captured by X-Ray. The response metadata was automatically setup by aws lambda powertools. Metadata isn’t searchable from AWS X-Ray but can provide valuable information such as troubleshooting expected vs actual responses.
In the next step, you will instrument your sample application so that user ids are captured as annotations. You will then be able to troubleshoot requests per user by filtering on the user ID annotation.
At this point, you have seen some of the benefits of AWS X-Ray in the context of your sample application and the default setup for AWS Lambda powertools. Next, you will update the AWS X-Ray implementation to also capture the user ID.
The sample application uses Amazon Cognito for user registration and authentication. Before you update the application, create a new user. Navigate to http://localhost:8080 to open the local front end server. Next, click the Sign In button and select the Create account link.
After you have registered, sign in with your newly created user. The sample application persists user shopping carts for registered users using their cognito user ID. If a user isn’t signed in, an anonymous shopping cart is also supported using cookies. In step 2, you saw that your shopping cart contents were retrieved using the
getCart() function in frontend/src/backend/api.js:The sample application is using the Amplify API class to make API Gateway requests. The Amplify SDK will automatically update your requests with the Authorization header for your Amazon Cognito signed in user. If you open the Network tab for Chrome Developer Tools and select the /Prod/cart request, you will see that your request now includes an Authorization header:
Now journey back to the AWS API Gateway console to see how API Gateway processes the Authorization header. Open the aws-serverless-shopping-cart-shoppingcart-service and select Authorizers from the navigation menu. You will see that a Cognito user pool authorizer has been setup for the API and is configured to use the Authorization header.
If you want to see this in action, click the Test button and copy the Authorization header value from your request in the Network Tab of Chrome Developer Tools. If you did this correctly, you should see the decoded JSON Web Token for your request:
Importantly, you can see that you can identify the users' Cognito username (
cognito:username) and Cognito user ID (sub) from the token.The authorizer is being used in the POST /checkout method. This means that checkout is accessible only to registered users:
Select the GET method for the /cart resource. The GET method for cart doesn’t require authorization. However, API Gateway will still include your Cognito user information when it sends your request to the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda Function:
Click on the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> link to open it.
Now click on the function name so you can navigate to the Code in the AWS Lambda console:
You will now annotate the AWS Lambda Function to include the username as an annotation. Annotations allow us to group and search traces in AWS X-Ray. The AWS X-Ray SDK includes a setUser() function to annotate a segment with the user ID. However, this can’t be used with AWS Lambda because segment documents created by AWS Lambda are immutable and can’t be annotated. Instead, you can create a sub segment in your AWS Lambda function and annotate the subsegment with the user ID.
In your cloned repository on your local computer, update your backend/shopping-cart-service/list_cart.py file to the following:
In this file, you implement the following changes:
- On line 5, you imported the xray_recorder function from the Python AWS X-Ray SDK.
- On line 9, you imported the new get_username function in shared.py that you will add in the next step.
- On line 24, you create a new subsegment called annotations using the X-Ray SDK for Python as soon as your Lambda function handler begins execution.
- On lines 28-32, you create an annotation called generated_cart to indicate whether or not the cart ID was retrieved from the cookie cartId set by the front end user interface.
- On lines 37-38 and 42, you retrieve the username and create an annotation called username which is set to the Cognito username if an authorized login is detected or set to the value 'anonymous'.
- On line 65, you closed the annotations subsegment you created on line 24.
Next, update the backend/layers/shared.py file to the following:
In this file you make the following changes:
- On lines 62-74, you create a new function named get_username that takes a Cognito JSON Web Token, validates it, and retrieves the username from the token.
After you make the changes, you can deploy them by executing make backend. You may need to Control-C to close your locally hosted frontend interface first.
Once the backend completes deployment of your changes, start the frontend web server again by executing make frontend-serve.
Now return to the locally hosted user interface at http://localhost:8080. This will generate a request to list the shopping cart contents from your deployed aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda function. Go ahead and do a little shopping!
Next, return to the AWS X-Ray console and select Traces under the X-Ray traces menu. You should now be able to filter your traces by username. You will do this next.
AWS X-Ray allows you to build queries to quickly zero in on traces. First click Run Query without any query specified. Next, select the Refine query by drop down menu and choose Node. Next, type list to narrow the node list and then select the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda function and select Add to query. Now click Run Query.
Remove the "list" text you entered earlier from the filter text box. Next, select the Refine query by drop down menu and select username under User annotations and click Run Query:
If you don't see any results, change the timeframe from to a longer period such as 1h and make sure the text box for filtering data underneath the Refine query by dropdown is empty.
From here, select your username and click Add to query:
You will notice that X-Ray is providing you with the aggregated response times and requests by username.
Next, click Run query to execute the query you just built:
Scroll down to the Traces section and select the first one.
In the Segments Timeline, select the annotations subsegment in the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> node. Select the Annotations tab and you should see the annotations that you implemented earlier:
At this point, you have the ability to filter your X-Ray traces by username, see user specific response times, and troubleshoot user specific issues. Next, you will group your traces by registered and anonymous users so that you can quickly view these groups of users on the AWS X-Ray service map, and create alarms for these user groups.
You have now implemented annotations that enable you to see X-Ray traces by username. Next, you will create groups for registered and anonymous users so you can distinctly view them within X-Ray and Amazon CloudWatch.
Proceed to the CloudWatch Settings section and select the View settings for Groups in the Traces tab.
Next, create a group called RegisteredUsers and paste the following query. Note, the carriage return is needed::
The Annotation.username CONTAINS "" part of the query will only return traces where the username annotation exists. The second part of the query returns all users who are not anonymous.
Also select the Enable Insights option and Enable notifications. You’ll cover these features in the next step. Your selections should look like this:
Click Create group to complete.
Next, create another group called AnonymousUsers and paste the following query. Note, the carriage return is needed:
This query will only return traces where the username annotation is set to anonymous. Also Enable Insights option and Enable notifications.
Now that you have created your groups, proceed to the Service map in the CloudWatch console to see them in action! It’s probably been some time since you used the frontend user interface, so select 3h or another suitable timeframe until you see your service map show some data:
Now, for the moment of truth. Select the Filter by X-Ray group box and select RegisteredUsers.
Wait... what? Shouldn’t there be some data here? Is the query incorrect?
No! AWS X-Ray Groups only groups data for traces that are created after your group is created. That’s why it's a good idea to plan and set up your groups before you need to use them. Let’s confirm this assertion. Go back to the frontend interface for your sample application and login as the registered user you created in step 3.
Next, generate some traffic by adding some products to the cart and refreshing the web page. Make sure you add some half-eaten cake to your cart, you’ll leverage this later and your future self will thank you. Maybe also add some leftover cheese? Whatever your fancy! Now, log out and generate some more traffic as an anonymous user. Again, make sure you add some half eaten cake to your cart. Studies show that anonymous users like half-eaten cake (no, not really!).
Now that you have generated some data, go back to the CloudWatch console and reload the Service map page. Select the RegisteredUsers group again and this time you should see something different similar to this:
That’s better! Now your service map allows you to focus on registered users.
It is important to note that you only implemented the username annotation in the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda function. Therefore, you will only get traces that include this function in these groups. You can update your other functions to include this annotation so they will also be included in this group.
When you are trying to decide which groups you should create, consider your user segmentation as well as functional and performance test cases that you may have already defined for your application. Since grouping and querying in X-Ray is on a trace basis, your query only needs to match one annotation in order to return the end-to-end interaction of all AWS X-Ray instrumented services that participated in the match. For example, the query Annotation.username = "anonymous" presents the following visualization:
You may need to choose a suitable timeframe that includes traces from requests you made when you weren't logged into the application.
Although you only implemented the annotation on your aws-serverless-shopping-cart-shop-ListCartFunction-<hash> Lambda function, you still see all the AWS X-Ray instrumented services in the end-to-end request. This is valuable because you can’t update and annotate the segments created by API Gateway or DynamoDB since their instrumentation and integration with AWS X-Ray is managed by AWS. However, you can observe and monitor them with X-Ray as a part of an end-to-end trace by annotating a service that participates in the flow, such as the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda Function.
Let’s get back to the original story. You have a registered user who can’t view their shopping cart when they login. It turns out, that user is a future version of you! Your shopping cart is working you say? Well, it’s time to break it! It turns out that selling half-eaten cake may not be the best choice for an online store. You need to remove it. Open backend/product-mock-service/product_list.json and remove the following JSON object:
While you’re there, you might as well remove the leftover cheese. That’s too bad, since it was such a good seller!
These updates will remove the products from the product list on the website. However, you still need to do something about customers that may have already added the items to their shopping cart. Being the code hero that you are, you decide to update the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> so this product can no longer be included in carts dynamically. Make the following change to backend/shopping-cart-service/list_cart.py:
Change:
to:
Now, deploy your changes to the backend with the command: make backend. You may need to close your local frontend web server with a Control-C.
Once your backend has finished deploying, restart the frontend web server with the command: make frontend-serve.
Load up the frontend interface in your web browser: http://localhost:8080
This time you should just get a spin, spin, spin. Something's wrong. Let’s start to investigate! Proceed to the Service map in X-Ray. The service map should indicate that there is an issue with one of our Lambda functions:
Click on the ApiGateway Stage node outlined partially red. You can see that this node is experiencing an issue. You can also see that the Lambda function it is calling is experiencing errors from the node graph. Click on the checkbox in the Metrics tab indicating the % Faults for the node.
Next, click on the View Filtered traces button to filter traces to only see the traces experiencing Faults. Scroll down to the Traces list and open the first one. The number of traces you have will depend on the number of times you tried to refresh. Scroll down and take a look at the Segments Timeline:
You can see here that API Gateway returned a 502 error to the client. A 500 series response is classified as a fault by AWS X-Ray. You can also see further upstream that your AWS Lambda function that API Gateway called experienced an error but the function responded with a 200 status code. In this case, the invocation of your AWS Lambda function was successful but the lambda execution resulted in an error. If AWS Lambda executes your lambda function but your function code produces an error, it will include the X-Amz-Function-Error header in the response. In the Overview tab, you will see that Errors is true but Fault is false.
Next, select the the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS::Lambda::Function node in the Segment Timeline and then select the first segment:
This is the first segment in the Lambda function execution node and the overview indicates that there was an error. Next, scroll down further so you can see the logs and click on the Exceptions tab in the Segment details drawer window. AWS X-Ray has captured an exception that occurred in the Lambda function as well as the stack trace in the Exceptions tab. The exception trace points to the update that you just made to filter out the Half Eaten Cake product SKU as the culprit. AWS X-Ray has provided you with the stack trace and the line number in your code where the issue occurred.
Before you go fix this issue, take a closer look at the logs provided by AWS X-Ray. One thing to note here is that the logs displayed in X-Ray don’t contain the exception as you would expect it to. You will notice in the screenshot above that the log statements that are included in X-Ray include the X-Ray trace ID. AWS X-Ray is using CloudWatch Logs Insights to retrieve the logs for the request. Click on the View in CloudWatch Logs Insights. You should see a query similar to this:
You can see that the query is looking for either the API Gateway request ID, AWS Lambda Function request id, or the AWS X-Ray trace ID in the message. When you log messages with the aws_lambda_powertools logger and you have also instrumented your function to use AWS X-Ray, it will automatically enrich the log messages with the AWS X-Ray trace ID. However, the exception that was thrown was not logged using the logger and therefore didn’t include the X-Ray trace ID. You can confirm this by proceeding to the logs for the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda function and viewing the exception message in the logs:
Because the exception abruptly ended your AWS Lambda Function execution, you weren’t able to end your annotations subsegment with your annotations and therefore your trace didn’t include annotations. You will also notice that upstream requests such as your call to DynamoDB didn’t show up in your trace as well.
Next, you will fix the error and also improve exception handling so that your X-Ray traces aren’t negatively impacted by exceptions. Proceed to the backend/shopping-cart-service/list_cart.py file. The exception is occuring because of this code block:
The exception was caused by incorrectly using the
remove() Python list function on the individual product dictionary. Next, you will fix this error and also improve the exception handling for the function by wrapping the execution in a try / except block:Replace the backend/shopping-cart-service/list_cart.py file contents with:
By adding exception handling, you can gracefully close the subsegment you created for annotations. You can also log the error message using the aws_lambda_powertools logger which includes the X-Ray trace ID resulting in the message being displayed in the X-Ray console.
After you make the changes, deploy them by running the command: make backend. You may need to exit the frontend web server using Control-C.
Once your deployment is complete, startup the frontend web server with the command: make frontend-serve.
Now, head back to the frontend user interface with your browser: http://localhost:8080.
This time, your experience should be better! Make sure you are logged in and check your shopping cart by clicking on the shopping cart icon:
You will notice that you no longer have the half-eaten cake in your shopping cart (although the leftover cheese is still there!). You’ll also notice that you are no longer selling half-eaten cake or left over cheese (although you are selling half-eaten lettuce and leftover ham!). The important thing is that you have restored the application to a healthy state for any users who had half-eaten cake in their shopping cart. Let’s consider this for a moment. The exception and error was only impacting users who had half-eaten cake in their shopping cart:
You can reasonably conclude that not too many people were considering half-eaten cake and had it in their shopping cart. Errors effecting a small or unique segment of the user population can be much harder to pinpoint. AWS X-Ray can help you by providing you with tools to query on annotations that classify user requests (e.g. query by username).
Refresh your browser a few times to generate some requests. Now proceed back to the service map option under the X-Ray menu in the CloudWatch console. This time your service map should look healthy:
You can also use the Filter by X-Ray group and select Anonymous and RegisteredUsers to see the user requests by user population. Since you added exception handling to close your annotations' subsegment, your user grouping should now also display exceptions and errors that occur during function execution.
Our story began with a cloud support engineer receiving a call from a customer saying the website wasn’t working for them. This means that you didn’t detect the issue until someone took the effort to call in and point out the problem. Ideally, you should be alerted when an issue begins to occur. There are many ways to do this by integrating with CloudWatch. You can use CloudWatch Synthetics Canaries and alarm on the SuccessPercent published by a canary. You can also configure alarms on the individual services themselves in your architecture. For example, you can alarm on the Errors or Throttles metric for your AWS Lambda functions. For this tutorial, you are going to focus on how you can alarm on X-Ray detected faults, errors, or response times.
Let’s focus on the RegisteredUsers group. Proceed to the AWS X-Ray groups in the CloudWatch settings menu in the Traces tab.
Click the Create group button to create a new group. For the query, we will enter:
Call this group SlowRegisteredUsers and save the group. Now, slow down the aws-serverless-shopping-cart-shop-ListCartFunction-<hash> AWS Lambda function by updating the backend/shopping-cart-service/list_cart.py file and introducing a sleep. Replace the contents of the file with this code:
After you make the changes, deploy them with the command: make backend
Now, start the frontend web server on your local machine if it isn’t started already: make frontend-serve.
Next, proceed to the web interface at http://localhost:8080. Make sure you are logged in and refresh the page a few times. The page should take a bit longer to load because of your 3-second sleep.
Next, proceed to the AWS X-Ray Service map in the CloudWatch console and select the SlowRegisteredUsers X-Ray group:
You can now see all your traces that took longer than 3 seconds. You could do some further investigation by looking at these traces by username to see which specific users are impacted. AWS X-Ray has an extensive filter expression library for you to quickly identify traces that are experiencing faults, errors, and slow response times.
AWS X-Ray publishes the Approximate number of traces for each X-Ray group every minute as an Amazon CloudWatch metric named ApproximateTraceCount in the X-Ray/Group Metrics namespace.
Proceed to the CloudWatch Metrics console and navigate to the X-Ray/Group Metrics namespace to see the metrics for your Anonymous, RegisteredUsers, and SlowRegisteredUsers AWS X-Ray groups you created earlier. Check the boxes for these groups and select an appropriate timeframe to display data:
Next, proceed to the Graphed metrics tab. The default statistic for the graphed metrics is average. Change this to Maximum for both graphed metrics:
Next, change the period from 5 minutes to 1 minute for all graphed metrics. AWS X-Ray publishes metric data every minute for the ApproximateTraceCount metric. It is important to remember that AWS X-Ray publishes traces based on a sampling of requests and the number presented here may not be the actual number of requests for each group. You can adjust the sampling rate for AWS X-Ray based on the needs of your application.
Now, you will create an alarm for the SlowRegisteredUsers metric. Click the alarm bell icon next to the metric:
Set the conditons for the alarm to Greater than 0 and click Next:
For notification, select Create new topic and enter a topic name (e.g. SlowRegisteredUsersNotification) and email address for notification and then click Create topic and click Next:
Enter a name for the alarm (e.g. SlowRegisteredUsersAlarm) and click Next. Finally, Review your choices and click Create alarm.
Now, check your email at the address you indicated to confirm your subscription for notifications.
At this point, any time your registered users experience a response time of greater than 3s in a one-minute period, you will receive an email notification.
Let’s test this out to see if it is working. Return to the frontend web interface at http://localhost:8080 and reload the page a few times. Within a minute or so, you should receive an email notification for your alarm:
Congratulations! You can now alarm and troubleshoot your key user groups using AWS X-Ray!
Now that you're finished, you can clean up everything you created in this tutorial:
- Delete the Anonymous, RegisteredUsers, and SlowRegisteredUsers AWS X-Ray groups you created in Step 4.
- Delete the SlowRegisteredUsersNotification SNS topic you created in Step 5.
- Delete the SlowRegisteredUsersAlarm CloudWatch Alarm you created in Step 5.
In this tutorial, you learned how to use AWS X-Ray to trace a serverless application architecture. You learned how to segment and identify your users in AWS Lambda using annotations, and how you can use AWS X-Ray groups to segment and identify user requests. AWS X-Ray can help you pinpoint and troubleshoot issues with your application, and it provides a valuable connected graph of your user requests so you can optimize your user experiences and improve application quality. Want to learn more? Check out the Scorekeep sample application instrumented with the AWS X-Ray Java SDK to see how X-ray works with Java applications in the AWS X-Ray documentation!
If you enjoyed this tutorial, found any issues, or have feedback for us, please send it our way!
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.