Instrumenting applications on AWS ECS and AWS Fargate with AWS X-Ray

This article describes the process to create the infrastructure to host an application on AWS ECS using the AWS Fargate, the serverless compute for containers from AWS, with the AWS X-Ray collector as a sidecar container, in the same ECS task running the main application.

The infrastructure will be built using the AWS Cloud Development Kit (CDK).

This article also describes how to prepare a NestJS application to be instrumented by the X-Ray, including a very useful tip to create a distributed tracing, in case of multiple services communicating using an asynchronous mechanism, such as AWS SNS topics and AWS SQS queues.

The problem:

The job to instrument an application running on AWS ECS with AWS Fargate has some challenges:

The X-Ray daemon, responsible to collect the traces from the application, needs to be deployed out of the application Docker image itself, to do not compete with the application in terms of memory and CPU usage;
Each microservice needs to have its own X-Ray collector, to do not overload a single collector in case of high traffic of segments being generated by all services.

The architecture:

The solution is to embed the AWS X-Ray daemon as a sidecar container, running in the same ECS Fargate task of the application, but with memory and CPU usage limits. In this way, the application and the X-Ray daemon will have their own resources and they will not compete on each other.

The following diagram will be used in this article to describe the architecture of the proposed architecture.

Image not found

AWS X-Ray as a sidecar container using AWS Fargate

Using the X-Ray as a sidecar container, it's possible to instrument the application and also the calls to other AWS resources, such as DynamoDB tables, made with the AWS SDK, which is something very useful.

The following sections will describe how to create the sidecar container to host the AWS X-Ray daemon.

Creating the ECS Fargate task definition:

First, the ECS Fargate task definition must created to define the task that will be executed:

1
2
3
4
5
const taskDefinition = new ecs.FargateTaskDefinition(this, "TaskDefinition", {
   cpu: 512,
   memoryLimitMiB: 1024,
   family: "products-service",
})

The amount of memory and CPU defined here will be used by the application and the X-Ray sidecar containers.

Adding the application's container:

Now it's possible to add a container to the task definition. In this case, with the Docker image of the application, with memory and CPU limits, as defined in the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
taskDefinition.addContainer("ProductsServiceContainer", {
   image: ecs.ContainerImage.fromEcrRepository(props.repository, "1.0.0"),
   containerName: "ProductsService",
   logging: logDriver,         
   portMappings: [{
      containerPort: 8080,
      protocol: ecs.Protocol.TCP          
   }],         
   cpu: 384,
   memoryLimitMiB: 896,
   environment: {
      PRODUCTS_DDB: productsDdb.tableName,
      AWS_XRAY_DAEMON_ADDRESS: "0.0.0.0:2000",
      AWS_XRAY_CONTEXT_MISSING: "IGNORE_ERROR",
      AWS_XRAY_TRACING_NAME: "products-service"           
   }      
})

Note that there are some environment variables defined specifically to the X-Ray being passed to the application. The most important variable here is the X-Ray daemon address, with the address pointing to the 2000 UDP port of the task itself.

Adding the X-Ray daemon as a sidecar container:

In the same ECS Fargate task definition, it's time to add a new container, with the X-Ray daemon. This is the so called sidecar container, because it's not the container of the application running in the ECS Fargate task, but the X-Ray collector.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
taskDefinition.addContainer('xray', {
   image: ecs.ContainerImage.fromRegistry('public.ecr.aws/xray/aws-xray-daemon:latest'),
   containerName: "XRayProductsService",
   logging: ecs.LogDriver.awsLogs({
      logGroup: new logs.LogGroup(this, "XRayLogGroup", {
         logGroupName: "XRayProductsService",
         removalPolicy: cdk.RemovalPolicy.DESTROY,
         retention: logs.RetentionDays.ONE_MONTH
      }),
      streamPrefix: "XRayProductsService",
   }),
   cpu: 128,
   memoryLimitMiB: 128,
   portMappings: [{
      containerPort: 2000,
      protocol: ecs.Protocol.UDP,
   }]
})

Note that in the container definition, the memory and the CPU allocated to this sidecar container will not compete with the resources allocated to the container of the application.

Also, note that here the port mappings has the 2000 UDP port defined, the same used to specify to the application where is the X-Ray collector.

Adding the permission to the application to write X-Ray traces and segments:

To finish the infrastructure with the AWS Cloud Development kit, it's necessary to add the managed policy to let the ECS Fargate task to write traces and segments to the the AWS X-Ray:

1
2
3
4
5
import * as iam from "aws-cdk-lib/aws-iam"

taskDefinition.taskRole.addManagedPolicy(
   iam.ManagedPolicy.fromAwsManagedPolicyName('AWSXrayWriteOnlyAccess')
);

The following sections will describe how to prepare a NestJS to be instrumented using the AWS X-Ray SDK. This application can be executed in a ECS Fargate task as described in this article.

Preparing the NestJS application to be instrumented by the X-Ray:

The first thing to do in the NestJS application is to add the AWS X-Ray SDK to the project:

1
$ npm install aws-xray-sdk

Now, in the main file of the project, configure the AWS X-Ray library:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import * as AWSXray from "aws-xray-sdk";

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  app.use(AWSXray.express.openSegment(process.env.AWS_XRAY_TRACING_NAME));
  AWSXray.config([AWSXray.plugins.ECSPlugin]);  
  AWSXray.captureHTTPsGlobal(require('http'));
  app.use(AWSXray.express.closeSegment());
    
  await app.listen(8080);
}
bootstrap();

With this, it's possible to instrument clients from the AWS SDK, such as the DynamoDB client. In this way, the X-Ray traces will have specific segments describing the execution time of each operation agains the DynamoDB table, as well the operation that was performed with additional informations.

1
2
3
4
5
6
7
import { captureAWSv3Client } from 'aws-xray-sdk';

constructor() {
    this.tableName = process.env.PRODUCTS_DDB
    const ddbClient = captureAWSv3Client(new DynamoDBClient({ }));
    this.ddbDocClient = DynamoDBDocumentClient.from(ddbClient)
}

Creating a distributed tracing mechanism:

In case of a microservice architecture, with multiples services communication through an asynchronous mechanism with SNS topics and SQS queues, it's very useful to continue generating segments, from the service consuming some message, to the original X-Ray trace created by the application responsible to publish the message.

To do this, the X-Ray traceId must be include in the published message:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
private async sendEvent(data: string): Promise<string> {
        const segment = AWSXRay.getSegment() as AWSXRay.Segment

        const command = new PublishCommand({
            Message: data,
            TopicArn: this.topicArn,
            MessageAttributes: {
                traceId: {
                    DataType: "String",
                    StringValue: segment.trace_id
                },
            }
        })

        const result = await this.snsClient.send(command)

        return result.MessageId
    }
}

Then, in the consumer, the X-Ray traceId can be restored and new segments can be attached to the original trace:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
async handleMessage(message: AWS.SQS.Message): Promise<void> {
  const startTime = new Date().getTime() / 1000
  const body = JSON.parse(message.Body) as SNSMessage;

  const traceId = body.MessageAttributes['traceId'].Value

  const segment = new Segment(process.env.AWS_XRAY_TRACING_NAME, traceId)

  segment.origin = "AWS::ECS::Container"
  segment.start_time = startTime
  segment.trace_id = traceId
  segment.addPluginData({
     operation: "HandleProductEvent",
     region: process.env.AWS_REGION,
     queue_url: process.env.AWS_SQS_QUEUE_PRODUCT_EVENTS_URL
  })

  const ns = getNamespace();
  ns.run(async () => {
     setSegment(segment)

     ///Code to process the message, let's say, to write some 
     //information to a DDB table
     segment.close()      
  })
}

The most important here is that other calls to AWS services can also be instrumented, giving to the developer the whole picture of a request, since the first time a service receive, until the end of the process.

Conclusion:

The most important part of this article is the concept of having the sidecar container to host the X-Ray daemon, in the same ECS Fargate task of the application, eliminating any resource conflict. Everything built with the AWS CDK.

Select your cookie preferences

Site Terms, Privacy, and more.