Following the path of Architecture as Code

Introduction

Architecture as Code (AaC) is gaining momentum as a powerful paradigm for building cloud applications. AaC allows us to model our cloud architecture using familiar programming languages and tools, focusing on the intention of a solution ideally described with a pattern language.

Time-tested patterns like Enterprise-Integration-Patterns provide a structured vocabulary for describing the flow of messages and events within distributed systems. By implementing EIPs we establish a ubiquitous language that transcends specific technologies, enabling clear communication and collaboration across teams.

In this blog post, I want to provide a practical example of my attempt to use AaC to build modern cloud applications on AWS. I leverage the AWS CDK (Cloud Development Kit) as my Infrastructure-as-Code tool and demonstrate how EIPs are embodied in custom constructs for seamless integration flows.

Building a Ubiquitous Language

One of the most powerful aspects of EIPs is the ability to bridge communication gaps between technical and non-technical stakeholders. By employing a shared language, we can collaborate on system designs and ensure that solutions align with technical and business needs. In my specific use case, the EIPs and their implementations we'll discover are:

Enterprise Integration Pattern	AWS Service Implementation
A Message Bus performs as a Middleware between applications that enable them to work together using messaging.	Amazon EventBridge’s event bus serves as the foundation of our integration. It provides a central channel for routing events between producers and consumers.
A Message Filter eliminates undesired messages from a channel based on a set of criteria.	EventBridge Rules allow us to selectively route events based on specific criteria, ensuring that only relevant events reach downstream processing components.
Content-Enricher accesses external data sources to augment a message with missing information.	An AWS Lambda function acts as a Content-Enricher. It processes events, fetches additional details, and appends this data to the original event.
A Recipient List inspects incoming messages, determines the list of recipients, and forwards messages to all channels associated with the recipients in the list.	EventBridge Rules establish a Recipient List to route events to a defined set of targets, directing them to appropriate downstream services.

Next, let's take a closer look into a practical example of enriching events from a managed AWS service, where we'll demonstrate how these patterns come to life using custom CDK constructs.

A Practical Example: Enriching AWS Service Events

I want to add detailed information about a Transcription job from Amazon Transcribe to events emitted by the service, making downstream processing more streamlined and efficient. Typical use cases could be, to forward the generated transcription text for a human review.

Here is a sample event that you receive on the default event bus from Amazon Transcribe. It follows a typical event notification style containing the name of the Transcription job to be used by an event consumer to fetch additional details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
    "version": "0",
    "id": "event ID",
    "detail-type":"Transcribe Job State Change",
    "source": "aws.transcribe",
    "account": "111122223333",
    "time": "timestamp",
    "region": "us-west-2",
    "resources": [ ],
    "detail": {
          "TranscriptionJobName": "my-first-transcription-job",
          "TranscriptionJobStatus": "COMPLETED" (or "FAILED")
    }   
}

Solution Design

The following diagram visualizes my initial designed data flow:

Amazon Transcribe emits events about Transcription job state changes onto the default EventBridge event bus, acting as a Message Bus pattern implementation.
An EventBridge Rule filters relevant events from Amazon Transcribe and routes them to an AWS Lambda function, representing a sequence of a Message Filter and a Recipient List pattern.
The AWS Lambda function serves as the Content-Enricher pattern. It receives the initial event from Amazon Transcribe, fetches additional details about the Transcription job, and appends these details to the event.
The enriched event is sent to a custom EventBridge bus via a Lambda Destinations channel.

Solution Design Diagram visualizing the implementation of enriching events from AWS Managed Services — Enriching events form Amazon Transcribe

I liked the term flow hence I made this an important part of my implementation and ubiquitous language. I started defining an interface providing consumers with features to design integration flows.

1
2
3
4
5
export interface Flow {
  withFilter(filter: events.EventPattern): Flow;
  withEnricher(code: lambda.AssetCode, policies?: iam.PolicyStatement[]): Flow;
  withEventTarget(eventTarget: events.IEventBus): void;
}

AWS CDK Implementation

The Flow interface promotes a declarative approach to codify integrations. Consumers specify the desired interactions between components using a fluent syntax (withFilter(), withEnricher(), etc.). The flow for our example will then look like the following snippet. It seamlessly ties together the messaging concepts from our solution design using an expressive interface. I wanted to provide a clean interface for my integration logic while hiding service-specific implementation details and promoting a pattern-focused approach.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
source
  .flow("TranscribeContentEnrichment")
  .withFilter(
    source: ["aws.transcribe"],
    detailType: ["Transcribe Job State Change"],
  })
  .withEnricher(
    lambda.Code.fromAsset(
      path.join(__dirname, "enricher.lambda.ts"),
    ),
    [
      new iam.PolicyStatement({
        resources: ["*"],
        actions: ["transcribe:GetTranscriptionJob"],
      }),
    ],
  )
  .withEventTarget(target);

Data Flow

The concept of a Flow must adhere to service-specific implementation details. The EventBridge-specific implementation encapsulates the complexity of configuring EventBridge rules including filters and targets, and integrating the Content-Enricher Lambda function. This allows a user to focus on the design of a flow itself rather than cracking service-specific implementation details.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
export interface EventBridgeFlowProps {
  eventSource: events.IEventBus;
}

export class EventBridgeFlow extends Construct implements Flow {
  readonly rule: events.Rule;
  enricher?: lambda.Function;

  constructor(scope: Construct, id: string, props: EventBridgeFlowProps) {
    super(scope, id);
    this.rule = new events.Rule(scope, "Rule", {
      eventBus: props.eventSource,
    });
  }

  withFilter(filter: events.EventPattern): EventBridgeFlow {
    this.rule.addEventPattern(filter);
    return this;
  }

  withEnricher(code: lambda.AssetCode, policies?: iam.PolicyStatement[]) {
    this.enricher = new lambda.Function(this.rule, "Enricher", {
      handler: "index.lambda",
      code,
      runtime: new lambda.Runtime(LAMBDA_RUNTIME, lambda.RuntimeFamily.NODEJS),
    });

    policies?.forEach((p) => this.enricher?.addToRolePolicy(p));

    this.rule.addTarget(new targets.LambdaFunction(this.enricher));
    return this;
  }

  withEventTarget(eventTarget: events.IEventBus) {
    if (this.enricher) {
      new lambda.EventInvokeConfig(this, "EnricherEventTarget", {
        function: this.enricher,
        onSuccess: new destinations.EventBridgeDestination(eventTarget),
      });
    } else {
      this.rule.addTarget(new targets.EventBus(eventTarget));
    }
    return this;
  }
}

MessageBus

The MessageBus construct is another pattern implementation, abstracting event bus creation and configuration. It further provides the flow(id: string) method, which acts as a factory method for defining a new integration flow specific to Amazon EventBridge.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
export interface MessageBusProps {
  readonly name: string;
}

export interface IMessageBus {
  readonly eventBus: events.IEventBus;
  flow(id: string): Flow;
}

export class MessageBus implements IMessageBus {
  eventBus: events.IEventBus;

  constructor(scope: Construct, id: string, props: MessageBusProps) {
    super(scope, id);

    this.eventBus = new events.EventBus(this, "CustomEventBus", {
      eventBusName: props.name,
    });
  }

  public flow(id: string): Flow {
    return new EventBridgeFlow(this, id, {
      eventSource: this.eventBus,
    });
  }
}

Content Enricher

The Lambda function just calls the Amazon Transcribe API to fetch details from a transcription job. By using Lambda Destinations, it is decoupled from any messaging or infrastructure logic. The AWS Lambda service itself takes care to route the response of our function to another event bus that we can model with a MessageBus construct.

1
2
3
4
5
6
7
8
9
10
11
12
const lambdaHandler = async (
  event: EventBridgeEvent<"Transcribe Job State Change", TranscribeJobStateChanged>,
): Promise<EventBridgeEvent<"Transcribe Job State Change", TranscriptionJob>> => {
  const transcriptionJob = await transcribeClient.getTranscriptionJob({
    TranscriptionJobName: event.detail.TranscriptionJobName,
  });

  return {
    ...event,
    detail: { ...transcriptionJob.TranscriptionJob },
  };
};

Messaging-related implementations are handled by our Flow implementation as you see in the following snippets as part of our Flow implementation. A Flow that contains a content-enricher implemented as an AWS Lambda function, creates an EventInvokeConfig Lambda Destination implementation. If the flow does not contain a content-enricher, it adds the event target to the underlying rule implementation.

1
2
3
4
5
6
7
8
9
10
11
withEventTarget(eventTarget: events.IEventBus) {
    if (this.enricher) {
      new lambda.EventInvokeConfig(this, "EnricherEventTarget", {
        function: this.enricher,
        onSuccess: new destinations.EventBridgeDestination(eventTarget),
      });
    } else {
      this.rule.addTarget(new targets.EventBus(eventTarget));
    }
    return this;
}

For now, the method signature of withEventTarget restricts us from using anything else than an Amazon EventBridge event bus as a target. Multiple other valid targets like AWS Step Functions, Amazon SQS, or Amazon SNS can be modeled at a later stage supporting more integration use cases.

Running our configured flow, we now see that the enriched event contains all the details of a Transcription job enveloped by the AWS Lambda Destination integration as part of the response payload.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
{
    "version": "0",
    "id": "a7dbf591-191f-0820-f780-d9dea822c9fc",
    "detail-type": "Lambda Function Invocation Result - Success",
    "source": "lambda",
    "detail": {
        "responsePayload": {
            "version": "0",
            "id": "....",
            "detail-type": "Transcribe Job State Change",
            "source": "aws.transcribe",
            "account": "...",
            "time": "2024-03-08T20:46:35Z",
            "region": "eu-central-1",
            "resources": [],
            "detail": {
                "CompletionTime": "2024-03-08T20:46:35.342Z",
                "CreationTime": "2024-03-08T20:45:57.014Z",
                "LanguageCode": "en-GB",
                "Media": {
                    "MediaFileUri": "s3://.../videos/3811065_DAL_5387406.mp4"
                },
                "MediaFormat": "mp4",
                "MediaSampleRateHertz": 48000,
                "ModelSettings": {
                    "LanguageModelName": "en-gb"
                },
                "Settings": {
                    "ChannelIdentification": false,
                    "ShowAlternatives": false,
                    "VocabularyName": "95f6b2b4-d779-44f9-9f76-25fe3e69c7bf"
                },
                "StartTime": "2024-03-08T20:45:57.049Z",
                "Subtitles": {
                    "Formats": [
                        "vtt"
                    ],
                    "SubtitleFileUris": [
                        "https://.../transcriptions/95f6b2b4-d779-44f9-9f76-25fe3e69c7bf/transcript-1.vtt"
                    ]
                },
                "Transcript": {
                    "TranscriptFileUri": "https://.../transcriptions/95f6b2b4-d779-44f9-9f76-25fe3e69c7bf/transcript-1.json"
                },
                "TranscriptionJobName": "95f6b2b4-d779-44f9-9f76-25fe3e69c7bf-1",
                "TranscriptionJobStatus": "COMPLETED"
            }
        }
    }
}

Conclusion

Throughout this blog post, you explored how I used the AWS CDK to implement my Architecture as Code. Three takeaways from my exploration:

Ubiquitous Language: EIPs provide a common vocabulary to describe integrations, bridging communication gaps, and promoting collaboration.
Finding good abstraction is hard: Only thought-through L3 constructs raise the level of abstraction when defining cloud architectures. They embody patterns, encapsulate AWS-specific details, and promote a declarative, configuration-focused approach. Finding the right level of abstraction when defining infrastructure with code is not an easy task. Adopting AaC requires fluency in programming, infrastructure as well as integration concepts. That is a learning curve of its own.
AWS CDK for the win: The expressiveness of the AWS CDK and the fact that you can model your infrastructure as real code opens the world for Architecture as Code. You can switch perspectives by communicating intentions instead of service selections.

I encourage you to give it a try in your projects. You can find the code examples for the Transcribe enrichment pipeline on GitHub as part of a general solution I am working on. To help you navigate the code, here are the direct links:

How are you using Architecture as Code in your AWS projects? Did I find a good abstraction for my integration concerns? Let me know and share your experiences in the comments or connect with me on LinkedIn.

Let's continue the conversation!

Site Terms, Privacy, and more.