AWS | Community | Secure connectivity patterns for Amazon MSK Serverless

Amazon MSK Serverless is a cluster type of Amazon Managed Streaming for Apache Kafka (Amazon MSK) that makes it straightforward for you to run Apache Kafka without having to manage and scale cluster capacity.

In a multi-account environment, Kafka producers and consumers can exist within the same VPC—however, they are often located in different VPCs, sometimes in a different account, or even in multiple different accounts. In some cases, connectivity can also be required across multiple regions.

Earlier this year I wrote a blog post on Secure connectivity patterns for Amazon MSK Serverless cross-account access. The blog covered multiple solution approaches that address the MSK Serverless cross-VPC and cross-account access connectivity options, and discussed the advantages and limitations of each approach. Today I am adding an extension to the solution that addresses multi-account, multi-region, any-to-any requirements. To be more specific, it builds on top of the third option in the blog, the option titled "Resolver rules and AWS Resource Access Manager" and expand this solution to cover any-to-any multi-region deployments.

The situation I am solving for here is when you have a centralized VPC per region that hosts a number of MSK Serverless clusters, let's call it an MSK Serverless centralized regional service VPC. We can have this same setup again repeated in multiple regions with every region having its own single and centralized MSK Serverless service VPC. In addition, you may have clients in any region to produce/consume messages of Kafka topics of both local and remote regions' clusters (any-to-any requirement). The approach can also be expanded to provide connectivity to Kafka clients on premises through private connectivity and hybrid DNS.

The solution will expand on the use of the resolver rules and resolver endpoints described in the initial blog referenced above. The diagram below has a visual representation of the solution that I will describe in details here.

Using Resolver Rules and Endpoints for MSK Serverless Cross Region Access — MSK Serverless Cross Region Access

In the diagram above I am using us-east-2 and us-west-2 as two example regions, you can have as many regions as you would like. In each region we have a centralized MSK Serverless service VPC with multiple clusters each. Clients are in multiple VPCs, within the same account, different accounts, and also some in the same and different regions. Connectivity can be extended to on-premises client as well with hybrid DNS and private connectivity (Direct Connect or VPN options)

Solution:

We are going to use the technique described in option three in the blog post, the "Resolver rules and AWS Resource Access Manager" option. We will repeat the setup in every region we have MSK Serverless Clusters. This will cover connectivity within each region but will run short on cross regional access.
Each remote region will have its own clusters that are centralized in one VPC per remote region, with a similar configuration as we configured is the step before for the local region. We will also create resolver endpoints and rules in the same fashion as we did in the local region.
For cross region access we will add a resolver rule in each local region, one for each of the remote region added. The resolver rule will use the same outbound resolver endpoint in the local region, no need to add a new one, and will forward name resolution of the foreign region MSK serverless domain to the Inbound resolver endpoint IPs of the remote region, located within the remote region MSK Serverless centralized VPC. i.e. if the remote region is us-west-2, at the local region we will forward name resolution for *.kafka-serverless.us-``east-2``.amazonaws.com to the Inbound resolver endpoint IPs of the remote region, us-west-2, within the MSK Serverless VPC.
The same logic can be applied in a hybrid DNS situation where you have on premises clients coming through Direct Connect or VPN to access your MSK clusters across all regions. We will just need to add to the on premises DNS a forwarding rule per region directing kafka.serverless.region domain to the Inbound resolver endpoint for that specific region within the MSK Serverless VPC. i.e.

*.kafka-serverless.us-``east-2``.amazonaws.com point to the inbound resolver endpoint of useast-2 within the MSK Serverless VPC, and
*.kafka-serverless.us-``west-2``.amazonaws.com point to the inbound resolver endpoint of us-west-2

Solution limitations:

If you have multiple environments. i.e. Prod/Dev/Test, each environment DNS resolution and clusters will need to remain separate. i.e. Dev / Test/ Prod
For each environment, and per region, you will need to create a centralized MSK service VPC that combines all your clusters within that region. You can have multiple clusters, but you can not have more than one MSK Servelress service VPC within the same region.
Clients can be in same account, different accounts, different VPCs , and in different regions as needed, or even on premises. That is as long as we maintain on a per environment and region bases a centralized VPC with all the MSK Sererless clusters and have private connectivity (VPC peering, Transit Gateways, VPN, and/or Direct Connect)

Conclusion:

We are able to accommodate any to any connectivity, including on-premises , cross VPC, cross account, and cross region, to multiple MSK Serverless clusters located in multiple regions, so long as, on a per region bases, all clusters within the same region are located in a single centralized VPC. The setup can be replicated per environment, i.e. Prod/Dev/Test with each with its own separate DNS resolution setup.

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.

Secure connectivity patterns for Amazon MSK Serverless - Cross account cross region setup

Comments