Leveraging on-premises APIs in cloud-based event-driven architectures
Your company owns important, and sometimes critical, private APIs that reside on-premises and you would like to leverage them in workloads deployed on AWS. More precisely, you would like to make them part of your event-driven architecture, triggered by events or part of complex workflows. This blog post explains how you can leverage new capabilities from Amazon EventBridge and AWS Step Functions to easily trigger/call on-premises APIs without compromising on security.
Published Apr 7, 2025
Last Modified Apr 11, 2025
- You’re building an event-driven architecture on AWS, leveraging services like Step Functions to build complex workflows and orchestrate different operations, and EventBridge for the event management and to improve the resilience of your system with asynchrony.
- You want to leverage some critical APIs which are not in the cloud but in your datacenter, by choice or because of regulatory or compliance reasons.
- You already have the connection between AWS and your datacenter, be it an AWS site-to-site VPN connection or AWS Direct Connect.
The following figure depicts this situation:
The main question is how to connect these two world securely and efficiently: How the state machine can retrieve information from the API? How events can flow through on premises APIs and notify internal applications?
The first approach, and the most straightforward solution until 2025, was to use a Lambda function in the private subnet of the VPC, as described in the following picture:
The state machine would use a standard Lambda invocation task and EventBridge would have a rule with a Lambda function as a target. And this Lambda would actually perform the call to the API. Being in the private subnet of the VPC connected to your datacenter ensures there is a route between the function and the internal API.
As a developper, this is probably the easiest way to achieve such a requirement. You can use any HTTP method to GET data or POST/PUT information, you control the code so you can do what you want.
There are some drawbacks with this solution:
- Writing custom code adds an additional burden for developers who could focus on business logic rather than integration logic.
- Adding code means you need to maintain it in the long run. Some bugs can be introduced, reducing the resilience of your system. And we speak about one Lambda function in this article but there would be potentially tens or hundreds of them to develop and maintain in your company.
- In some case, if there's no constant load on the API (and the Lambda function), it can introduce some latency due to the cold start of the function.
EventBridge and Step Functions both have features to call HTTPs endpoints:
- EventBridge API Destinations permit to use an HTTPs endpoint as a target for an EventBridge rule.
- Step Functions provides a dedicated state (HTTP task) to call HTTP endpoints.
Both are leveraging EventBridge connections to configure the connection (mainly the authentication aspects) to a target HTTP endpoint:
Until recently, connections were only possible to public endpoints (SaaS applications, public webhooks or "open" APIs). The ability to connect to private endpoints was announced at re:Invent 2024. With this feature, you can now call any private API (within a VPC) or internal API (on premises), without a Lambda in between.
This integration is possible thanks to Amazon VPC Lattice and two new components:
- Resource gateway, which acts as a secured entry point into the VPC to access a ressource: an EC2 instance, an RDS database, a DNS target or a simple IP address
- Resource configuration, which defines a resource (or group of resources), how and who can access it (them). Resource configuration is associated with a resource gateway through which it receives traffic.
When creating an EventBridge Connection, you can now select if you need to connect to a public or private endpoint. When choosing private, you can select the resource configuration. Concretely, this is how it looks like:
Coming back to our initial requirement, we would get the following big picture:
- The HTTP task in Step Functions and the rule with an API destination in EventBridge are leveraging an EventBridge Connection. Each one defines the target endpoint (e.g.
https://my-internal-api.company.com/customer) and HTTP method (e.g.GET) as well as eventual HTTP headers. - The EventBridge Connection defines the authentication mechanism (OAuth, Basic or API Key) for the target endpoint as well as the resource configuration to use for a private/internal endpoint.
- The resource configuration defines the target endpoint itself, generally an on-premise IP address or DNS name (e.g.
my-internal-api.company.com). Resource configuration is associated to a resource gateway. - The resource gateway "opens a door" to the VPC and allow ingress. It is linked to the chosen subnets (generally private) and is also protected by a security group to further protect your backend API. Note: You could stop here at the VPC with a private API deployed in a private subnet.
- The site-to-site VPN or Direct Connect connection establishes the connection between the AWS cloud (generally with a VPN Gateway or a Transit Gateway) and your datacenter (through a Customer Gateway).
- Finally the internal API that resides in your datacenter can be accessed via this "route".
The direct integration approach offers several advantages over the Lambda function solution:
- Operational Efficiency: There's no need to write and maintain custom code to interact with the on-premises APIs. Developers can focus on building core business functionality instead of integration logic.
- Resilience: By eliminating the custom code, the risk of introducing bugs is minimised.
- Performance: This solution leverages networking components rather than a compute node, which can help reduce latency by avoiding potential cold starts associated with Lambda functions.
The direct integration approach may appear more complex than the Lambda function solution, as it involves configuring networking components. Developers might find this setup more complex than writing a few lines of Python code. As a software engineer myself, I personally don't like to deal with network configuration and could understand the reluctancy to apply this pattern, as it may require different skillset.
This blog post will focus on the left part of the VPC. I assume the network setup (VPN / Direct Connect) is already implemented and you also have a VPC and private subnets connected to your datacenter, with proper route configuration. If this is what you're looking for, I encourage you to look at this documentation.
I will provide the terraform code to create the required resources. If you prefer CDK, have a look at this blog post.
We first need to define the security group used by the resource gateway. Here, we allow HTTP and HTTPS egress. Then we create the resource gateway itself, linked to the VPC, private subnets and security group created before.
Now comes the definition of the "resource" you want to communicate with, via the gateway created before, and thanks to a resource configuration. In the example above, we use a
dns_resource, specifying the domaine name (e.g. my-internal-api.company.com). If you don't have it configured, you can directly use an ip_resource (IP address) from your on-premise network.Nota bene: When using a domain name, this one must be public for VPC Lattice, the target API must use HTTPS and the certificate must be trusted/valid publicly (root CA).
We can now create the EventBridge connection. In this case, we have an API key protecting the internal API. This key has been made available in AWS Secrets Manager, it is retrieved in the first 3 lines and used in the
auth_parameters of the connection. The link to the resource configuration, and thus to the on-premises API, is done on lines 18-22. Once deployed, it can take a minute or two for the connection to be "Active" (more on connection states).And that's it! That's really just it. With this connection established, you can securely call your on-premises API from Step Functions (with an HTTP Task) or EventBridge (with API Destination).
Here is a very basic state machine definition that leverages the connection created before (line 13):
And the terraform code that provision it. You can notice how we define the endpoint, using the api domain name, and adding the target endpoint (line 7) :
Note that the state machine needs some specific permissions:
- The ability to invoke the HTTP endpoint (lines 8-18). Here we restrict this permission to our api domain name only, further enhancing the security.
- The ability to retrieve the EventBridge connection (lines 19-24) and leverage it.
- The ability to retrieve the secrets associated with the connection (lines 25-33). Note this is not the secret I mentioned above. This one is created and managed by EventBridge to store the authorization information.
Executing the Step Functions, you should retrieve the result of the API call to your on premises API:
You can find the full code source of this example on GitHub. A CDK version is also available. (Note that these links are referring to pull requests for serverlessland patterns and are currently in review).
In this blog post, we've seen how to securely connect to on premises API from Step Functions and EventBridge, leveraging VPC Lattice new components. It was already possible to have workloads in the cloud interacting with on premises API, through Lambda functions. But this new mechanism provides a more consistant, secure and resilient integration, by removing integration code. It enables companies to further improve and modernise their event-driven architecture by integrating seamlessly with public and private APIs.