AWS StepFunctions HTTP Endpoint demystified

Calling an API from AWS StepFunctions has for a long time required us to write a Lambda function to handle that. With the release of HTTP EndPoint there is no need for that anymore. In this post I'll go over the functionality and show how to use it.

Published Feb 14, 2024
I started to play around with StepFunctions new HTTP Endpoint state as soon as it was released. I found it to be a very nice addition and it worked really well, I was happy with it. At a recent re:Invent reCap I did a presentation and a demo on it, so I once again dove deep into it. I still like the functionality, not having to write a Lambda function just to call an API is very refreshing. However, there are some parts on it that I feel is just a bit misplaced. In this post I will go over the good and the bad parts with StepFunctions HTTP Endpoint State.

Solutions in this post

To demonstrate the HTTP Endpoint state we'll create two applications. The first will pull weather data from an open weather api, for a specific longitude and latitude. It will use the weather data to determine if it's shorts or not weather.
The second solution will use the Slack API to post a message to a Slack channel.

EventBridge Connection

To call an API using a HTTP Endpoint state a EventBridge Connection is required. This so we don't have to hard code or include any authorization parameters in our StateMachine.
Why to use EventBridge Connections
The thought behind this is good, but I feel it get a bit strange, why is there not a StepFunction Authorization resource? Why reuse what was created for EventBridge API Destinations? I would prefer if this was part of StepFunctions or was a standard SecretsManager secret.

EventBridge Connection Secrets

Speaking of secrets. When a EventBridge Connection is created there is automatically a SecretsManager secret created, with a format like this.
Secrets in SecretsManager
I understand that the secret might need to be in certain format that the connection understand, but I would have preferred that I supply the secret to the connection and that the connection just populate it with data. I have seen and got several questions from people, that has been using the HTTP Endpoint state, and then discovers this secret, wondering what it is and what created it.

Create EventBridge Connection

Let's create an EventBridge Connection for our open weather API. To do this, we need to navigate to the EventBridge console. Yes, it feels so strange. In the EventBridge Console there is also no "Connections" in the menu. Instead we have to navigate to "API Destinations" and select the "Connections" tab. Yes it is very well hidden. It feels that connections are not a first class citizen, I understand the placement when it was only for API Destinations. But it's not anymore, make it an option in the menu please!
EventBridge Connections
In our first solution we'll use an Open Weather API from yr.no a weather service in Norway. Click Create Connection and fill in the details. Supply a name and a description and then we need to set an authentication.
Create an EventBridge Connection
Again this is where things get highly opinionated from AWS. We must select one of the three authentication types, it's not possible to create a connection without an authentication. But the API I'm going to use is fully open and doesn't require an authentication. Sure, a fully open API might not be the best of practices, but it does exists. Anyway, let's select API Key and just fill in some bogus values. And for that matter, I think API Key should have been named "Use Header" instead, more on that later.
After the connection is created let's build the first solution.

Shorts or not

The first solution will call an open Weather API, fetch the current conditions at a location and decide if it's short weather or not. The state machine that will be used looks like this.
The StepFunction Graph
Call the API, use a ResultSelector to pick out the current conditions, use a Choice state to determine if it's shorts or not weather.
Create a StepFunction and add an HTTP Endpoint state to the graph, we configure it to use YR API, make sure you read the documentation if you are going to try this. The method will be GET and we use connection created before.
Call the weather API
Now, the API require that we'll supply a longitude and latitude as query parameters, we can't set this directly in the endpoint. Instead scroll down and expand "Advanced parameters" and set the query parameters there.
show the Query Param
The output from this state will be a json object with the headers and the body that the endpoint responds with, Headers and Body will be in separate objects, like this.
The data we get back from the API is an array with time series data, we are only interested in the current conditions, so here we can use a ResultSelector to get only what we need. In the output section add the following as ResultSelector.
Show the result Selector
Drag in a Choice State and two Pass state. Create a condition for the shorts weather, in the choice state. Set that if temperature is above 10 degrees C it's shorts weather. Let the default state go to Not Shorts Weather.
Show the full state machine
Save it and let's try it out. We need to start a new execution and supply the longitude and latitude, I supply the coordinates close to where I live.
Start the state machine
After the run we can see that it's apparently not shorts weather in the middle of the winter.

Deploy it with SAM

This SAM template will create EventBridge connection and the state machine if you prefer to use that.
State machine ASL:

Post to Slack

First of all, we need to create and setup an Slack application how to do that can be found in my post Serverless and event-driven translation bot
Now we need to create an EventBridge connection for our Slack application / bot. When posting to a channel using the Slack API we need to send the token in the Authorization header, setting the value to Bearer [TOKEN]. Here it become clear that this Authorization type in EventBridge Connection should be called Header and not API Key, that would have made so much more sense I think.
Just as with Shorts or Not we can create a StepFunction that use the connection and post to Slack. In the Body we need to supply the text and channel, we supply that as input parameters to the StepFunction invocation.
When calling the API from a StepFunction we need to set the Content-Type header, Slack doesn't parse our data and determine if it's json or not. This is done under Advanced Parameters.
We can now invoke the StepFunction and supply the parameters needed.

Deploy with SAM

Just as before, this SAM template will create EventBridge connection and the state machine if you prefer to use that.
StateMachine definition

Final Words

In this post I took a look at HTTP Endpoint state in StepFunctions. This functionality, to be able to call APIs directly, is a great feature. I will use this over and over again that is for sure, I love it. However, it doesn't come flawless, there are thing that I find really strange. But, I think this is such a great feature that I'm willing to live with the weirdness, for now!
Don't forget to follow me on LinkedIn and X for more content, and read rest of my Blogs
 

Comments