AWS StepFunctions HTTP Endpoint demystified

I started to play around with StepFunctions new HTTP Endpoint state as soon as it was released. I found it to be a very nice addition and it worked really well, I was happy with it. At a recent re:Invent reCap I did a presentation and a demo on it, so I once again dove deep into it. I still like the functionality, not having to write a Lambda function just to call an API is very refreshing. However, there are some parts on it that I feel is just a bit misplaced. In this post I will go over the good and the bad parts with StepFunctions HTTP Endpoint State.

To demonstrate the HTTP Endpoint state we'll create two applications. The first will pull weather data from an open weather api, for a specific longitude and latitude. It will use the weather data to determine if it's shorts or not weather.

The second solution will use the Slack API to post a message to a Slack channel.

To call an API using a HTTP Endpoint state a EventBridge Connection is required. This so we don't have to hard code or include any authorization parameters in our StateMachine.

The thought behind this is good, but I feel it get a bit strange, why is there not a StepFunction Authorization resource? Why reuse what was created for EventBridge API Destinations? I would prefer if this was part of StepFunctions or was a standard SecretsManager secret.

Speaking of secrets. When a EventBridge Connection is created there is automatically a SecretsManager secret created, with a format like this.

I understand that the secret might need to be in certain format that the connection understand, but I would have preferred that I supply the secret to the connection and that the connection just populate it with data. I have seen and got several questions from people, that has been using the HTTP Endpoint state, and then discovers this secret, wondering what it is and what created it.

Let's create an EventBridge Connection for our open weather API. To do this, we need to navigate to the EventBridge console. Yes, it feels so strange. In the EventBridge Console there is also no "Connections" in the menu. Instead we have to navigate to "API Destinations" and select the "Connections" tab. Yes it is very well hidden. It feels that connections are not a first class citizen, I understand the placement when it was only for API Destinations. But it's not anymore, make it an option in the menu please!

In our first solution we'll use an Open Weather API from yr.no a weather service in Norway. Click Create Connection and fill in the details. Supply a name and a description and then we need to set an authentication.

Again this is where things get highly opinionated from AWS. We must select one of the three authentication types, it's not possible to create a connection without an authentication. But the API I'm going to use is fully open and doesn't require an authentication. Sure, a fully open API might not be the best of practices, but it does exists. Anyway, let's select API Key and just fill in some bogus values. And for that matter, I think API Key should have been named "Use Header" instead, more on that later.

After the connection is created let's build the first solution.

The first solution will call an open Weather API, fetch the current conditions at a location and decide if it's short weather or not. The state machine that will be used looks like this.

Call the API, use a ResultSelector to pick out the current conditions, use a Choice state to determine if it's shorts or not weather.

Create a StepFunction and add an HTTP Endpoint state to the graph, we configure it to use YR API, make sure you read the documentation if you are going to try this. The method will be GET and we use connection created before.

Now, the API require that we'll supply a longitude and latitude as query parameters, we can't set this directly in the endpoint. Instead scroll down and expand "Advanced parameters" and set the query parameters there.

The output from this state will be a json object with the headers and the body that the endpoint responds with, Headers and Body will be in separate objects, like this.

The data we get back from the API is an array with time series data, we are only interested in the current conditions, so here we can use a ResultSelector to get only what we need. In the output section add the following as ResultSelector.

Drag in a Choice State and two Pass state. Create a condition for the shorts weather, in the choice state. Set that if temperature is above 10 degrees C it's shorts weather. Let the default state go to Not Shorts Weather.

Save it and let's try it out. We need to start a new execution and supply the longitude and latitude, I supply the coordinates close to where I live.

After the run we can see that it's apparently not shorts weather in the middle of the winter.

This SAM template will create EventBridge connection and the state machine if you prefer to use that.

State machine ASL:

First of all, we need to create and setup an Slack application how to do that can be found in my post Serverless and event-driven translation bot

Now we need to create an EventBridge connection for our Slack application / bot. When posting to a channel using the Slack API we need to send the token in the Authorization header, setting the value to Bearer [TOKEN]. Here it become clear that this Authorization type in EventBridge Connection should be called Header and not API Key, that would have made so much more sense I think.

Just as with Shorts or Not we can create a StepFunction that use the connection and post to Slack. In the Body we need to supply the text and channel, we supply that as input parameters to the StepFunction invocation.

When calling the API from a StepFunction we need to set the Content-Type header, Slack doesn't parse our data and determine if it's json or not. This is done under Advanced Parameters.

We can now invoke the StepFunction and supply the parameters needed.

Just as before, this SAM template will create EventBridge connection and the state machine if you prefer to use that.

StateMachine definition

In this post I took a look at HTTP Endpoint state in StepFunctions. This functionality, to be able to call APIs directly, is a great feature. I will use this over and over again that is for sure, I love it. However, it doesn't come flawless, there are thing that I find really strange. But, I think this is such a great feature that I'm willing to live with the weirdness, for now!

Don't forget to follow me on LinkedIn and X for more content, and read rest of my Blogs