Save Money on AWS IoT: Master Basic Ingest for High-Volume Data !

In this blog we will dive into the exciting world of AWS IoT, specifically a cost-saving feature called Basic Ingest.

For those who are new, AWS IoT provides the cloud services that connect your IoT devices to other devices and AWS cloud services. . AWS IoT is a powerful service, however managing large amounts of data can get expensive at times ,that's where Basic Ingest comes in.

What is Basic Ingest?

Basic Ingest is a feature in AWS IoT service that helps you optimize data flow for high-volume data streams. Here's the key: it bypasses the message broker, which is where data typically goes through a middleman before reaching its destination.

By skipping the message broker, Basic Ingest reduces processing overhead and cuts down on messaging charges. This can be a huge cost saver, especially if you're dealing with millions of messages per day.

Basic Ingest mechanism ingests messages from your devices or application on reserved topic names that start with $aws/rules/rule_name for their first three levels, where rule_name is the name of the AWS IoT rule that you want to invoke when message gets ingested from your device. You can't subscribe to these basic ingest topics directly.

You can use an existing rule with Basic Ingest by adding the Basic Ingest prefix ($aws/rules/rule_name) to the message topic that you'd use to invoke the rule. For example, if you have a rule named BuildingManager that's invoked by messages with topics like Buildings/Building5/Floor2/Room201/Lights ("sql": "SELECT * FROM 'Buildings/#'"), you can invoke the same rule with Basic Ingest by sending a message with topic $aws/rules/BuildingManager/Buildings/Building5/Floor2/Room201/Lights.

Basic Ingest isn't for everything. It's ideal for scenarios where you have a lot of simple data that needs to be delivered quickly. Think sensor readings, machine logs, or any situation where complex message processing isn't necessary.

Setting Up Basic Ingest

First, we need to create a Basic Ingest(simple AWS IoT rule) rule. It's pretty straightforward. You just give it a name and choose the AWS service where you want to send the data.
Verify that your device or application is using a policy that has publish permissions on $aws/rules/*. Or, you can specify permission for individual rules with $aws/rules/rule_name/* in the policy
Send data from your device to the topic $aws/rules/rule_name/
Verify data is received at the rule destination

Cost Reduction Example with Basic Ingest

Below is an example of how Basic Ingest can significantly reduce costs in the AWS IoT use case. I used The AWS Cost Calculator a very useful tool provided by AWS to help customers estimate and plan their AWS costs.

The cost calculation was run on 29th August 2024 , US East(Ohio) region for AWS IoT Core service.
Used most of the default values .
Number of Devices: 1,000
Number of Messages per Device per Month: 100,000

Without Basic Ingest:

Total Cost: $103.50 per month

With Basic Ingest:

Total Cost: $3.50 per month

That's an astounding 96.6% reduction in the monthly costs for this IoT use case scenario!

Basic Ingest Use Cases and Best Practices

Basic Ingest has some hidden gems under the hood. For example, you can use substitution templates to include the original topic name in the data you send. This can be handy for maintaining context.
Basic Ingest is great for high-volume, simple data. If you need complex message processing, stick with the standard message broker flow.
Remember about security – Basic Ingest maintains the same secure data transfer as the regular approach.
The basic ingest path is a special path that only routes messages to the IoT Rules Engine.
A topic in a publish or subscribe request can have no more than 7 forward slashes (/). This excludes the first 3 slashes in the mandatory segments for Basic Ingest topics ($AWS/rules/rule-name/).
Be mindful of rule engine engine service quota limits .

When not to use Basic Ingest

If you need a publish/subscribe broker to distribute messages to multiple subscribers (for example, to deliver messages to other devices and the rules engine), you should continue to use the AWS IoT message broker to handle the message distribution. However, make sure that you publish your messages on topics other than Basic Ingest topics. Please note you can not Subscribe to Basic Ingest topics.
When you need AWS IoT Device Defender metric 'num-messages-received' metrics for your observability needs . Your devices and rules can't subscribe to Basic Ingest reserved topics hence metrics is not emitted as it doesn't support subscribing to topics.
Latency is a concern for your design pattern .On the basic ingest path the service uses a different messaging path internally with a goal of minimized cost per request, and higher latency is a tradeoff of that.

Now that you've learned about the power of Basic Ingest to optimize your IoT costs, it's time to put this knowledge into practice. The cost savings from Basic Ingest will give you more room to innovate and scale your IoT solutions.

Enjoy building on AWS IoT. Connect devices, drive value, and have fun!

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.