Use Fast Fourier Transform to Detect Musical Notes and Make them Command IoT Devices

title: "Use Fast Fourier Transform to Detect Musical Notes and Make them Command IoT Devices"
description: "This step-by-step guide will show you how to use Java, Amazon Corretto, Ukulele and Fast Fourier Transform to make your IoT devices dance to the music."
tags:

cloud
iot
java
amazon-corretto
aws
authorGithubAlias: vsenger
authorName: Vinicius Senger
movedFrom: /posts/fourier-iot
date: 2023-09-19

Fourier Transform algorithms show up all over the place: in image recognition, instrument tuners, signal processing in complex medical devices, machine learning algorithms, cryptography, and more. But how exactly do they work - and why are they so useful in so many contexts?

To learn about Fourier Transform in a practical and fun way, we will build an application that uses a Fast Fourier Transform (FFT) algorithm to detect musical notes using a laptop mic and send them as IoT messages to the cloud that can command devices, games, or anything else you can imagine!

Let's learn little more about this famous mathematician and his formula and then go build!

Fourier and Fourier Transform

Joseph Fourier has born in 1768 and grew up to be a brilliant French mathematician and physicist. He's best known for his work in the field of mathematics, especially in the field of Fourier analysis.

Fourier analysis is a branch of mathematics that deals with the study of functions and their properties in the frequency domain. The Fast Fourier Transform (FFT) is a mathematical technique used to analyze and transform signals from one domain to another. In simpler terms, it's a way to take a signal, which is essentially a set of numbers representing something like sound, light, or even stock market data, and break it down into its individual frequencies.

Think of it like taking apart a puzzle. Just as you might sort out the different colored pieces of a puzzle to put it together, the FFT sorts out the different frequencies that make up a signal. By doing so, it can help identify patterns, trends, and other important characteristics of the signal.

The FFT is particularly useful in fields like audio and signal processing, where it's often used to filter out unwanted noise, compress data, and perform other important tasks. While it may seem like a complex mathematical concept, the FFT is actually a widely used and very practical tool for understanding and manipulating signals in a variety of fields.

Let's Build!

So let's explore how the FFT can be put to work. The architecture of our solution is simple, and for now we are using the AWS IoT Core service to manage messages that we will be able to react to later:

The first component is your laptop with microphone running Processing.org with a notes detection script.
Once it detects a note, it will send the note name to the "control/sound" IoT topic.
From AWS IoT Core we can forward the message to other topics. For example, if the note is a G, we could turn on the smart bulb in a lamp by sending a MQTT message.
We can also trigger other services like AWS DynamoDB for storing the message data without one line of coding or we could call an AWS Lambda function (yes, we could literally build sound-triggered functions!).

The main steps to build our application are:

Install Processing.org, a Java-based programming language and development environment that is used to create visualizations, games, and other interactive media. It has a library for FFT processing called Minim.
Create IoT security credentials and a Thing on AWS IoT Core.
Create a Processing.org sketch that uses FFT to detect musical notes and send as IoT messages to the cloud.
React to messages: the sky is the limit! I will show how I did a cool integration with lamps and Roblox gaming!

Processing.org

Processing is a flexible software sketchbook and a language for learning how to code within the context of the visual arts. Since 2001, Processing has promoted software literacy within the visual arts and visual literacy within technology. There are tens of thousands of students, artists, designers, researchers, and hobbyists who use Processing to explore code and create works of visual art.

With just a few lines of code, you can make objects appear and move across the screen, interact with your mouse, and much more. Processing is a great way to get started with programming and to learn more about how code works.

The following image was generated by a few lines of code:

We won't break down this code here; I just want to show how a few lines of code can generate a beautiful image like this.

Installing Processing.org

Download Processing.org from here.
Install it and open it.
Go to Sketch > Import Library > Manage Libraries > Add Library... and search for Minim. Install it.
Now you can navigate into cool samples that comes with this library, click on File > Examples and navigate to Contributed Libraries > Minim
This is the SoundSpectrum example:

AWS IoT Core

Before we move to our Sketch to detect notes, we will create a Thing (yes, that's a technical term) in the AWS IoT Core service. This will manage the IoT / MQTT messages with security. AWS IoT Core is a message broker that can coordinate delivery of messages between devices with a publish / subscriber architecture: one device can publish a message to hundreds or thousands of subscriber devices.

Open your AWS Console and go to AWS IoT Core.
Click "All Devices" > "Things" > "Create Things".
Choose "Create single thing".
Name it fft_notes and configure it with default options.
Choose "Auto-generate a new certificate".
Skip the Attach policies.
Download the device certificate, keys, and root CA.
Now click on the AWS IoT > Settings in the main AWS IoT Core Console.
Copy the endpoint address to some place. We will need this information to configure our Processing.org connection to AWS IoT Core.

Now we have a representation of our notes Thing in the AWS Cloud and also the security keys to communicate using TLS 1.2.

Detecting Notes Sketch

Now that we have installed Processing.org, Minim library, and already have our Thing created in AWS IoT Core, let's start to explore the Sketch we did for detecting and converting notes to IoT / MQTT Messages!

Open Processing.org and copy and paste this code that is explained in comments:

Before running the code, we need to install some Java libraries that we developed and also dependencies to make the AWS IoT Core communication.

Download the zip here.

Unzip the file libs.zip in a temporary folder.
Drag-and-drop each .jar file inside your processing.org sketch.
You can also save your sketch and add the jars inside `code`` folder.

We are good to detect notes and send them to AWS IoT Core, so let's have some fun! Click the "Run" button and you will see the frequency graph.

Open the AWS IoT Core Console and Click MQTT Test Client > Subscribe to a topic > playground/sensors > Subscribe:

Open your Processing.org sketch and try to make some notes using your voice:

Then you can check the AWS Console:

Check out the live-demo video here:

Conclusion

Now you can imagine infinite possibilities: our team already did home automation integration to turn on and off lamps using Ukulele; we made it rain inside Minecraft based on notes, and we also changed Roblox game behavior. Now it's up to you to build on your own stuff using this magical algo: Fast Fourier Transform. Have fun!

More information about Fast Fourier Transform on Wikipedia:

This is a youtube video I really enjoy that explains FFT with a visual approach:

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Site Terms, Privacy, and more.