Picturesocial - How to analyze images with Machine Learning?
Image recognition sounds like some high tech computer science topic, and it is. Fortunately, there are tools that abstract the complexity of creating your own algorithms into a REST API. In this post, you are going to learn how to add image recognition to your Picturesocial app with an API.
José Yapur
Amazon Employee
Published Oct 15, 2022
Last Modified Apr 25, 2024
This is an 8-part series about Picturesocial:
We started this journey with containers, registries, Kubernetes, Terraform, and some other tools that enabled us to deploy our first API, but this journey is just starting. The core of Picturesocial is the capability to add tags automatically based on the pictures uploaded to our social network platform and this is what we are going to learn in this post by using artificial intelligence services with picture and pattern recognition.
As humans, we are very good at recognizing things we have seen before. You can look at this picture and almost instantly recognize that it shows a cat lying on a laptop with some flowers in the background. If you know a little bit more about cats, maybe you can also tell that this is an adorable Persian cat.
Computers do not possess this innate ability to recognize different things in an image, but they can be trained to do so. Deep learning is a machine learning technique that can be used to allow computers to recognize objects in images with varying confidence levels. In order for it to work, deep learning requires us to train models with thousands of labeled images: cat photos labeled "cat", dog photos labeled "dog", and so on. This can take up a significant amount of data, time, and compute resources, making it harder for us to train deep learning models on our own.
Luckily for us, we are able to add image detection capabilities to Picturesocial without having to create, train, or deploy our own machine learning models using simple, easy to use API’s.
On Picturesocial, we will use Amazon Rekognition to automatically tag the images uploaded by our users. This is an artificial intelligence service that requires no deep learning knowledge and will make our application capable of analyzing images with a simple API. To do this, we will use an Amazon Rekognition API called
DetectLabels that receives an image as input and outputs a list of labels. A label can be an object, scene, or concept. For example, the cat picture above could contain labels such as Cat, Computer, Flower (objects), Office (scene), and Indoors (concept).First, we will send our image to Amazon Rekognition so it can identify different things on it. To do so, we will create a request specifying the Amazon S3 Bucket in which our image is stored and its file name. We also tell the service the maximum number of labels we want to retrieve and the minimum confidence level for each label. The confidence level means how certain Amazon Rekognition is about the label assigned to an image.
Sample Request
Amazon Rekognition will analyze our image and return a response containing a list of labels and the level of confidence for each label. Labels for more common objects will also have a list of instances with the location where that object is located in the image.
Sample Response
The image below is an interpretation of these results. Amazon Rekognition was able to identify a laptop and a cat on this image within the bounded areas.
In this post, we are going to develop an API that will be in charge of detecting the relevant attributes, known as labels, of pictures stored on an S3 Bucket using Amazon Rekognition. This API will be created using .NET 6.0 with the Web API Template. Also, we are going to have a method that will receive two parameters: 1) the name of the file and 2) the name of the bucket. But we are going to implement the routing method just to receive the name of the file and set the bucket as default. So, let’s code!
- An AWS Account.
- If you are using Linux or macOS, you can continue to the next bullet point. If you are using Microsoft Windows, I suggest you to use WSL2.
- Install Git.
- Install AWS CLI 2.
- Install .NET 6.
Or
If this is your first time working with AWS CLI or you need a refresher on how to set up your credentials, I suggest you follow this step-by-step guide of how to configure your local AWS environment. In this same guide, you can also follow steps to configure AWS Cloud9, as that will be very helpful if you don’t want to install everything from scratch.
- First we are going to create the web API using the .NET CLI. The API name should be specified using the -n parameter, in our case
pictures.
- Now, we are going to open the newly created project using VS Code. We are going to use the following command from the terminal. This is the cool way but you can always just open the IDE and find the folder :D
- Now, if we look at the project structure, you are going to realize that a Default Controller called
WeatherForecastController.csis already in place, as well as aWeatherForecast.cs
- We are going to rename the controller file as
PictureController.csand we’ll delete theWeatherForecast.csclass. - Now let’s add the Nuggets that we are going to use for this project. In the same terminal that we used to create the web API let's position our cursor in
pictures
- And using .NET CLI we are going to add the following packages, inside the pictures directory.
- Let’s create the Class for handling the lists of labels from the Amazon Rekognition response. We are gonna name it
Labels.cs.
- Now, we are going to open
PictureController.csand add the package reference on the top. This way, we can use the packages added in the project inside our API.
- We are going to create a route for our API Controller, so we can call the API with the following URL format
http://url/api/pictures/photo.jpg.
- We have to change the Controller name to this:
- Then we define the HTTP Method GET and the route. We are going to create the Method
DetectLabelsthat will receive 2 parameters: 1) file name including the file extension as aStringand 2) bucket name set as default in the same method, as aString. We are using an async method as Amazon Rekognition will detect labels asynchronously. We are also returning the response as a JSON array of Labels. - At this point, create an S3 bucket in the same region that you are using for Amazon Rekognition, in our case it is gonna be
us-east-1, and you are going to use the bucket name as default in the method definition.
- Now, we are going to initialize the
AmazonRekognitionClientand set the region forus-east-1. We will initialize theListthat will contain the objects ofLabelsthat we are going to use as output.
- We are going to prepare the payload for Amazon Rekognition to detect the labels from images stored on an S3 Bucket. I will specify that I only need a maximum of 10 labels per response (
MaxLabels) and only the ones that have more than 80% of confidence (MinConfidence).
- We are going to send the request asynchronously and save just the Label name and probability inside our Labels List. Then we'll return the list.
The final
PicturesController.cs file should look like this:- Now, we are going to edit the
launchSettings.jsonfile inside thePropertiesfolder and replace it with the following example. Here, we are saying that we are going to use the port 5075 for HTTP only.
- We are going to save everything and test it by running the following command in the Terminal:
You should get something similar to this output:
Note the URL the app is listening on above:
http://localhost:5075. We'll use that in the next step.18 Let’s upload a picture into our S3 bucket. For example, in my case I uploaded two pictures.
- Let's compose the URL request using one of my pictures as example and paste it in the browser.
The result will be similar to this:
Now, we have our hashtags ready for Picturesocial! If you want to clone the API Project, you can do it with the following command:
If you got here, that means you are now using artificial intelligence services to label images!
Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.