Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”

AWS Logo
Menu

Image Text Validation using Amazon Rekognition and Bedrock

This post provides details of a serverless image text validation solution built using AWS AI services and Bedrock.

amazon-bedrockrekognitionamazon-rekognition
Swagat Kulkarni
Published Mar 5, 2024
Comments

Introduction

In this article, we aim to address a use case: validating the pictures of restaurant hours of operation taken by delivery drivers to ascertain whether the restaurant is closed or not. Given the wide variety of store operating hour signs, it can be challenging for a system to perform such validation. The solution leverages Amazon Rekognition to detect the text in the pictures and the cognitive abilities of large language models (LLM), readily available on Amazon Bedrock.Architecture Overview
Architecture Diagram
The diagram above provides the high level overview of the end to end architecture powering this use case.
  1. A mobile client app uploads the image along with metadata (including restaurant name, driver id, driver name, timestamp when picture was taken) to an S3 bucket.
  2. The picture upload triggers an event, which in turn invokes a Lambda function.
  3. The Lambda function initiates the orchestration flow, first invoking detectText api in Amazon Rekognition.
  4. The api response contains the text, in this case, restaurant hours of operations.
  5. Next, the Lambda function invokes an LLM, passing it the extracted text in step #4 and the metadata containing the time when the picture was taken.
  6. The LLM processes the information provided in the prompt and, in response, informs whether the restaurant is open or closed along with the reasoning.
  7. Finally, the LLM response along with the metadata is stored in Amazon DynamoDB table for analysis.
NOTE - The code sample is available in the amazon-bedrock-samples git repository here.

Prompt Engineering Technique

We used the one-shot prompting technique to prompt the LLM to provide accurate responses. One-shot prompting is a machine learning technique that uses a single example to guide the model's output. Here is an example of the one-shot prompt with Anthropic Claude's v2 model used in this solution.
1
2
3
4
Human:Current day is MONDAY and time is 8:00 AM. Take into account the current day and time and Based on the following text, tell if the restaurant is Closed or not closed.
Show your reasoning and respond with Closed or Not Closed. <text>Hours of Operation MON-FRI 10AM - 8PM SAT-SUN 10AM - 1PM</text>

Assistant:The current day is Monday and the current time is 8:00 AM. Since it is Monday and the restaurant is open 10AM - 8PM on weekdays, the restaurant is closed at the current time of 8:00 AM.

What happens when the image is invalid?

If the text in the image is illegible or unclear, the Language Learning Model (LLM) will indicate its inability to determine whether the restaurant is open or closed due to insufficient information. This feature can aid in identifying potential fraudulent situations when an image is posted by an untrustworthy individual.

Conclusion

In this post, we presented a demonstration of a sample solution for validating text, specifically focusing on restaurant operating hours. Nonetheless, this architectural approach can be adapted for various scenarios where there's a need for extracting and validating text from images using Amazon Rekognition and Bedrock.
 
This post have been co-authored with Tony Howell (anhwell)
 
Comments

Comments