Leveraging AWS AI for an Intelligent Document Processing
Discover how to apply intelligent document processing on AWS combining Serverless, AI/ML, GenAI and Microservices.
Published Sep 16, 2024
In today's data-driven business environment, efficient document processing is no longer a luxury but an absolute necessity. From invoices and contracts to customer correspondence and internal reports, everything buries an organization in volumes of unstructured information. Conventionally, it has been one of the most challenging tasks to manage such information overflows, which involve great human resources for tasks related to the extraction, classification, and analysis of data. It was not only time-consuming but also prone to errors and inconsistencies. Putting the power of AI and ML to work, AWS provides a set of tools with which businesses could reimagine how they handle their documentation workflow. These next-generation solutions undertake automation of the smooth functioning of the whole document lifecycle, right from ingestion to insight generation, with minimum human touch and a great boost in operational efficiency. In this post, see how AWS uses AI-powered document processing to transform businesses in every industry by unlocking the full value of unstructured data to save time, reduce costs, and drive more accurate business outcomes.
Artificial intelligence is the process by which software applications gain the ability to generate, classify and perform tasks without specific programming. However, since it uses the resources generated by previous human endeavours to learn this expertise, a large amount of data is required to produce stable and accurate patterns of behaviour. Cloud computing allows users of AI to both use pre-trained models or train custom models and finally deploy systems at scale using microservice architectures.
What is often referred to as AIaaS (Artificial Intelligence as a Service) has become increasingly common as a means to leverage AI solutions without massive capital investment by paying only for what you need and relying on serverless systems, that reduce the operational overhead that you need to handle. On AWS you can access and integrate the state-of-the-art AI/ML models, by simply interact with the APIs of services like Amazon Bedrock and paying only for the resources that you used
AWS, powered by cloud computing, serverless architectures, and state-of-the-art AI/ML models was able to democratise the access of AI/ML technologies and redefined the document processing domain, which has for a long time been very human-and time-intensive. In this way, AWS is enabling organizations to turn their labor-intensive document workflows into smart, efficient operations.
There are several benefits to this approach.
- Speed. Manual processes can be a major bottleneck in business workflows. Machine learning systems can dramatically reduce the document processing time.
- Accuracy. With AI applications, human mistakes are minimized, reducing the risk of costly errors that may induce financial or other liabilities.
- Scalability. If workloads increase rapidly, cloud-based AI/ML systems can quickly scale to meet the needs, without the need for extra recruitment and staff training.
- Compliance. Automated systems make it easier to enforce compliance standards for both regulators and customers.
Amazon Web Services is a widely used cloud offering that provides modular solutions for a range of computing tasks. An intelligent process automation pipeline normally requires complex steps like:
· Ingestion: An automated process gathers documents in various formats collecting them using integration to external systems
· Process: Documents are prepared and converted to standard machine-readable formats. Amazon Textract allows to easily perform OCR at the lowest cost possible.
· Classify: Using AI Natural Language Processing (NLP) documents are categorized by content or purpose
· Extract: NLP, GenAI, and custom code are used to read important data and build a datalake based on Amazon S3 and NoSQL databases like Opensearch. While Amazon Bedrock can simplify access to GenAI capabilities required to perform the data extraction.
· Enhance: the extracted data is validated and enhanced by linking it to other sources with additional AI/ML models as well as customer systems
Figure 1: An example of an intelligent processing pipeline built by Capgemini on AWS
To leverage the power and flexibility of serverless cloud solutions, many are now utilizing microservice architectures. Microservices allow you to structure your AI/ML applications as a series of loosely coupled services. Using lightweight protocols, a microservice infrastructure can easily scale to meet application needs in terms of both capacity and complexity. In particular, given the massive infrastructural demands of machine learning applications, microservices provide the entry point for data-rich training, analysis and storage requirements.
Python is a great choice for developing your microservices architecture. Its adaptability, readability and wide support can be used:
· AWS Lambda function to connect different AWS services and manipulate data
· Docker images that can be executed inside Amazon ECS or Amazon EKS
· Amazon Sagemaker Notebook to automate MLOps processes
· AWS CDK for infrastructure as code (IaC) to automate complex pipelines
The document processing workflow is usually complex and required multiple coordinated steps. AWS Step Functions is a serverless orchestration service that lets developers create and manage multi-step application workflows in the cloud. By using the service’s drag-and-drop visual editor, teams can easily assemble individual microservices into unified workflows. At each step of a given workflow, Step Functions manages input, output, error handling, and retries, so that developers can focus on higher-value business logic for their applications.
Figure 2: An example of Step Functions built by Capgemini used to orchestrate a complex document intelligent pipeline
AI/ML model training is the process of feeding an AI/ML model curated data sets to evolve the accuracy of its output. Amazon Web Services gives us a lot of different AI/ML services having a different approach on training.
An intelligent process automation pipeline normally can leverage on the following services in the different steps:
· Process: Amazon Textract can easily execute large OCR and even performs simple data extraction without any training required
· Classify: NLP Models can be easily deployed inside Amazon SageMaker and normally requires large training sessions, Amazon Bedrock with Claude Haiku (the cheapest version of Antropic models) can be used for classification when there is no enough data for training
· Extract: NLP Models can be easily deployed inside Amazon SageMaker and normally requires large training sessions, Amazon Bedrock with Claude Sonnet or Opus (the most expensive version of Antrophic models) can be used for classification and it doesn’t require training. The latest versions of GenAI models have improved a lot performance and cost allowing to largely focus on GenAI for future implementation of this step.
· Enhance: Amazon Bedrock with Claude can be used to validate and correlate data; it doesn’t require training but could require a large knowledge base. Knowledge base can be easily created and updated with Amazon Bedrock but costs must be evaluated carefully.
In action, Capgemini’s serverless cloud machine learning systems have shown themselves to be successful in several ways.
- Easier management. For starters, orchestrated architectures like AWS Step Functions have proved easier to configure and administer when compared with more conventional choreographed architectures.
- Reduced costs. Serverless architectures have removed the need for upfront commitment to expensive fixed hardware solutions. Smart process automation projects typically have variable input demands, needing to scale or contract according to demand. Serverless solutions mean operational costs are reduced, as only the resources that are used are paid for.
- Process monitoring. The implementation of process monitoring and retry mechanisms has been an invaluable means to address potential issues during implementation. Pipeline troubleshooting is simplified, and automatic alerts notify developers of any potential problems, resulting in better-performing and more stable services.
Manual intervention massively reduced. The combination of Python and AI models for document classification and extraction has achieved pretty high accuracy levels. These automation pipelines have thus removed almost entirely the need for manual intervention, improving efficiency and reducing costs.