I work with many clients who have evaluated generative AI for various use-cases here at AWS, mainly in financial services. Many have completed initial prototypes and are finding that transitioning these into production is often more challenging than anticipated while others have found more rapid success. Based on those experiences, I’ve gathered some best practices to consider for your generative AI prototype or production project. Apologies if some of these seem obvious, but I’ve found it’s often useful to reassert the basics especially for senior business decision makers new to generative AI.

• Identify a data-driven use case that is a challenge for your business today

• Work with the primary stakeholder experiencing this challenge and work backwards from their needs

• Select a use-case that is data-driven

• Identify data sets that that differentiate your business – select key internal data assets over publicly available data

• Especially consider unstructured data (e.g. PDFs, emails, audio, video, images, ….) as generative AI offers programmatic extraction and interpretation where traditional analytics methods struggle

• Don’t be afraid to go big on data set size. A use case that can scale to larger data sets will demonstrate more value.

• It’s likely a given that your PoC will include a natural language interface for an end user. List the questions that your solution should be able to answer as well as the expected answer for each

• Identify the format of the answer – text, graphical, numeric, …

• Identify the likely data sources required for each question to provide accurate answers

• Where quantitative data inputs or quantitative responses are expected, be prepared to put in extra effort on prompt engineering or embedded code – LLMs are less-great with numeric computations than with text processing. Useful reference here.

• Define what a successful prototype looks like and define measurable KPI. Avoid qualitative benefits, strive for measurable KPI quantitative benefits.

• KPIs might include cost reduction, revenue improvement, accuracy improvement, false positive and false negative reduction, etc….

• Establish a cross-disciplinary team that will evaluate results and measure the results of the PoC

• A major cause of slow to transition into production is data security. Consider secure data access and data privacy early – anticipate how these will be addressed in production

• For regulated industries such as financial services, you will likely have to document several aspects of your solution for compliance if you transition your solution into production. Such measures include Fairness, Explainability, Privacy & Security, Controllability, Veracity & Robustness, Governance, and Transparency. More info here.

• There are also current and emerging regulations to be considered such as the EU AI Act as well as many data privacy policies which apply

• When using a pre-trained LLM with access to one or more data sets, you will need to provide your solution the context and detailed guidance. These instructions are inserted along with each prompt from a user.

• Defining a role, context, and other detailed guidance in these injected prompts are critical to getting accurate and consistent answers – an avoiding select topic and hallucinations

• Agentic flows, where the LLM is providing some reasoning in how to break down complex questions, will need special attention and testing. You may find some unexpected behaviors initially.

In working with clients, I provide them with guidance on these points and often provide a spreadsheet to help them get started with their thinking. A list of applicable regulatory policies is also useful. Below is a list of AWS references you may find useful:

Use Case: Portfolio Analysis

Primary User: Portfolio Manager

Success Criteria: 1.) Improved returns and reduced risk measured at 3 mos, 2.) reduction in time spent in portfolio analysis (hrs/week), 3.) improved justification of investment strategies in clear text with supporting quant data (per expert review)

Prompt Guidance: You are an expert in portfolio analysis and design. You will combine quantitative and qualitative data to provide answers within the context of a financial advisor. You are aware of applicable financial regulations and will cite these as necessary as part of your responses. You are aware of internal corporate guidance including investment policies and data privacy policies and will respect these limitations in all of your responses - citing the applicable policy where appropriate.