About AWS AI Practitioner (Beta) Exam

Exam Overview

What You Need to Know for the Exam?

WarningBear in mind that this is not a comprehensive guide. If you are seriously considering attempting this certification, you may want to enroll in training provided by AWS or by a third-party trainer. See the next section for my recommendations for courses and learning material.

1. Cloud Computing Basics

• Cloud vs on-prem: what are the advantages, what are the drawbacks of both
• Public vs private cloud
• Pricing of cloud services: one important keyword that you will encounter is on-demand. Everything in the cloud is pay-per-use, you only pay for what you use. Whenever you see a question talking about pricing, as a rule of thumb, you can default to the answer that mentions the on-demand keyword. Of course, there might be exceptions, so use your best judgment.

2. Basic Machine Learning Concepts

• AI/Machine learning: What is AI and what is machine learning? What use cases are solved by AI systems?
• AI application components: Data layer, ML algorithms, model layer, application layer.
• Neural Networks: What are neural networks? Also, you should know about their components: neurons, layers, activation functions, and loss functions.
• You should understand what backpropagation is: the process of updating the weights in the network to minimize the loss function
• Deep Learning: Know what deep learning is and what it is used for. Remember the keyword convolution, as it refers to a type of deep learning network (convolutional networks), primarily used for computer vision and image recognition
• GenAI: What is generative AI and what is it used for?
• Transformer Models/Diffusion Models/Multi-Modal Models: These are types of GenAI models. There is a high likelihood that you will encounter questions regarding the properties of these models, so it's recommended to have a basic understanding of them. You likely won't need to go into detail about their inner workings.
• Supervised vs. Unsupervised learning: Understand the difference between these two approaches and know when to use one over the other.
• Reinforcement Learning (RL): How does it work, and what is it used for? Be familiar with some key use cases.
• Machine learning model training and evaluation:
  • Model fit: The exam covers some more technical topics, such as underfitting and overfitting. These concepts are essential when training a model. A model is considered to underfit if it does not perform well on the training data. Conversely, it is overfitting if it performs well on the training data but poorly on evaluation or real-world data. If neither of these applies to your model, you can assume you have a balanced model.
  • Bias and variance: Both of these refer to errors introduced by your model. A model is biased when it consistently makes the same error in its predictions, often due to erroneous assumptions during training. Variance, on the other hand, reflects a model's sensitivity to small fluctuations in the input data. For the exam, you should be able to identify whether a model is highly biased or has high variance. Additionally, you should understand how bias and variance relate to model fit. For example, a model with high variance will likely overfit, while a model with high bias will probably underfit.
  • Model evaluation metrics: it is expected that you are familiar with concepts of how to evaluate a model. It is not required to know the math behind those concepts, but it expects you to know when you should use one metric versus another. These metrics are the following:
    • Metrics used for classification models:
      • Confusion Matrix: used to evaluate the performance of classification models. It is usually structured as a square matrix, where rows represent the actual classes, and columns represent the predicted classes.
      • Accuracy: measures the fraction of correct prediction.
      • Recall (or Sensitivity): measures the true positive rates of the predictions.
      • Precision: measures the correct positive rate.
      • Specificity: measures the true negative rate.
      • F1 score: combines precision and recall into a final score. Use it when both precision and recall are considered important for evaluating your model.
    • Metrics used for regression models: similar to the metrics from the classification models, you don't need to know the formulas and mathematics behind these metrics. What you should know for the exam is to recognize them and know if they can be used with a presented model or not:
      • MAE (Mean Absolute Error): measures the average magnitude of errors between the predicted values and the actual values.
      • MAPE (Mean Absolute Percentage Error): used to assess the accuracy of a predictive model by calculating the average percentage error between the predicted values and the actual values. MAPE expresses the error as a percentage.
      • RMSE (Root Mean Squared Error): measures the average magnitude of the error between the predicted values and the actual values, with a higher emphasis on larger errors.
      • R Squared: explains variance in our model. R2 close to 1 means predictions are good.
    • Metrics used to evaluate LLMs - the exam might ask you about evaluating the performance of a large language model. In this case, you would want to know about these:
      • Perplexity loss: measures how well the model can predict the next word in a sequence of text
      • Recall-Oriented Understudy for Gisting Evaluation (ROUGE): a set of metrics used in the field of natural language processing to evaluate the quality of machine-generated text.

3. Generative AI and AWS Services for GenAI

• Knowledge Bases: A solution for provisioning and using vector databases. You may want to use vector databases if you are building a system based on RAG (Retrieval-Augmented Generation)
• Agents: BBedrock's method for function calling. You can integrate your model with an "action," allowing the model to perform tasks in addition to responding to messages, such as executing a Lambda function.
• Guardrails: You can use Guardrails to limit the model's responses in terms of what it should be able to answer. Additionally, Guardrails offer features such as detecting personally identifiable information (PII) and removing it from the response.
• Model Evaluation: You can evaluate a model using AWS-provided metrics or by employing human reviewers.

• Amazon Q Business: it is a question-answer-based chatbot built on Amazon Bedrock. It can ingest your company-owned internal data, and it will be able to answer your questions based on that data.
• Amazon Q Developer: it servers two completely different use cases:
  • It is a chatbot built on AWS documentation, so it can answer your questions related to AWS services.
  • It is a code companion (previously known as CodeWhisperer) similar to GitHub Copilot. It can generate source code. It can integrate with a bunch of IDEs and it can help you write code.

4. Prompt Engineering

• Zero-Shot Prompting: This is a more "scientific" definition of what I was doing before adopting any prompt engineering techniques. You simply present a query to the model without specific wording, formatting, or examples of expected output. The model's output may vary. It could be useful or it might be complete garbage.
• Few-Shots Prompting: In this approach, you provide a prompt with examples of what you would expect as output from the model. Surprisingly, this technique works better than I had imagined. In terms of the exam, you should choose this technique when asked for a low-cost solution with the highest precision.
• Chain of Thought Prompting: This involves dividing your queries into a sequence of reasoning steps, using phrases like "Think step by step." Interestingly, the GPT-3 model utilizes chain of thought prompting under the hood, which has contributed to its recent popularity. Therefore, expect some questions about it.
• Retrieval-Augmented Generation (RAG): RAG is also considered a prompting technique. It relies on using a vector database to fetch content related to the user query. This content is then injected into the user prompt as additional context.

• Temperature: value between 0 and 1.0, defines the creativity of a model. The higher the value, the more creative responses you will get.
• Top P: value between 0 and 1.0, defines the size of the set of available words when generating the parts of an answer. For example, for a value of 0.25, the model will use the 25% most likely next word.
• Top K: similar to Top P, but it is an integer value. By setting a Top K value, we tell our model that it only should use words from the next Top K available options.

4. Amazon SageMaker

1. Prompt engineering (excluding RAG): least expensive
2. RAGs: they are more expensive than other prompt engineering techniques because they usually require the presence of a vector database
3. Instruction-based fine-tuning: it is a fine-tuning approach that uses labeled data to modify the weights of a model. It requires model training, which demands specific hardware, so it is considered more expensive than RAGs
4. Domain adaptation fine-tuning: uses unlabeled data for fine-tuning, it is the most expensive approach

5. AWS Managed AI Services

• Amazon Comprehend and Amazon Comprehend Medical: extract relevant information from documents of all kinds.
• Amazon Translate: an on-demand translation service, think of it as an on-demand version of Google Translate.
• Amazon Transcribe and Amazon Transcribe Medical: speech-to-text service.
• Amazon Polly: text-to-speech service.
• Amazon Rekognition: used for image recognition and classification.
• Amazon Forecast: used with time series data to forecast stuff. Discontinued by AWS, but is still part of the exam.
• Amazon Lex: it is similar to Amazon Q, or an Amazon Bedrock agent. It is technically Alexa as a service.
• Amazon Personalize: recommendation service.
• Amazon Textract: used to extract text from images (OCR).
• Amazon Kendra: it is a document search service. It is somewhat similar to Amazon Q, but it is way more restricted and it cannot do summarization (good idea to keep this in mind!)
• Amazon Mechanical Turk: it is not necessarily an AI service. With Mechanical Turk you rely on a human workforce to carry out certain tasks for machine learning, such as labeling, classification, and data collection.
• Amazon Augmented AI (A2I): likewise Mechanical Turk, is not necessarily a managed AI service. It is a service that lets you conduct a human review of machine learning models. It can use Mechanical Turk under the hood.
• AWS DeepRacer: this is also an interesting thing to mention. It is a game, where you use reinforcement learning to drive a race car. While DeepRaces is still part of the exam, the service is discontinued by AWS.

6. AI Challenges and Responsibilities

• Regulatory violation
• Social risks
• Data security and privacy concerns
• Toxicity
• Hallucinations
• Nondeterminism

• Scope 1: your app is using public GenAI services
• Scope 2: your app is using a SaaS with GenAI features
• Scope 3: your app is using a pre-trained model
• Scope 4: your app is using a fine-tuned model
• Scope 5: your app is using a self-trained model

7. AWS Security Services and Other Services

• P3, P4, P5, G3, G6 instances: These are instances with a GPU assigned to them, they can be used for training and for inference as well;
• AWS Trainium and AWS Inferentia: these are instances specifically built for training and inference. They provide a lower cost for either training or inference.

• You should know what a VPC is. A VPC (Virtual Private Cloud) is a virtual private network, isolated by default from internet traffic or other AWS services. Certain resources, such as EC2 instances or most databases, need to be placed in a VPC to function properly. You can provide internet access to the VPC if desired.
• The exam often asks about VPC Endpoints in terms of security. VPC Endpoints allow communication with AWS services from a private VPC, one that does not have internet access. When using a VPC Endpoint, the traffic does not pass through the public internet.

• CloudWatch: used for monitoring and logging
• CloudTrail: used for having a trail about any action in an AWS account
• AWS Config: used to enforce compliance in an account
• AWS Lambda Function: serverless functions that run only when needed. In the context of this exam, usually they are used for integration between 2 services

Courses and Practice Exams that I Recommend

My Experience Taking the Exam

My List of Recommendations for You for the Exam

• Keep an eye on the clock. There are 85 questions, and you have less than 1.5 minutes per question. Don't spend too much time on any single question. Many of them can be answered quickly as long as you're well-prepared. Try to select the correct answer and move on.
• If you're unsure about a question, flag it and move on. You'll have time to return to it later and figure it out. It's important not to panic if you don't know something right away.
• You will encounter questions you don't know the answer to, and that's okay. The official exam guide is intentionally vague. Any course you take will have some gaps. Don't be discouraged if you come across something completely new during the exam. The important thing is to pass. Your final score doesn't matter much.
• The exam is challenging, but with adequate preparation, anyone should be able to pass it. You'll be fine.

Closing Notes