Crafting AWS AI/ML Exams
AWS Certification Exams: my perspective on shaping the industry AI/ML standard.
Published Mar 13, 2025
Last Modified Mar 16, 2025
Nowadays, cloud skills are important. People are eager to learn the best tools, integrate them into their workflows, and showcase the expertise on their CVs. Cloud certifications are designed to meet these demands.
In Oct 2023, I passed the AWS ML Specialty exam. I noticed a few questions that didn't align with my view on AI/ML concepts. Some were theoretical rather than practical, while others were unnecessarily verbose. I left my comments in the feedback form, hoping that the next time the exam content will be better.
In a day, I received a congratulatory email. I passed. While reviewing the provided benefits, I was surprised to find a mention of the Subject Matter Expert (SME) Program. Intrigued, I clicked the link and discovered that SMEs are responsible for creating exam content: they choose the topics being tested and write questions. With hesitation, I applied.
In Jan 2024, I received an invitation from Amazon to contribute to the development of a brand-new AI certification. Over the following months, I participated in every stage of the exam creation process. This put me on Amazon’s radar, and I soon began receiving invitations to help develop other exams.
First of all, a lot of people are behind these exams.
I did not expect that the exam content comes from SMEs, industry practitioners, who translate their experience and knowledge into questions. This is a diverse group.
Experts from Amazon and various companies, ranging from startups to large financial institutions, from AI-focused boutiques to global system integrators.
Experts with different backgrounds:
- Solutions Architects, MLEs, SREs, Data Scientists, Product Managers, Researchers, etc.
- Individuals with just a few AI initiatives under their belt to seasoned experts with decades of experience and even published books
- Both individual contributors and managers. I even "shook hands" with a few VPs and at least one CTO
Experts are coming from across the globe. Some motivated SMEs come on-site from as far as a 20-hour flight (layover time not counted). Virtual workshops are more convenient. Again, enthusiasm was evident - some SMEs join sessions as early as 3 AM in their local time.
Developing an exam turned out to be more elaborate than I thought. While I won’t delve into every detail, I’ll highlight a few steps I enjoy most.
This was the most challenging step. As a large group, we met a few times to define the exam content: topics to be tested and the depth of coverage for each.
Reaching consensus was tough. We debated how much emphasis each topic deserved, which metrics and models to prioritize, and how to craft clear, actionable objectives.
This step is more straightforward. SMEs contribute by
- Writing questions and response options
- Providing explanations for why each option is correct or incorrect
- Adding links to reputable sources to support the explanations.
There are guidelines on writing style, allowed abbreviations, naming conventions. Sounds familiar? Yes, it's like following a code style.
SMEs are tasked with writing questions at the appropriate cognitive level.
- "Understand/describe" questions are common on Foundation exams.
- "Analyze" questions are required for Professional-level exams.
- Memory-based questions (e.g., recalling the syntax of AWS CLI commands) are discouraged.
While coming up with a question is easy, crafting high-quality response options is an art. The challenge lies in aligning the question with the expected difficulty level and ensuring clarity for candidates from diverse backgrounds.
This is my favorite step, as it allows room for discussion, feedback, and collaboration.
SMEs meet in small groups to review questions. Poorly written questions are either rejected or sent back for rework. Good questions have chances to get into the exam. It is not a binary classification, though. The real effort lies in refining good questions with a few thoughtful edits. Transforming mediocre questions into good ones, instead of simply rejecting, is also satisfying.
This step is similar to ML models monitoring. You see statistics on how people perform on the exam—which answers they tend to choose, which comments they leave—and make adjustments accordingly.
Every complaint is reviewed and addressed, even if it's a single instance per question. I tested this myself. After taking an exam, I left comments in the feedback form—they were read aloud and taken seriously.
From this blog post you can get more details on the exact sequence of steps.
Over the past year, I've participated in several workshops, gaining a solid understanding of the process. Along the way, I've collected swag items and earned my AWS SME badge. AWS offers an impressive collection of badges for this program: click this link to review the variety. Notably, "Lead" SME badges are particularly rare.
And I remain in the program, since it allows me to
- Stay updated about the latest development across the entire AI/ML AWS ecosystem
- Practice clear writing and group discussion skills
- Make certifications better
A few souvenirs from Seattle:
