AWS Learning Material Certification Cheatsheets
AIF-C01 Domain 3 28% domain

Task 3.4: Describe methods to evaluate FM performance

Applications of Foundation Models · 4,757 words · source: Certified-AI-Practitioner-AIF-C01/domain-3/task-3-4-cheat-sheet.md

Domain 3: Applications of Foundation Models

Task Statement 3.4: Describe methods to evaluate FM performance

Domain 3 is "Applications of Foundation Models" and represents 28% of the scored AIF-C01 exam content. Task 3.4 focuses on how to evaluate foundation models and applications built with foundation models: model quality, benchmark datasets, human review, Amazon Bedrock Evaluations, RAG evaluation, agent/workflow evaluation, and business objective alignment.

Use this sheet for questions such as "which metric fits this task?", "which evaluation method should the team use?", "how do you know the FM meets business goals?", and "how should a RAG or agentic application be evaluated?"

Official Study Path

Use these in order:

  1. AWS Skill Builder: AWS Artificial Intelligence Practitioner Learning Plan
  2. AWS Skill Builder: Exam Prep Plan: AWS Certified AI Practitioner (AIF-C01)
  3. AWS Skill Builder: Domain 3 Review: AWS Certified AI Practitioner
  4. AWS Skill Builder: Domain 3 Practice: AWS Certified AI Practitioner
  5. AWS Exam Guide: Content Domain 3, Task Statement 3.4
  6. AWS Docs: Amazon Bedrock Evaluations, RAG evaluations, LLM-as-a-judge, SageMaker Clarify foundation model evaluations, and SageMaker JumpStart model evaluation

Exam Objective Map

You should be able to answer questions about:

Objective What to know for the exam
Evaluation approaches Human-in-the-loop evaluation, benchmark datasets, custom prompt datasets, automatic evaluation, Amazon Bedrock Model Evaluation, SageMaker Clarify foundation model evaluation
Evaluation metrics ROUGE, BLEU, BERTScore, F1, accuracy, semantic robustness, toxicity, LLM-as-a-judge metrics
Benchmark datasets GLUE, SuperGLUE, MMLU, BIG-bench, HELM, and task-specific datasets used by evaluation tools
Business fit Productivity, user engagement, task engineering, task completion, cost per interaction, user satisfaction, latency, quality, and ROI
FM application evaluation Evaluate the model, prompt, RAG retrieval, RAG generation, agents, workflows, tools, APIs, security, and user experience
Tradeoffs Smaller/faster models can reduce latency and cost but might reduce quality; more context can improve answers but increase cost and latency

One-Minute Mental Model

Foundation model evaluation is not one number. Evaluate at four layers:

  1. Model quality: Does the FM produce accurate, relevant, safe, useful, and well-structured outputs for the task?
  2. Application quality: Does the full application work after prompts, RAG, tools, workflows, APIs, guardrails, and UI are included?
  3. Operational performance: Is latency, throughput, availability, and cost acceptable?
  4. Business outcome: Does the solution improve productivity, user engagement, task completion, satisfaction, or cost per interaction?

Exam shortcut:

Model evaluation tells you whether the FM is good for the task.
Application evaluation tells you whether the whole system works.
Business evaluation tells you whether the solution is worth using.

1. Why FM Evaluation Is Harder Than Traditional ML

Traditional ML predictions are often deterministic and easier to compare against labels. For example, a classification model predicts a label and you compare that label with the ground truth label.

Generative AI outputs are harder to evaluate because:

  • The same prompt can produce different valid answers.
  • There can be many acceptable phrasings for one answer.
  • Quality depends on the task: summarization, translation, question answering, chat, code, retrieval, or agent actions.
  • Fluency does not guarantee correctness.
  • A response can be relevant but incomplete, complete but too slow, or correct but unsafe.
  • A model can perform well on a benchmark but poorly on a specific business workflow.

Skill Builder memory hook:

For generative AI, metrics are task-specific.
Use the metric that matches the job the model is doing.

2. Evaluation Approach Chooser

Evaluation approach Best for Exam signal phrase
Benchmark datasets Comparing general model capability across known tasks "Use established datasets to compare LLMs"
Custom prompt dataset Evaluating your real use case, domain, users, and edge cases "Use company-specific prompts and expected responses"
Automatic/programmatic evaluation Fast scoring with computed metrics "Quickly compare models with metrics"
Human-in-the-loop evaluation Subjective quality, tone, brand voice, nuance, safety, expert judgment "Use SMEs or human reviewers"
LLM-as-a-judge Scalable model-based review of responses with scores and explanations "Use a second LLM to score outputs"
A/B testing Measuring real user behavior between versions "Compare variants with live users"
Production monitoring Ongoing quality, drift, cost, latency, feedback, and safety after launch "Measure and review metrics over time"

Exam Pattern

If the question asks for fast automated comparison, choose automatic evaluation or LLM-as-a-judge.

If the question asks for subjective quality, brand tone, or expert correctness, choose human-in-the-loop evaluation.

If the question asks for a known public comparison, choose benchmark datasets.

If the question asks whether the model solves the company's actual problem, choose a custom dataset and business metrics.

3. Amazon Bedrock Evaluations

Amazon Bedrock Evaluations helps evaluate models, knowledge bases, and RAG sources. It supports automatic evaluation, human-based evaluation for models, and LLM-as-a-judge evaluation.

What Bedrock Can Evaluate

Resource What evaluation checks
Foundation models Whether a model performs well for a task or use case
Customized/imported models Whether customization improved the model
Prompt routers Whether routing choices produce better results
Knowledge bases Whether retrieval and generation are relevant, correct, complete, and safe
External models or RAG systems Bring your own inference responses for evaluation

Bedrock Evaluation Types

Type How it works Use when
Programmatic model evaluation Uses prompt datasets and computed metrics You need fast, repeatable scoring
Human-based model evaluation Human workers rate or compare responses You need subjective or expert judgment
LLM-as-a-judge A second LLM scores a generator model's response and explains the score You need scalable qualitative evaluation
RAG retrieval evaluation Scores retrieved context quality You need to tune retrieval settings
RAG retrieve-and-generate evaluation Scores the end-to-end RAG answer You need to evaluate final generated responses

Bedrock Prompt Datasets and Task Types

Model evaluation needs prompts to test against. In Amazon Bedrock, automatic model evaluation can use built-in prompt datasets or custom prompt datasets.

Know this:

  • Prompt datasets contain the inputs used to invoke the model during evaluation.
  • Custom datasets are useful when you need to test your own domain, users, prompts, and expected answers.
  • Ground truth or reference responses are used when the metric needs an expected answer for comparison.
  • Bedrock model evaluation reports can be viewed in the console and saved to Amazon S3.

Bedrock automatic model evaluation task types include:

Task type Common evaluation focus
General text generation Accuracy, robustness, toxicity, factual knowledge
Text summarization BERTScore, robustness, toxicity
Question and answer F1, robustness, toxicity
Text classification Accuracy and robustness

Bedrock LLM-as-a-Judge Metrics

Amazon Bedrock provides built-in LLM-as-a-judge metrics such as:

Metric What it checks
Correctness Whether the answer is correct, optionally compared with ground truth
Completeness Whether the response answers all parts of the prompt
Faithfulness Whether the response sticks to the available context and avoids unsupported claims
Helpfulness Whether the response is useful, coherent, and aligned with user needs
Logical coherence Whether the response is internally consistent
Relevance Whether the answer addresses the prompt
Following instructions Whether the model follows exact directions
Professional style and tone Whether the answer is appropriate for a professional setting
Harmfulness Whether the response contains harmful content
Stereotyping Whether the response contains stereotypes
Refusal Whether the model refused the request

Exam shortcut:

LLM-as-a-judge = a second model evaluates the first model's output.
Human evaluation = people evaluate the output.
Programmatic evaluation = metrics and algorithms evaluate the output.

4. SageMaker Clarify and SageMaker JumpStart Evaluation

Amazon SageMaker Clarify can evaluate text-based foundation models from SageMaker JumpStart and compare model quality and responsibility metrics.

Know this for the exam:

  • SageMaker Clarify foundation model evaluations can use automated evaluation, human worker evaluation, or the fmeval library.
  • Supported task types include open-ended generation, text summarization, question answering, and classification.
  • It can evaluate quality and responsibility dimensions such as accuracy, semantic robustness, factual knowledge, prompt stereotyping, and toxicity.
  • SageMaker JumpStart integrates with SageMaker Clarify foundation model evaluations for supported text generation models.
  • The fmeval library can give fine-grained control and can support models outside AWS.

Exam shortcut:

Bedrock Evaluations = evaluate Bedrock models, knowledge bases, RAG, and external responses.
SageMaker Clarify/FMEval = evaluate and compare foundation models in SageMaker/JumpStart workflows.

5. Core Metrics to Recognize

Task-Specific Metrics

Metric Best for What it measures Exam clue
ROUGE Summarization Overlap between generated summary and reference summary, often recall-oriented "Evaluate automatic summarization"
BLEU Translation N-gram overlap between machine translation and reference translation "Evaluate translated text"
BERTScore Summarization, QA, semantic similarity Semantic similarity using contextual embeddings rather than only exact word overlap "Compare meaning against a reference"
F1 score Question answering, extraction, classification Balance of precision and recall "Need both precision and recall"
Accuracy Classification and labeled tasks Fraction of correct predictions "Predicted label equals ground truth label"
Exact match Structured answers, short QA, IDs, labels Whether output exactly matches expected answer "Answer must match exactly"
Toxicity Responsible AI evaluation Whether output contains toxic or harmful language "Measure harmful content risk"
Semantic robustness Reliability under minor prompt changes Whether outputs stay stable when prompts are perturbed "Typos/case/spacing changes should not change answer much"

ROUGE

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It is commonly used for summarization.

Know this:

  • It compares a generated summary with one or more reference summaries.
  • It is based on overlap, such as words, n-grams, or sequences.
  • It is useful when the expected output is a summary of source text.
  • It does not fully judge factuality, usefulness, or tone by itself.

Exam trap: Do not choose ROUGE as the best metric for machine translation. BLEU is the classic translation metric.

BLEU

BLEU stands for Bilingual Evaluation Understudy. It is commonly used for machine translation.

Know this:

  • It compares machine-translated text with human reference translations.
  • It uses n-gram overlap and usually rewards translations that match reference wording.
  • It is not ideal for subjective chat quality, brand voice, or open-ended reasoning.

Exam trap: BLEU can penalize a valid translation if it uses different wording from the reference.

BERTScore

BERTScore uses contextual embeddings to compare generated text with reference text.

Know this:

  • It captures semantic similarity better than simple word overlap.
  • It is useful when wording can differ but meaning should match.
  • Bedrock automatic model evaluation uses BERTScore for summarization accuracy and robustness calculations.
  • It helps evaluate meaning similarity, but it should not be treated as a complete guarantee of factual correctness.

LLM-as-a-Judge

LLM-as-a-judge uses one model to evaluate another model's response.

Best for:

  • Correctness
  • Completeness
  • Faithfulness
  • Helpfulness
  • Relevance
  • Instruction following
  • Tone
  • Harmfulness
  • Custom rubric scoring

Strengths:

  • Faster and cheaper than large-scale human review.
  • Useful for many qualitative dimensions.
  • Can provide scores plus explanations.
  • Good for comparing models, prompts, RAG settings, or application variants.

Limits:

  • The judge model can be biased or wrong.
  • Results depend on the rubric and prompt.
  • Critical use cases still need human review, ground truth, and monitoring.
  • The evaluator model should not be blindly trusted as the only source of truth.

Exam shortcut:

LLM-as-a-judge is scalable qualitative evaluation, not absolute truth.

6. Benchmark Datasets

Benchmark datasets help compare models on standardized tasks. They are useful early in model selection, but they are not a substitute for testing your own use case.

Benchmark What to remember
GLUE General Language Understanding Evaluation; language understanding tasks such as sentiment analysis and question answering
SuperGLUE Harder successor to GLUE with more challenging reasoning and reading comprehension tasks
MMLU Massive Multitask Language Understanding; broad knowledge and problem-solving across many subjects
BIG-bench Broad set of challenging tasks beyond older model capabilities
HELM Holistic Evaluation of Language Models; combines multiple scenarios and metrics, including transparency concerns

AWS Evaluation Dataset Examples

Amazon Bedrock automatic model evaluation can use built-in or custom prompt datasets. Examples from AWS documentation include:

Dataset Typical use
TREX Real-world factual knowledge
BOLD Bias and robustness in open-ended generation
RealToxicityPrompts Toxicity evaluation
WikiText2 General text generation
Gigaword Text summarization
BoolQ Question answering
Natural Questions Question answering
TriviaQA Question answering
Women's E-Commerce Clothing Reviews Text classification

Exam trap:

Benchmarks help compare general capability.
Custom datasets show whether the model works for your actual users and business task.

7. Human-in-the-Loop Evaluation

Human evaluation uses people to judge model outputs. The reviewers can be internal employees, subject-matter experts, end users, or an AWS-managed workforce depending on the tool and setup.

Use human evaluation when:

  • The task requires domain expertise.
  • Quality is subjective.
  • Brand voice, style, tone, or empathy matters.
  • Safety, fairness, or policy compliance requires judgment.
  • You need to compare two good responses and choose the better one.
  • The answer cannot be scored reliably with exact labels or overlap metrics.

Human evaluation can score:

  • Helpfulness
  • Relevance
  • Correctness
  • Completeness
  • Tone
  • Brand alignment
  • Safety
  • User preference
  • Clarity
  • Conciseness

Exam trap: Human evaluation is valuable but slower and usually more expensive than automatic metrics.

8. Evaluating RAG Applications

RAG evaluation is not only model evaluation. You must evaluate retrieval and generation separately.

RAG Evaluation Layers

Layer What to evaluate Example metric or question
Data source quality Are documents accurate, current, approved, and complete? Source quality, freshness, governance
Chunking Are chunks the right size and overlap? Does retrieval return usable context?
Embeddings/vector search Are relevant chunks retrieved? Context relevance, context coverage, recall@k
Prompt assembly Is retrieved context injected clearly and safely? Does the prompt separate instructions from retrieved content?
Generation Does the final answer use the context correctly? Correctness, completeness, faithfulness
Citations Are source references present and accurate? Citation coverage, citation precision
Safety Are unsafe or irrelevant answers filtered? Harmfulness, refusal, guardrail outcomes

Bedrock RAG Evaluation

Amazon Bedrock RAG evaluations can evaluate:

Job type What it tests
Retrieve only Whether the RAG source retrieves relevant and sufficient information
Retrieve and generate Whether the full RAG workflow retrieves context and generates a correct, useful response

Useful RAG metrics include:

  • Context relevance
  • Context coverage
  • Response correctness
  • Response completeness
  • Faithfulness or hallucination detection
  • Helpfulness
  • Harmfulness
  • Citation coverage
  • Citation precision

RAG Exam Scenarios

Scenario Best evaluation focus
Answers miss important source facts Retrieval coverage and chunking
Answers cite irrelevant documents Context relevance and citation precision
Answers use correct documents but still invent facts Faithfulness and response correctness
Answers are safe but unhelpful Helpfulness, completeness, and user satisfaction
Retrieval works in tests but user results are poor Custom prompt dataset and production feedback

Exam shortcut:

RAG quality = retrieval quality + generation quality + source/citation quality.

9. Evaluating Agents and Workflows

Agents and workflows need broader evaluation because the FM may take actions, call tools, use memory, invoke APIs, or coordinate multiple steps.

Agent Evaluation Checklist

Area What to check
Goal completion Did the agent complete the user's task?
Tool selection Did it call the right tool or API?
Tool arguments Were the parameters correct and safe?
Permission boundaries Did it stay within allowed actions?
Grounding Did it use approved data and sources?
Multi-step reasoning Did it follow the right sequence of steps?
Recovery Did it handle tool errors, missing data, and ambiguous input?
Human approval Did it request review for risky actions?
Latency and cost Did multi-step execution stay acceptable?
Traceability Can you inspect prompts, tool calls, responses, and decisions?

Workflow Evaluation

For FM-powered workflows, evaluate:

  • Step success rate
  • End-to-end task completion rate
  • Error rate by workflow step
  • Manual intervention rate
  • Time saved per task
  • Cost per completed workflow
  • User satisfaction
  • Output quality at each stage
  • Safety and policy compliance

Exam trap: A model can generate good text but still fail an agent workflow if it calls the wrong tool or takes an unsafe action.

10. Business Objective Alignment

A foundation model meets business objectives only if it improves the target outcome at acceptable cost, risk, and quality.

Business Metrics

Business objective Metrics to watch
Productivity Time saved, tasks completed per user, manual effort reduced, automation rate
User engagement Active users, session length, repeat usage, feature adoption
Task completion Task completion rate, first-contact resolution, successful workflow rate
User satisfaction CSAT, thumbs up/down, survey score, qualitative feedback
Cost efficiency Cost per interaction, cost per resolved ticket, infrastructure cost, token cost
Revenue or conversion Conversion rate, average revenue per user, upsell rate
Quality Human rating, defect rate, escalation rate, rework rate
Risk reduction Harmful output rate, policy violation rate, hallucination rate, complaint rate

Technical Metrics That Support Business Goals

Technical metric Why it matters
Latency A slow assistant can reduce adoption even if answers are good
Throughput The system must support expected user volume
Availability Users and workflows need reliable access
Token usage Input and output tokens affect cost and latency
Context length Too little context can hurt quality; too much can increase cost
Retrieval precision/recall Poor retrieval causes weak RAG answers
Guardrail intervention rate Too high can block valid requests; too low can miss risks
Human escalation rate Shows whether automation is handling the intended work

Business Fit Questions

Ask:

  1. What problem are we solving?
  2. What task should the user complete?
  3. What does success look like numerically?
  4. What quality, safety, latency, and cost thresholds are acceptable?
  5. How will we collect feedback and monitor the system?
  6. What metric tells us to change the prompt, model, RAG settings, tool, or workflow?

Exam shortcut:

Good model score + poor business metric = not successful yet.

11. Performance, Latency, and Cost Tradeoffs

Skill Builder emphasizes that model integration requires deployment questions:

  • How fast must the model generate completions?
  • What compute budget is available?
  • Can the team trade model quality for lower latency or storage?
  • Does the application require real-time or near real-time interaction?
  • Does the application need external data, tools, APIs, or storage?

Optimization Techniques

Technique Helps with Tradeoff
Use a smaller model Lower latency, lower cost, easier deployment May reduce answer quality
Reduce prompt length Lower cost and latency May remove useful instructions/context
Reduce retrieved snippets Lower cost and latency in RAG May miss relevant facts
Reduce output length Lower latency and token cost May make answers incomplete
Tune inference parameters More concise or stable output Can reduce creativity or breadth
Cache repeated prompt prefixes Lower latency and cost for repeated static context Only helps when supported and prefix is stable
Distill a model Smaller/faster student model Requires extra customization effort

Exam trap: Optimizing for speed or cost can reduce quality, so evaluate after each change.

12. Evaluation Workflow for Real Projects

Use this sequence:

  1. Define the business goal and task.
  2. Choose candidate models, prompts, RAG settings, or agent designs.
  3. Build a custom evaluation dataset with realistic prompts, edge cases, and expected answers.
  4. Select task-specific metrics such as ROUGE, BLEU, BERTScore, F1, accuracy, or LLM-as-a-judge rubrics.
  5. Run automatic evaluation for fast comparison.
  6. Add human review for subjective or high-risk criteria.
  7. Evaluate application behavior, including RAG retrieval, tool calls, workflows, latency, and cost.
  8. Compare results against business metrics.
  9. Deploy with monitoring and feedback collection.
  10. Re-evaluate after model, prompt, data, tool, or workflow changes.

13. Decision Shortcuts for Exam Questions

Scenario Best answer pattern
Need to evaluate summarization quality Use ROUGE or BERTScore against reference summaries
Need to evaluate machine translation Use BLEU against reference translations
Need semantic similarity beyond exact word overlap Use BERTScore
Need subjective brand voice or tone review Use human-in-the-loop evaluation
Need scalable qualitative scoring with explanations Use LLM-as-a-judge
Need to compare general LLM capability Use benchmark datasets such as GLUE, MMLU, BIG-bench, or HELM
Need to know if a model works for company use case Use a custom prompt dataset and business metrics
Need to evaluate Bedrock model choices Use Amazon Bedrock Model Evaluation
Need to evaluate SageMaker JumpStart FMs Use SageMaker Clarify foundation model evaluations
Need to tune a Bedrock Knowledge Base Use RAG retrieval evaluation
Need to evaluate final RAG answer quality Use retrieve-and-generate evaluation
Need to evaluate an agent Measure goal completion, tool use, safety, latency, cost, and traceability
Need to prove business value Track task completion, user satisfaction, productivity, and cost per interaction

14. Common Wrong Answers

  • Use a single generic accuracy metric for every generative AI task.
  • Use ROUGE for translation when BLEU is the better exam answer.
  • Trust a fluent response without checking correctness or grounding.
  • Evaluate only the FM and ignore the RAG pipeline, tools, APIs, and workflow.
  • Use public benchmarks as the only evidence for business readiness.
  • Use human evaluation for everything when automatic metrics would be faster and cheaper.
  • Use only LLM-as-a-judge for high-risk decisions without human validation.
  • Optimize latency by shrinking the model or context without rechecking quality.
  • Measure only technical quality and ignore business outcomes.
  • Assume a good benchmark score means the model meets company-specific requirements.

15. Mini Flashcards

Q: Why is GenAI evaluation harder than traditional ML evaluation? A: Outputs are often non-deterministic and open-ended, so there can be many valid answers.

Q: What metric is commonly associated with summarization? A: ROUGE.

Q: What metric is commonly associated with translation? A: BLEU.

Q: What metric uses contextual embeddings to compare semantic similarity? A: BERTScore.

Q: What does LLM-as-a-judge mean? A: A second LLM scores another model's output using selected metrics or a rubric.

Q: When should you use human-in-the-loop evaluation? A: When quality is subjective, expert judgment is needed, or tone/safety/brand alignment matters.

Q: What are examples of benchmark datasets for LLMs? A: GLUE, SuperGLUE, MMLU, BIG-bench, and HELM.

Q: What is the difference between benchmark and custom dataset evaluation? A: Benchmarks compare general capability; custom datasets test the real business use case.

Q: What are the two Bedrock RAG evaluation types? A: Retrieve-only and retrieve-and-generate.

Q: What should you evaluate in an agent besides text quality? A: Goal completion, tool choice, tool parameters, permissions, safety, latency, cost, and traceability.

Q: Name three business objective alignment metrics. A: Task completion rate, user satisfaction, and cost per interaction.

Q: Why can a smaller model be attractive in production? A: It can reduce latency, storage, and cost, but it may reduce quality.

16. Practice Questions

  1. A company wants to compare two models for summarizing support tickets. Each test item has a reference summary. Which metric is most appropriate?

    • Answer: ROUGE or BERTScore. ROUGE is the classic summarization metric; BERTScore helps compare semantic similarity.

  2. A team wants to evaluate whether a translation model produces high-quality German translations from English source text. Which metric should it recognize?

    • Answer: BLEU.

  3. A support assistant sounds professional but sometimes invents policy details not found in the retrieved context. What should the team evaluate?

    • Answer: Faithfulness/hallucination, response correctness, and RAG grounding.

  4. A company needs SMEs to judge whether financial answers follow internal policy and use the right tone. Which evaluation method fits best?

    • Answer: Human-in-the-loop evaluation with subject-matter experts.

  5. A team wants fast qualitative scoring for thousands of prompt-response pairs and wants explanations for each score. What evaluation approach fits?

    • Answer: LLM-as-a-judge.

  6. A RAG application returns answers that are well written but based on irrelevant source chunks. What should be evaluated first?

    • Answer: Retrieval quality, especially context relevance and context coverage.

  7. An agent generates good responses but sometimes calls the wrong API action. What should evaluation include?

    • Answer: Tool selection, tool arguments, permission boundaries, workflow success, and traceability.

  8. A model performs well on MMLU but users still abandon the application. What kind of metric is missing?

    • Answer: Business and user metrics, such as task completion rate, user satisfaction, engagement, and cost per interaction.

  9. A team reduces the number of retrieved RAG snippets to lower latency. What must they do next?

    • Answer: Re-evaluate answer quality, retrieval coverage, correctness, and hallucination risk.

  10. A company needs to evaluate Bedrock Knowledge Bases and compare RAG configurations. Which AWS capability should it use?

    • Answer: Amazon Bedrock RAG evaluations.

17. Last-Day Review Checklist

  • I can explain why GenAI evaluation is task-specific and harder than deterministic ML evaluation.
  • I know when to use benchmark datasets, custom datasets, automatic evaluation, human evaluation, and LLM-as-a-judge.
  • I can match ROUGE to summarization and BLEU to translation.
  • I know BERTScore compares semantic similarity with contextual embeddings.
  • I can explain Amazon Bedrock Evaluations at exam level.
  • I can distinguish Bedrock programmatic evaluation, human evaluation, LLM-as-a-judge, and RAG evaluation.
  • I know SageMaker Clarify and JumpStart can evaluate text-based foundation models in SageMaker workflows.
  • I can list RAG evaluation dimensions: context relevance, context coverage, correctness, completeness, faithfulness, citations, and harmfulness.
  • I can explain why agents need evaluation of goal completion, tools, permissions, latency, cost, and traceability.
  • I can map FM evaluation to business metrics such as task completion rate, user satisfaction, productivity, and cost per interaction.
  • I remember that faster or cheaper model configurations must be re-evaluated for quality.

Official Sources