AWS Learning Material Certification Cheatsheets
AIF-C01 Domain 1 20% domain

Task 1.3: Describe the AI/ML development lifecycle

Fundamentals of AI and ML · 4,284 words · source: Certified-AI-Practitioner-AIF-C01/domain-1/task-1-3-cheat-sheet.md

Domain 1: Fundamentals of AI and ML

Task Statement 1.3: Describe the AI/ML development lifecycle

Domain 1 is 20% of scored exam content. Task 1.3 focuses on the lifecycle of AI/ML systems: how data, models, evaluation, deployment, monitoring, and business value fit together.

Use this sheet to answer lifecycle, ordering, matching, and service-selection questions. You do not need to build production ML pipelines for the exam, but you should recognize the major stages, outputs, AWS services, and operational tradeoffs.

Exam Objective Map

You should be able to:

AWS exam objective What to know for the exam
Describe and differentiate components of an AI/ML pipeline Data collection, data preparation, feature engineering, training, validation, evaluation, deployment, inference, monitoring, retraining
Describe sources of foundation models AWS-managed FMs in Amazon Bedrock, Bedrock Marketplace models, imported/customized FMs, open source pretrained models, SageMaker JumpStart pretrained models, custom-trained models
Describe methods to use a model in production Managed API service, managed endpoint, serverless API, batch inference, asynchronous inference, self-hosted API
Identify AWS services and features for each pipeline stage Amazon Bedrock, Amazon SageMaker AI, Amazon Q, Amazon Quick Sight, Kiro, S3, SageMaker Pipelines, Experiments, Model Registry, Model Monitor, Clarify
Describe MLOps fundamentals Experimentation, repeatable processes, scalable systems, CI/CD, model registry, lineage, technical debt, production readiness, monitoring, retraining
Describe evaluation metrics Accuracy, precision, recall, F1 score, cost per user, development cost, customer feedback, ROI

The Big Picture

Think of AI/ML development as a loop, not a one-time project:

Business problem
  -> Data collection and preparation
  -> Model or foundation model selection
  -> Training, customization, prompting, or RAG setup
  -> Evaluation against technical and business metrics
  -> Deployment for inference
  -> Monitoring and feedback
  -> Retraining, re-evaluation, or replacement

Exam shortcut:

Traditional ML lifecycle: data -> train -> evaluate -> deploy -> monitor -> retrain.
Generative AI lifecycle: use case -> choose FM -> prompt/RAG/customize -> evaluate -> deploy -> monitor -> improve.

Pipeline Components

Component Purpose Output AWS anchor
Business problem definition Identify what decision or workflow the model improves. Use case, success criteria, constraints AWS Well-Architected ML thinking
Data collection Gather relevant input data. Raw dataset Amazon S3, databases, data lakes
Data labeling Add known answers for supervised learning. Labeled dataset SageMaker Ground Truth
Data preparation Clean, normalize, transform, split, and validate data. Training/validation/test datasets SageMaker Data Wrangler, SageMaker Processing
Feature engineering Create useful model inputs from raw data. Features SageMaker Feature Store
Model selection Choose algorithm, pretrained model, foundation model, or managed AI service. Candidate model approach SageMaker built-in algorithms, SageMaker JumpStart, Amazon Bedrock
Training or customization Train a custom model or adapt an FM. Model artifact or customized model SageMaker training, Bedrock model customization
Experiment tracking Record parameters, datasets, metrics, and artifacts. Reproducible experiment history SageMaker Experiments
Evaluation Compare model outputs against target metrics. Metrics, approval decision SageMaker evaluation, Bedrock evaluations
Model registry Catalog versions, metadata, approval status, and lineage. Approved model version SageMaker Model Registry
Deployment Make the model available to applications. Endpoint, API, batch job, managed app SageMaker endpoints, Bedrock API, Amazon Q, Amazon Quick Sight
Inference Run the model on new inputs. Prediction, classification, recommendation, generated output SageMaker inference, Bedrock runtime
Monitoring Detect data quality issues, model quality drift, bias drift, and cost/usage changes. Alerts, reports, dashboards SageMaker Model Monitor, CloudWatch, Clarify
Feedback and retraining Use new data and observed performance to improve the system. Updated model or pipeline SageMaker Pipelines, CI/CD, human review

Exam trap: a pipeline is more than training. Production ML includes data preparation, evaluation, deployment, monitoring, governance, and improvement.

Typical ML Lifecycle Stages

Stage Exam-ready meaning Common mistake
Define problem Convert a business problem into an ML objective and success metric. Starting with a model before defining value
Gather data Collect relevant, representative data. Assuming more data is always better even if it is low quality
Prepare data Clean inconsistencies, handle missing values, split data, and transform features. Training directly on messy raw data
Train Use an algorithm and data to create a model. Confusing training with inference
Validate/test Check how the model performs on data not used for training. Judging only by training accuracy
Deploy Make the model available for production use. Thinking deployment means the model is done
Monitor Track model behavior, drift, errors, latency, and cost. Ignoring changing production data
Retrain Update the model when data, business patterns, or performance changes. Retraining blindly without evaluation

Memory hook:

Define, Data, Train, Test, Deploy, Monitor, Retrain.

Generative AI Lifecycle

Generative AI often starts with an existing foundation model instead of training from scratch.

Stage What happens AWS anchor
Select use case Define task: summarize, generate, classify, answer, automate, or assist. AWS AI/ML use case planning
Choose foundation model Compare model capability, modality, latency, cost, context length, region, and risk. Amazon Bedrock model catalog, SageMaker JumpStart
Prompt design Write instructions, examples, and output constraints. Bedrock playgrounds, Amazon Q Developer, Kiro
Grounding / RAG Retrieve relevant company data so answers are grounded in trusted sources. Bedrock Knowledge Bases, Amazon Q Business
Guardrails and controls Reduce harmful, unsafe, or off-policy responses. Amazon Bedrock Guardrails
Customize if needed Fine-tune or continue pretraining when prompting/RAG is not enough. Bedrock customization, SageMaker AI
Evaluate Measure model quality, safety, retrieval quality, and business fit. Amazon Bedrock evaluations
Deploy Integrate through API, app, assistant, or workflow. Bedrock runtime API, Amazon Q, SageMaker endpoint
Monitor and improve Track quality, user feedback, cost, latency, safety, and drift. CloudWatch, Bedrock logging/evaluation patterns

Exam trap: RAG and fine-tuning are not the same. RAG retrieves external knowledge at inference time. Fine-tuning changes model behavior using training examples.

Foundation Model Sources

Source What it means Choose when... AWS anchor
Managed foundation model Use an FM provided through a managed service API. You want fast access without managing infrastructure. Amazon Bedrock
Bedrock Marketplace model Use specialized models available through Bedrock Marketplace. You need model choice beyond default provider catalogs. Amazon Bedrock Marketplace
Customized foundation model Start with an FM and adapt it with your data. Prompting or RAG is not enough for style, domain behavior, or task format. Bedrock customization
Imported foundation model Bring a supported external model into Bedrock for managed use. You already have a model and want Bedrock operational controls. Amazon Bedrock imported models
Open source pretrained model Use a publicly available pretrained model. You need control, portability, or a specific open model. SageMaker AI, self-hosting
SageMaker JumpStart pretrained model Use pretrained models and solution templates from SageMaker. You want a quick start for common ML, text, image, and FM use cases. SageMaker JumpStart
Custom-trained model Train a model from your own data and algorithm. You need a specialized predictive model or own the full model lifecycle. SageMaker AI

Exam shortcut:

Bedrock -> managed FMs for GenAI apps.
SageMaker AI -> build, train, customize, deploy, and govern custom ML/FMs.
JumpStart -> pretrained models and templates to get started quickly.

Production Use Methods

Production method Best for AWS anchor Exam clue
Managed AI service API Prebuilt capability with minimal ML operations. Transcribe, Translate, Comprehend, Lex, Polly, Rekognition, Amazon Q "No need to build a model"
Managed FM API Generative AI through hosted foundation models. Amazon Bedrock "Use FMs through a serverless API"
Real-time endpoint Low-latency online predictions for sustained traffic. SageMaker real-time inference "User waits for response now"
Serverless inference Intermittent or unpredictable request traffic. SageMaker serverless inference "Do not pay for idle endpoint capacity"
Batch transform Offline scoring for large datasets available upfront. SageMaker Batch Transform "Score a file overnight"
Asynchronous inference Queued requests with large payloads or long processing. SageMaker asynchronous inference "Return later; long-running request"
Self-hosted API Full control over runtime, model server, network, or compliance setup. EC2, ECS, EKS, Lambda for small models "Need custom hosting/control"
Embedded assistant or app End users interact through a business, developer, BI, or coding assistant. Amazon Q Business, Amazon Q Developer, Amazon Quick Sight, Kiro "Natural language assistant experience"

Exam trap: self-hosting gives control but increases operational responsibility. Managed services usually reduce undifferentiated infrastructure work.

AWS Service Chooser For Task 1.3

Service or feature Lifecycle role Know this for the exam
Amazon SageMaker AI End-to-end ML platform Build, train, deploy, monitor, and govern ML models and FMs
SageMaker Processing / Data Wrangler Data preparation Clean, transform, and process datasets
SageMaker Feature Store Feature management Store, share, and reuse ML features
SageMaker built-in algorithms Model training Use AWS-provided algorithms for common ML problems
SageMaker JumpStart Pretrained models and templates Start from pretrained models or solution templates
SageMaker Experiments Experiment tracking Track datasets, parameters, metrics, and artifacts
SageMaker Pipelines Workflow automation Build repeatable ML workflows and CI/CD-style model pipelines
SageMaker Model Registry Model versioning and approval Catalog versions, metadata, lineage, approval status, and deployments
SageMaker Model Monitor Production monitoring Detect data quality, model quality, bias drift, and feature attribution drift
SageMaker Clarify Bias and explainability Help detect bias and explain model predictions
Amazon Bedrock Managed foundation models Build GenAI apps with FMs through API access and no infrastructure management
Bedrock Knowledge Bases RAG Connect FMs to enterprise data for grounded answers
Bedrock Agents Agentic workflows Let FMs choose actions and use tools/APIs to complete tasks
Bedrock evaluations Model and RAG evaluation Compare models, knowledge bases, and RAG systems with automatic, human, or judge-model evaluations
Amazon Q Business Enterprise assistant Answers, summaries, content generation, and task completion using enterprise data with permissions-aware responses
Amazon Q Developer Developer assistant Helps understand, build, extend, and operate AWS applications; supports IDE coding assistance
Amazon Quick Sight / Amazon Quick Generative BI Natural-language BI authoring, data Q&A, executive summaries, and data stories
Kiro Agentic coding and specs Turns prompts into specs, code, documentation, and tests; useful as a development productivity tool
Amazon S3 Data and artifact storage Common storage layer for datasets, model artifacts, logs, and evaluation data
Amazon CloudWatch Monitoring and alerts Metrics, logs, alarms, and operational visibility

Exam trap: Amazon Q and Kiro are user-facing AI assistants/productivity tools. Amazon Bedrock and SageMaker AI are core services for building AI/ML solutions.

MLOps Fundamentals

MLOps applies software engineering and operations practices to machine learning. The goal is repeatable, governed, production-ready model delivery.

Concept Meaning AWS anchor
Experimentation Try multiple datasets, algorithms, parameters, prompts, or FMs and track results. SageMaker Experiments, Bedrock evaluations
Repeatable processes Make data prep, training, evaluation, and deployment reproducible. SageMaker Pipelines
Scalable systems Support larger data, more users, more models, and reliable deployment. SageMaker AI, Bedrock, autoscaling, managed services
CI/CD for ML Automate tests, model approval, and deployment across environments. SageMaker Pipelines, Model Registry, CodePipeline patterns
Model registry Store approved model versions and metadata. SageMaker Model Registry
Lineage Trace a model back to data, code, training job, metrics, and approvals. SageMaker lineage, Model Registry
Technical debt Hidden long-term cost from ad hoc notebooks, untracked data, manual deployments, stale models, or duplicated features. MLOps governance and automation
Production readiness Confirm model quality, safety, performance, cost, security, and rollback strategy before release. Model evaluation, deployment guardrails
Monitoring Watch production behavior and detect degradation. SageMaker Model Monitor, CloudWatch
Retraining Update the model when new data or drift makes the current model less effective. SageMaker Pipelines, Model Monitor alerts

Exam trap: MLOps is not only automation. It also includes governance, reproducibility, monitoring, cost control, and managing technical debt.

Monitoring and Drift

Monitoring area What it detects Example
Data quality Production input data differs from training data expectations. New missing values, changed ranges, unexpected categories
Model quality Prediction performance drops. Accuracy or F1 score declines after launch
Bias drift Bias patterns in predictions change over time. Outcomes become uneven across relevant groups
Feature attribution drift The importance of features changes in production. Model relies on different signals than before
Operational health Endpoint availability, latency, errors, throughput, and cost. Inference latency exceeds target
GenAI quality Responses become unhelpful, unsafe, ungrounded, or too costly. Hallucinated answer, poor retrieval, high token spend

SageMaker Model Monitor uses captured production data, baselines from training data, scheduled monitoring jobs, reports, and alerts to detect quality issues.

Exam trap: drift does not automatically mean the model is broken, but it is a signal to investigate, evaluate, and possibly retrain.

Model Evaluation Metrics

Metric Meaning Best for Exam clue
Accuracy Fraction of total predictions that are correct. Balanced classification problems "Overall percent correct"
Precision Of predicted positives, how many were actually positive. Reducing false positives "Avoid flagging legitimate users as fraud"
Recall Of actual positives, how many were found. Reducing false negatives "Do not miss true fraud or disease cases"
F1 score Harmonic mean of precision and recall. Balancing precision and recall "Need one metric for imbalanced classification"

Confusion Matrix Terms

Term Meaning Example in fraud detection
True positive Model predicted positive and it was positive. Fraud correctly flagged
False positive Model predicted positive but it was negative. Legit transaction incorrectly flagged
True negative Model predicted negative and it was negative. Legit transaction correctly allowed
False negative Model predicted negative but it was positive. Fraud incorrectly allowed

Metric formulas:

Accuracy  = (TP + TN) / (TP + TN + FP + FN)
Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
F1 score  = 2 * (precision * recall) / (precision + recall)

Exam shortcuts:

False positives are costly -> optimize precision.
False negatives are costly -> optimize recall.
Need balance -> use F1.
Balanced simple classification -> accuracy may be acceptable.

Business Metrics

Technical metrics do not prove business success by themselves.

Business metric What it answers Example
Cost per user How much the AI/ML solution costs for each served user. Monthly model/API cost divided by active users
Development cost What it costs to build, integrate, evaluate, and maintain the solution. Engineering time, labeling, infrastructure, licenses
Customer feedback Whether users trust and value the system. Ratings, complaints, adoption, support tickets
ROI Whether benefits exceed costs. Revenue lift, time saved, loss avoided minus total cost
Latency and availability Whether the system meets user expectations. Chat response time, endpoint uptime
Adoption Whether people actually use the solution. Number of active users or workflows completed
Risk reduction Whether the system reduces compliance, safety, or operational risk. Fewer manual review errors

Exam trap: the best technical model may be the wrong business choice if it is too expensive, too slow, hard to operate, or poorly accepted by users.

Metric Selection Scenarios

Scenario Metric focus
Balanced image classifier Accuracy
Fraud model where blocking good customers is expensive Precision
Fraud model where missing fraud is expensive Recall
Rare-event classification with both false positives and false negatives important F1 score
GenAI chatbot with company knowledge Correctness, groundedness, helpfulness, customer feedback, cost per conversation
Recommendation system Click-through, conversion, revenue lift, customer satisfaction
Internal assistant Time saved, task completion, user adoption, answer quality
Production endpoint Latency, errors, availability, throughput, cost

Production Readiness Checklist

Before production, be able to answer:

  • Is the business problem and success metric clear?
  • Is the training/evaluation data representative and high quality?
  • Were model candidates compared against a baseline?
  • Are technical metrics acceptable for the risk level?
  • Are business metrics such as cost, ROI, and user feedback considered?
  • Is there a deployment method that fits latency, traffic, and payload needs?
  • Is monitoring configured for data quality, model quality, drift, latency, errors, and cost?
  • Is there a retraining or rollback plan?
  • Are security, privacy, compliance, bias, and explainability requirements understood?

Exam trap: production readiness includes operational and business concerns, not only model accuracy.

Lifecycle Ordering Drills

Prompt Correct order
Build traditional ML model Define problem -> collect/prepare data -> train -> evaluate -> deploy -> monitor -> retrain
Create GenAI app with company documents Define use case -> choose FM -> create knowledge base/RAG -> evaluate answers -> deploy app/API -> monitor feedback and quality
MLOps release flow Experiment -> register model -> approve model -> deploy -> monitor -> retrain/redeploy
Model monitoring setup Enable data capture -> create baseline -> schedule monitoring -> inspect reports/alerts -> take corrective action
Metric decision Identify business risk -> choose technical metric -> evaluate model -> compare business impact -> decide production readiness

Exam Trap Table

Scenario wording Best answer
"Model is trained and deployed, so the project is complete." Incorrect. Monitor, collect feedback, and retrain as needed.
"Need a managed API for foundation models without managing infrastructure." Amazon Bedrock
"Need to build, train, deploy, and monitor a custom model." Amazon SageMaker AI
"Need reusable, automated ML workflow steps." SageMaker Pipelines
"Need to track training runs, parameters, and metrics." SageMaker Experiments
"Need to catalog model versions and approval state." SageMaker Model Registry
"Need to detect production data quality and model quality drift." SageMaker Model Monitor
"Need grounded answers from enterprise documents." Bedrock Knowledge Bases or Amazon Q Business, depending on app shape
"Need an enterprise assistant with permissions-aware answers and citations." Amazon Q Business
"Need a developer assistant for AWS apps and IDE coding help." Amazon Q Developer
"Need natural-language BI summaries and dashboard Q&A." Amazon Quick Sight / Amazon Q in Quick
"False positives are the main business risk." Optimize precision
"False negatives are the main business risk." Optimize recall
"Need balance between precision and recall." Use F1 score
"The dataset is balanced and the question is simply percent correct." Accuracy can be appropriate
"Traffic is intermittent and endpoint should not run idle." Serverless inference
"Large offline dataset is available upfront." Batch transform
"Payloads are large and requests can be queued." Asynchronous inference

One-Page Memorization Version

  • AI/ML development is a lifecycle: define, data, train/customize, evaluate, deploy, monitor, retrain.
  • Traditional ML often trains a model from data; GenAI often starts with a foundation model.
  • Bedrock is for managed foundation models and GenAI applications.
  • SageMaker AI is for building, training, deploying, monitoring, and governing ML models and FMs.
  • JumpStart provides pretrained models and templates.
  • Experiments tracks runs, parameters, datasets, metrics, and artifacts.
  • Pipelines automates repeatable ML workflows.
  • Model Registry catalogs versions, metadata, lineage, approval, and deployment state.
  • Model Monitor detects data quality, model quality, bias drift, and feature attribution drift.
  • Managed API service means less infrastructure responsibility.
  • Self-hosting gives more control but more operations work.
  • RAG retrieves knowledge at inference time; fine-tuning/customization changes model behavior.
  • Accuracy is overall correctness.
  • Precision reduces false positives.
  • Recall reduces false negatives.
  • F1 balances precision and recall.
  • Business metrics include cost per user, development cost, customer feedback, and ROI.
  • Production readiness requires monitoring, rollback/retraining plans, and business value, not just high accuracy.

Mini Practice Questions

  1. A team wants to automate data preparation, model training, model evaluation, and deployment approval in a repeatable workflow. Which SageMaker feature fits?

    • Answer: SageMaker Pipelines.

  2. A model performs well during testing but gradually becomes less accurate after production data changes. What lifecycle activity is needed?

    • Answer: Model monitoring and possible retraining. SageMaker Model Monitor is the AWS anchor.

  3. A company wants to build a GenAI application using managed foundation models without managing model infrastructure. Which service fits?

    • Answer: Amazon Bedrock.

  4. A bank's fraud model must minimize incorrectly blocking legitimate transactions. Which metric should it prioritize?

    • Answer: Precision, because false positives are costly.

  5. A security model must catch as many true threats as possible, even if analysts review some extra alerts. Which metric matters most?

    • Answer: Recall, because false negatives are costly.

  6. A team needs to store model versions, approval status, training metrics, and lineage before production deployment. Which feature fits?

    • Answer: SageMaker Model Registry.

  7. A company wants answers from enterprise documents with user-permission-aware responses and citations. Which AWS service is a strong fit?

    • Answer: Amazon Q Business.

  8. A data analyst wants to generate dashboard summaries and ask natural language questions about BI data. Which AWS capability fits?

    • Answer: Amazon Quick Sight Generative BI / Amazon Q in Quick.

  9. A team wants to score millions of historical records overnight and does not need a persistent endpoint. Which inference method fits?

    • Answer: Batch transform.

  10. A support chatbot has good model scores but users complain that answers are slow and expensive. What kind of evaluation is missing?

    • Answer: Business and operational metrics such as latency, cost per user, customer feedback, and ROI.

Sources

Official AWS sources used: