Running AI/ML systems · Lesson 2 of 5 · 3 min read
Training and the model lifecycle
Tracking, versioning and pipelines
- Experiment tracking records each training run's parameters, metrics and artifacts so results can be compared and reproduced: MLflow, Weights & Biases, SageMaker Experiments.
- A model registry stores versioned, approved models with stages (staging, production) and lineage: MLflow Model Registry, SageMaker Model Registry. Deployments reference a registry version, never a file on someone's laptop.
- Data versioning: training data snapshots in S3 with versioning, or tools like DVC and table formats like Apache Iceberg, so you know exactly which data trained which model.
- Pipelines automate the steps — fetch data, validate it, train, evaluate, register — on a schedule or trigger: Kubeflow Pipelines, Airflow, SageMaker Pipelines, Argo Workflows.
- Reproducibility: pin library versions in container images, record random seeds, code commit and data version with every model.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Compute for training
- GPUs do the heavy maths. On AWS:
p5/p5e/p6families (NVIDIA H100, H200, B200) for large training,g5/g6/g6e(A10G, L4, L40S) for smaller training and inference, and AWS's own Trainium (trn1,trn2) for training and Inferentia (inf2) for inference. - Capacity for large GPUs is limited: use On-Demand Capacity Reservations or EC2 Capacity Blocks for ML for planned large jobs, and check quotas well in advance.
- Spot works for training if the job checkpoints regularly to S3 and can resume after interruption; it can cut cost dramatically.
- Data throughput often limits GPU utilisation: GPUs sit idle waiting for data. Use fast storage (FSx for Lustre linked to S3, or local NVMe), parallel data loading and the right file formats.
- Distributed training spreads one job across many GPUs and nodes; it needs fast networking between nodes (EFA on AWS) and a scheduler that starts all workers together.
- Managed option: SageMaker training jobs run your container on the instances you choose and shut them down afterwards; on Kubernetes, the Kubeflow Training Operator or Ray do the same.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.