Opscraft
Running AI/ML systems · Lesson 2 of 5 · 3 min read

Training and the model lifecycle

Tracking, versioning and pipelines

  • Experiment tracking records each training run's parameters, metrics and artifacts so results can be compared and reproduced: MLflow, Weights & Biases, SageMaker Experiments.
  • A model registry stores versioned, approved models with stages (staging, production) and lineage: MLflow Model Registry, SageMaker Model Registry. Deployments reference a registry version, never a file on someone's laptop.
  • Data versioning: training data snapshots in S3 with versioning, or tools like DVC and table formats like Apache Iceberg, so you know exactly which data trained which model.
  • Pipelines automate the steps — fetch data, validate it, train, evaluate, register — on a schedule or trigger: Kubeflow Pipelines, Airflow, SageMaker Pipelines, Argo Workflows.
  • Reproducibility: pin library versions in container images, record random seeds, code commit and data version with every model.
Enlarge diagramdata v12S3 versionedpipelinevalidate, train,evaluateregistrymodel v31endpointserves v31approvedEvery model version records its data version, code commit and metrics.
Figure 1. Training runs are tracked; only evaluated, approved models enter the registry, and deployments reference registry versions.
Swipe sideways to see all of it, or tap to open it full screen.

Compute for training

  • GPUs do the heavy maths. On AWS: p5/p5e/p6 families (NVIDIA H100, H200, B200) for large training, g5/g6/g6e (A10G, L4, L40S) for smaller training and inference, and AWS's own Trainium (trn1, trn2) for training and Inferentia (inf2) for inference.
  • Capacity for large GPUs is limited: use On-Demand Capacity Reservations or EC2 Capacity Blocks for ML for planned large jobs, and check quotas well in advance.
  • Spot works for training if the job checkpoints regularly to S3 and can resume after interruption; it can cut cost dramatically.
  • Data throughput often limits GPU utilisation: GPUs sit idle waiting for data. Use fast storage (FSx for Lustre linked to S3, or local NVMe), parallel data loading and the right file formats.
  • Distributed training spreads one job across many GPUs and nodes; it needs fast networking between nodes (EFA on AWS) and a scheduler that starts all workers together.
  • Managed option: SageMaker training jobs run your container on the instances you choose and shut them down afterwards; on Kubernetes, the Kubeflow Training Operator or Ray do the same.
Enlarge diagramS3training dataFSx for Lustre /NVMefast local readsdata loadersparallel, prefetchGPUskept near 100%busy
Figure 2. GPUs are often idle waiting for data. Fast storage and parallel loading keep them busy.
Swipe sideways to see all of it, or tap to open it full screen.