Opscraft
Running AI/ML systems · Lesson 1 of 5 · 3 min read

What is different about ML systems

More and more DevOps and platform roles now include running machine-learning workloads: GPU node pools, model serving endpoints, pipelines that retrain models, and applications built on large language models (LLMs). You don't need to be a data scientist, but interviewers expect you to know what is different about ML systems, how models are trained, versioned and served, how GPUs work on Kubernetes, how to monitor models as well as servers, and the security and cost traps. This topic covers those basics for AWS and Kubernetes.

The ML lifecycle

A traditional service is code plus configuration. An ML system is code plus data plus a trained model, and the model's behaviour comes from the data it learned from, not from rules someone wrote. Its life cycle:

  1. Collect and prepare data: raw data is cleaned, labelled and turned into features (inputs the model uses).
  2. Train: an algorithm fits model weights to the training data, often on GPUs, for minutes to weeks.
  3. Evaluate: measure accuracy (or other metrics) on data the model hasn't seen; compare with the current production model.
  4. Register: store the approved model artifact with its version, metrics and the data and code that produced it.
  5. Deploy: serve it for real-time predictions, batch jobs or streaming.
  6. Monitor: watch both system health and model quality; when quality drops because the world changed, retrain.

MLOps applies DevOps practices — version control, automation, CI/CD, monitoring, reproducibility — to this loop.

Enlarge diagramPrepare dataclean, label, featuresTrainGPUs, minutes to weeksEvaluatebeat the current model?Registerversion, metrics, lineageDeployreal-time, batch, streamMonitorhealth, drift, quality → retrain123456
Figure 1. The ML lifecycle is a loop: production monitoring feeds back into new data and retraining.
Swipe sideways to see all of it, or tap to open it full screen.

Why it's harder to operate

  • Silent failure: a model can return confident, wrong answers with HTTP 200 and normal latency. Server metrics alone won't catch it.
  • Data dependence: a change upstream (a renamed field, a new currency, a different camera) can break a model with no code change at all.
  • Expensive hardware: GPUs cost many times more than CPUs, are scarce in some regions and must be kept busy.
  • Large artifacts: model files from hundreds of megabytes to hundreds of gigabytes, which affects image sizes, start-up time and storage.
  • Non-determinism: retraining on the same data can produce a slightly different model, and LLM outputs vary between calls.
Enlarge diagramWhat dashboards showWhat is really happeningHTTP 200 on every requestp99 latency 80 msno errors in logsupstream field changed unitsmodel predictions shiftedfraud slips through
Figure 2. Why model failures are silent: server metrics look healthy while answers get worse.
Swipe sideways to see all of it, or tap to open it full screen.

Who does what

Role Focus
Data scientist Explores data, chooses and trains models, evaluates them
ML engineer Turns models into production pipelines and services
DevOps / platform engineer GPU infrastructure, clusters, pipelines, serving platform, observability, security, cost

In interviews, position yourself clearly: you build and run the platform that makes training and serving reliable, repeatable and affordable.