Platform, cost & AI
Running AI/ML systems
More and more DevOps and platform roles now include running machine-learning workloads: GPU node pools, model serving endpoints, pipelines that retrain models, and applications built on large language models (LLMs). You don't need to be a data scientist, but interviewers expect you to know what is different about ML systems, how models are trained, versioned and served, how GPUs work on Kubernetes, how to monitor models as well as servers, and the security and cost traps. This topic covers those basics for AWS and Kubernetes.
After this topic you can
- Explain how ML systems differ from normal services
- Run GPU workloads and model serving on EKS
- Deploy, monitor and secure models and LLM applications
Lessons
-
1What is different about ML systems
- The ML lifecycle
- Why it's harder to operate
- Who does what
-
2Training and the model lifecycle
- Tracking, versioning and pipelines
- Compute for training
-
3Serving models
- Inference patterns
- Serving LLMs
- GPUs on Kubernetes
-
4Monitoring, releases and safety
- Monitoring models, not just servers
- Releasing models safely
- Security and governance
- Retrieval-augmented generation (RAG)
-
5Interview questions and model answers
- "What's different about operating ML systems compared with normal services?"
- "How would you set up GPU workloads on EKS?"
- "How do you deploy a new model version safely?"
- "What is data drift and how do you detect it?"
- "Hosted LLM API or self-hosted model?"
- "What metrics matter for an LLM endpoint?"
- "What are the main security risks in an LLM application?"