Running AI/ML systems · Lesson 4 of 5 · 5 min read
Monitoring, releases and safety
Monitoring models, not just servers
| Layer | What to watch |
|---|---|
| Service | Request rate, errors, latency percentiles, queue depth, autoscaling |
| Infrastructure | GPU utilisation and memory, node health, model load time |
| Data | Input schema violations, missing values, data drift (input distributions moving away from training data) |
| Model quality | Prediction distribution, confidence, accuracy once true labels arrive (often days later), business KPIs |
| LLM apps | Tokens and cost per request, time to first token, refusal and error rates, quality scores from evaluation sets, user feedback |
Data drift and falling quality trigger investigation or retraining. Log inputs and outputs (with care for personal data) so you can debug specific bad predictions and build evaluation sets.
Releasing models safely
Models are released with the same care as code, plus some ML-specific techniques:
- Offline evaluation gates: a new model must beat the current one on a held-out dataset or an evaluation suite before deployment.
- Shadow deployment: the new model receives a copy of live traffic and its predictions are logged but not used, so you compare with production at no risk.
- Canary and A/B tests: send a small share of real traffic to the new model and compare business metrics.
- Rollback means pointing the endpoint back to the previous registry version, so keep it deployable.
- Training/serving skew: features computed differently in training and in production cause silent quality loss. A feature store (Feast, SageMaker Feature Store) serves the same feature definitions to both.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Security and governance
- Model artifacts are supply chain: Python
pickle-based model files can execute code when loaded. Prefer safetensors or other safe formats, load models only from your registry, scan and sign artifacts, and pin versions of models pulled from public hubs. - Data protection: training data and prompts may contain personal data; control access, encrypt, minimise, and keep logs of prompts and outputs under the same rules.
- Prompt injection: text from users, documents or web pages can contain instructions that hijack an LLM application. Treat model output as untrusted, give the application (not the model) least-privilege permissions, validate outputs before acting, and keep humans in the loop for risky actions.
- Access and cost control: authenticate every model endpoint, set per-team quotas and rate limits on token usage, and track cost per feature or customer.
- Governance: record which model version made which decision, with its training data lineage; regulations increasingly require this for higher-risk uses.
Retrieval-augmented generation (RAG)
Many LLM applications use RAG: instead of retraining a model on company data, the app retrieves relevant documents and puts them in the prompt.
- Ingest: split documents into chunks and compute embeddings (numeric vectors representing meaning) with an embedding model.
- Store the vectors in a vector database: pgvector in PostgreSQL, OpenSearch, or a dedicated store.
- Query: embed the user's question, find the nearest chunks, add them to the prompt, and call the LLM.
Operational concerns: keeping the index fresh as documents change, enforcing document permissions at retrieval time (users must not get answers from documents they can't see), monitoring retrieval quality, and the cost and latency of each step.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.