Opscraft
Running AI/ML systems · Lesson 4 of 5 · 5 min read

Monitoring, releases and safety

Monitoring models, not just servers

Layer What to watch
Service Request rate, errors, latency percentiles, queue depth, autoscaling
Infrastructure GPU utilisation and memory, node health, model load time
Data Input schema violations, missing values, data drift (input distributions moving away from training data)
Model quality Prediction distribution, confidence, accuracy once true labels arrive (often days later), business KPIs
LLM apps Tokens and cost per request, time to first token, refusal and error rates, quality scores from evaluation sets, user feedback

Data drift and falling quality trigger investigation or retraining. Log inputs and outputs (with care for personal data) so you can debug specific bad predictions and build evaluation sets.

Releasing models safely

Models are released with the same care as code, plus some ML-specific techniques:

  • Offline evaluation gates: a new model must beat the current one on a held-out dataset or an evaluation suite before deployment.
  • Shadow deployment: the new model receives a copy of live traffic and its predictions are logged but not used, so you compare with production at no risk.
  • Canary and A/B tests: send a small share of real traffic to the new model and compare business metrics.
  • Rollback means pointing the endpoint back to the previous registry version, so keep it deployable.
  • Training/serving skew: features computed differently in training and in production cause silent quality loss. A feature store (Feast, SageMaker Feature Store) serves the same feature definitions to both.
Enlarge diagramEvaluatebeats current modelofflineShadowcopy of live traffic,not usedCanary5% of real usersFull rolloutkeep v30 ready1234
Figure 1. Releasing a model: evaluation gate, shadow traffic, canary, then full rollout — with rollback to the previous registry version.
Swipe sideways to see all of it, or tap to open it full screen.

Security and governance

  • Model artifacts are supply chain: Python pickle-based model files can execute code when loaded. Prefer safetensors or other safe formats, load models only from your registry, scan and sign artifacts, and pin versions of models pulled from public hubs.
  • Data protection: training data and prompts may contain personal data; control access, encrypt, minimise, and keep logs of prompts and outputs under the same rules.
  • Prompt injection: text from users, documents or web pages can contain instructions that hijack an LLM application. Treat model output as untrusted, give the application (not the model) least-privilege permissions, validate outputs before acting, and keep humans in the loop for risky actions.
  • Access and cost control: authenticate every model endpoint, set per-team quotas and rate limits on token usage, and track cost per feature or customer.
  • Governance: record which model version made which decision, with its training data lineage; regulations increasingly require this for higher-risk uses.

Retrieval-augmented generation (RAG)

Many LLM applications use RAG: instead of retraining a model on company data, the app retrieves relevant documents and puts them in the prompt.

  1. Ingest: split documents into chunks and compute embeddings (numeric vectors representing meaning) with an embedding model.
  2. Store the vectors in a vector database: pgvector in PostgreSQL, OpenSearch, or a dedicated store.
  3. Query: embed the user's question, find the nearest chunks, add them to the prompt, and call the LLM.

Operational concerns: keeping the index fresh as documents change, enforcing document permissions at retrieval time (users must not get answers from documents they can't see), monitoring retrieval quality, and the cost and latency of each step.

Enlarge diagramquestionfrom the userembedto a vectorvector DBnearest chunks,permission-filteredLLMquestion + chunksanswerwith sources
Figure 2. Retrieval-augmented generation: find relevant document chunks first, then ask the model with them in the prompt.
Swipe sideways to see all of it, or tap to open it full screen.