Opscraft
Running AI/ML systems · Lesson 3 of 5 · 5 min read

Serving models

Inference patterns

Pattern Example Design priorities
Real-time Fraud score on each payment, chat responses Low latency (p99), autoscaling, high availability
Batch Nightly recommendations for every user Throughput and cost; Spot; runs as jobs
Streaming Scoring events from Kafka or Kinesis Consistent throughput, back-pressure, lag monitoring
Asynchronous Video analysis, long document processing Queue requests, return results later; scale on queue depth
Enlarge diagramReal-timeper request, low latencyBatchnightly, throughput, SpotStreamingKafka or Kinesis eventsAsyncqueue, results later
Figure 1. Four ways to serve predictions.
Swipe sideways to see all of it, or tap to open it full screen.

Model servers handle loading models, batching requests and exposing HTTP or gRPC APIs:

  • KServe on Kubernetes: an InferenceService resource with autoscaling (including to zero), canary rollouts and support for many model formats.
  • NVIDIA Triton Inference Server: high-performance serving for many frameworks with dynamic batching.
  • vLLM (and similar engines such as SGLang): optimised serving for LLMs.
  • Ray Serve: Python-native serving for composing several models.
  • SageMaker endpoints (real-time, serverless, asynchronous, batch transform) as the managed option, and Amazon Bedrock for calling foundation models as an API without hosting them at all.

A common decision: call a hosted model API (Bedrock or a vendor API) when you need a large general model and moderate volume; self-host an open-weight model when you need data control, customisation, very high volume or predictable cost, and accept running GPUs.

Serving LLMs

LLM serving has its own vocabulary:

  • Tokens: text is split into tokens (roughly three-quarters of an English word). Cost, limits and speed are measured in tokens.
  • Latency metrics: time to first token (how long before the answer starts streaming) and tokens per second per request (how fast it continues), plus total throughput across all users.
  • GPU memory must hold the model weights (a 70-billion-parameter model in 16-bit precision needs about 140 GB, so it is split across several GPUs) plus the KV cache, which grows with context length and the number of concurrent requests.
  • Continuous batching (in vLLM and others) adds new requests into running batches, greatly improving throughput.
  • Quantisation stores weights in 8 or 4 bits, cutting memory and cost with a small quality loss.
  • Streaming responses over server-sent events: proxies and load balancers must not buffer them (see the Nginx topic).
Enlarge diagram70B parameters, 16-bit, on 4 × 80 GB GPUsweights ≈ 140 GBKV cache: context × requestsHeadroom left decides how many users can be served at once. Quantising weights frees space.
Figure 2. What fills GPU memory when serving an LLM: the weights, plus a KV cache that grows with context and concurrent requests.
Swipe sideways to see all of it, or tap to open it full screen.

GPUs on Kubernetes

  • Install the NVIDIA device plugin (or the full NVIDIA GPU Operator, which also manages drivers and monitoring) so nodes advertise nvidia.com/gpu. EKS-optimised accelerated AMIs (and Bottlerocket NVIDIA variants) include drivers.
  • Pods request GPUs as whole units in limits:
resources:
  limits:
    nvidia.com/gpu: 1
  • Taint GPU nodes (nvidia.com/gpu=present:NoSchedule) so ordinary pods don't land on expensive nodes, and give GPU workloads the matching toleration.
  • Karpenter NodePools for GPU instance families scale GPU nodes from zero when GPU pods are pending, and remove them when idle.
  • Sharing: a small model may not need a whole GPU. MIG (Multi-Instance GPU, on A100/H100-class GPUs) splits a GPU into isolated slices; time-slicing shares a GPU without isolation.
  • Start-up time is often minutes: huge container images (CUDA libraries) plus downloading model weights. Pre-pull images into the node AMI, keep weights on fast shared storage or local cache, and use readiness probes that wait until the model is loaded.
  • Monitor GPUs with NVIDIA DCGM exporter for Prometheus: utilisation, memory, temperature, errors. Low utilisation on expensive GPUs is the first cost problem to fix.
Enlarge diagraminference podlimits nvidia.com/gpu: 1· tolerationKarpenterGPU NodePool, fromzerog6e nodetaint nvidia.com/gpu ·device pluginDCGM exporterGPU metrics to Prometheuspending
Figure 3. GPUs on Kubernetes: tainted GPU nodes, pods that request and tolerate them, Karpenter adding nodes only when needed.
Swipe sideways to see all of it, or tap to open it full screen.