Operating a mesh
Observability
Every proxy reports request rate, errors and latency histograms (istio_requests_total, istio_request_duration_milliseconds) labelled with source and destination workload, so you get a service graph without touching application code. Kiali visualises the graph and the config; Grafana dashboards come with Istio.
Tracing needs the application's help: proxies create spans, but to link the incoming and outgoing calls into one trace the app must forward the trace headers (traceparent, b3) from each request to the calls it makes. Without that, you get disconnected one-hop traces.
Debugging
Envoy's access log has a response flags field that explains most mesh 503s:
| Flag | Meaning | Look at |
|---|---|---|
UF |
Upstream connection failure | Is the destination pod up and listening? mTLS mismatch? |
UH |
No healthy upstream | All endpoints ejected or none ready |
NR |
No route configured | VirtualService or Gateway API route host mismatch |
UO |
Upstream overflow (circuit breaker) | Connection-pool limits in the DestinationRule |
URX |
Retry limit exceeded | Underlying failures being retried |
UT |
Upstream request timeout | Route timeout too short or slow service |
UC |
Upstream connection terminated | App closed keep-alive connections early |
Useful commands:
istioctl analyze -A # config mistakes across namespaces
istioctl proxy-status # are proxies in sync with istiod?
istioctl proxy-config routes deploy/checkout # what routes this proxy actually has
istioctl proxy-config endpoints deploy/checkout --cluster "outbound|80||payments.payments.svc.cluster.local"
kubectl logs deploy/checkout -c istio-proxy # access log with response flags
A common mistake: a DestinationRule that sets tls.mode: DISABLE or the wrong subset labels, giving 503s with UF or NR only for one route.
Upgrades and cost
- Upgrade the control plane with revisions (canary upgrades): install the new
istiodrevision next to the old one, move namespaces to it by label, restart workloads (sidecar mode) to pick up the new proxy, then remove the old revision. Ambient mode upgrades ztunnel as a DaemonSet instead. - Track the mesh's own resource use: sidecar memory scales with the number of services each proxy knows about, so scope configuration.
- Keep CRDs and mesh config in Git with GitOps, and validate with
istioctl analyzein CI.
When not to use a mesh
A mesh adds a critical component that can break every call at once, a learning curve and per-pod overhead. Consider simpler options first when:
- You have a handful of services: libraries, an ingress controller and network policies are enough.
- The main need is encryption in transit: Cilium or a CNI with WireGuard encryption, or TLS at the app, may do.
- The main need is canary releases: Argo Rollouts with an ingress controller or Gateway API handles traffic splitting without a mesh.
Adopt a mesh when you have many services across teams and a real need for uniform mTLS identity, L7 policy and traffic control, and a platform team ready to own it. Ambient mode lowers the cost of starting with just mTLS.