Observability: metrics, logs, traces · Lesson 4 of 7 · 3 min read
Distributed tracing and OpenTelemetry
Concepts
- Trace = tree of spans for one request; a span has name, start/end, attributes, status, events, parent.
- Context propagation — the W3C
traceparentheader carries trace and span IDs between services (and through queues via message attributes). - Sampling: head-based (decide at the start, e.g. 10%) is cheap but may miss rare errors; tail-based (decide after seeing the whole trace, keep all errors and slow traces) needs a collector tier that buffers traces.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
OpenTelemetry (OTel)
The CNCF standard for generating and shipping telemetry:
- APIs/SDKs per language, plus auto-instrumentation (Java agent, Python, Node, .NET) for HTTP, DB, messaging libraries — in Kubernetes, the OTel Operator injects agents via an
Instrumentationresource and a pod annotation. - OTLP protocol for traces, metrics and logs.
- Collector — receivers → processors (batch, memory_limiter, k8sattributes, tail_sampling, redaction) → exporters (Tempo, Jaeger, X-Ray, Prometheus remote_write, Loki, vendors). Deploy as agent (DaemonSet/sidecar) and gateway (Deployment).
receivers:
otlp: { protocols: { grpc: {}, http: {} } }
processors:
memory_limiter: { check_interval: 1s, limit_percentage: 80 }
k8sattributes: {}
batch: {}
tail_sampling:
policies:
- { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
- { name: slow, type: latency, latency: { threshold_ms: 1000 } }
- { name: sample, type: probabilistic, probabilistic: { sampling_percentage: 10 } }
exporters:
otlp/tempo: { endpoint: tempo-distributor:4317, tls: { insecure: true } }
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, tail_sampling, batch]
exporters: [otlp/tempo]
Backends: Grafana Tempo, Jaeger, AWS X-Ray (via ADOT — AWS Distro for OpenTelemetry), commercial APMs (Datadog, New Relic, Dynatrace, Elastic APM). Vendor-neutral instrumentation with OTel avoids lock-in.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.