Opscraft
Observability: metrics, logs, traces · Lesson 3 of 7 · 3 min read

Logging in depth

Principles

  • Log to stdout/stderr in containers; the platform collects.
  • Structured logs (JSON) with consistent fields: timestamp, level, service, environment, trace_id/span_id, request ID, user-safe identifiers — not free-text only.
  • Never log secrets or PII (tokens, passwords, card numbers); mask at source, and add redaction in the pipeline as a safety net.
  • Log levels with discipline; avoid DEBUG floods in prod; sample high-volume success logs.
  • Retention by value: hot searchable days, cheap archive (S3) for compliance.

Kubernetes log pipeline

Enlarge diagramcontainerstdoutJSON lines/var/log/podsfiles on the nodeFluent BitDaemonSet:parse, enrich,bufferbackendLoki, OpenSearch,CloudWatch, S3
Figure 1. The Kubernetes log pipeline: apps write to stdout, the node keeps files, a Fluent Bit pod on every node ships them on with Kubernetes labels attached.
Swipe sideways to see all of it, or tap to open it full screen.

Fluent Bit (lightweight C) is the usual node agent; Fluentd or Vector/OTel Collector for heavier aggregation. Key settings: Kubernetes filter (pod labels, namespace), multiline parsing (Java stack traces), memory/filesystem buffering and retry for backpressure, throttling noisy namespaces.

Backends compared

Loki OpenSearch/Elasticsearch CloudWatch Logs
Indexing Only labels indexed; content scanned with LogQL at query time Full-text inverted index Managed; Logs Insights query language
Cost Low (object storage, small index) Higher (index storage/compute) Pay per GB ingested + stored + queried
Strength Cheap, Grafana-native, Prometheus-like labels Powerful search/analytics Zero ops, native AWS integration, subscription filters, metric filters
Pitfall High-cardinality labels kill it Cluster ops, shard sizing Costs at volume; set retention

LogQL example:

sum by (status) (count_over_time({namespace="orders", app="orders-api"} | json | status >= 500 [5m]))

CloudWatch Logs Insights example:

fields @timestamp, @message
| filter level = "ERROR" and service = "orders"
| stats count(*) by bin(5m)