Opscraft
Amazon EKS · Lesson 6 of 11 · 4 min read

Storage, add-ons and observability

Storage

  • EBS CSI driver (EKS add-on, needs an IAM role): block storage for StatefulSets. gp3 StorageClass with WaitForFirstConsumer (volume created in the pod's AZ), encryption, expansion, snapshots (VolumeSnapshot). EBS is single-AZ and ReadWriteOnce.
  • EFS CSI driver: shared NFS file storage, ReadWriteMany, multi-AZ — for shared content, ML datasets, legacy apps needing shared files. Dynamic provisioning creates access points. Higher latency than EBS.
  • FSx (Lustre for HPC/ML, NetApp ONTAP) and S3 Mountpoint CSI for object storage use cases.
Enlarge diagramEBS (gp3)EFSBlock disk, one node at a timeLives in one AZFast: databases, KafkaStatefulSets + WaitForFirstConsumerShared NFS, ReadWriteManyMulti-AZHigher latencyShared content, ML datasets
Figure 1. EBS or EFS for pods.
Swipe sideways to see all of it, or tap to open it full screen.

Managed add-ons

EKS add-ons install and upgrade core components with AWS-validated versions: VPC CNI, CoreDNS, kube-proxy, EBS CSI, EFS CSI, Pod Identity Agent, CloudWatch Observability, ADOT, GuardDuty agent, Snapshot controller. Pin add-on versions per cluster version and upgrade them as part of every cluster upgrade.

Enlarge diagramVPC CNICoreDNSkube-proxyEBS CSIEFS CSIPod Identity AgentCloudWatchObservabilityGuardDuty agent
Figure 2. Core pieces EKS can install and upgrade for you as managed add-ons.
Swipe sideways to see all of it, or tap to open it full screen.

Observability

  • Control plane logs to CloudWatch: api, audit, authenticator, controllerManager, scheduler — enable at least audit and authenticator for security investigations ("who deleted this?" — correlate with CloudTrail for AWS API calls).
  • Container Insights (CloudWatch Observability add-on) — metrics and logs for clusters, nodes, pods.
  • Prometheus/Grafana — self-managed or Amazon Managed Service for Prometheus (AMP) + Amazon Managed Grafana (AMG); scrape with Prometheus agent/ADOT.
  • Logs — Fluent Bit DaemonSet to CloudWatch Logs, OpenSearch, or S3/Loki.
  • Tracing — OpenTelemetry (ADOT) to X-Ray or other backends.
Enlarge diagramControl plane logsaudit, authenticator → CloudWatchMetricsPrometheus / AMP + GrafanaLogsFluent Bit → CloudWatch, OpenSearch, LokiTracesOpenTelemetry (ADOT) → X-Ray
Figure 3. Observability on EKS: four signals and where they usually go.
Swipe sideways to see all of it, or tap to open it full screen.

Secrets

  • Enable KMS envelope encryption for Kubernetes Secrets on the cluster.
  • Source secrets from AWS Secrets Manager / SSM Parameter Store via External Secrets Operator (syncs into Kubernetes Secrets; works nicely with GitOps since only the ExternalSecret reference is in Git) or the Secrets Store CSI driver with the AWS provider (mounts as files).
  • For GitOps-managed secrets in Git, SOPS with a KMS key (Flux decrypts natively).
Enlarge diagramSecrets Managerthe real valueExternal Secretsoperator, uses IRSAKubernetes Secretencrypted with KMSpodenv or fileOnly the ExternalSecret reference lives in Git.
Figure 4. Getting secrets into pods without putting them in Git: External Secrets copies them from Secrets Manager.
Swipe sideways to see all of it, or tap to open it full screen.

GitOps on EKS

Flux runs in the cluster and pulls from Git. On EKS: Flux controllers authenticate to ECR (image automation, OCI artifacts) and S3/KMS (SOPS decryption) with IRSA/Pod Identity; the source of truth is a Git repo with Kustomize bases and overlays per environment/cluster; Flux Kustomizations per cluster define what's applied, with dependsOn ordering (CRDs and controllers before apps).