Opscraft
Senior DevOps interviews · Lesson 6 of 10 · 4 min read

DevOps system design: worked outlines

CI/CD for microservices on EKS

Enlarge diagramcheckslint, tests, SAST, secretsbuild imagetag = git SHAscanTrivy / Inspectorpush to ECRimmutable tagsupdate GitOps repostaging overlayFlux: stagingsmoke testspromotion PRreviewedFlux: prodcanary + auto rollbackPull requestMerge
Figure 1. A whiteboard answer for CI/CD for microservices on EKS: CI builds and scans once, Git holds what runs, Flux deploys, and canaries guard production.
Swipe sideways to see all of it, or tap to open it full screen.

Talking points: build once/promote the same digest, OIDC to AWS (no static keys), reusable workflow/shared library templates, policy checks (conftest/Kyverno), DORA metrics (deploy frequency, lead time, change failure rate, MTTR).

Logging and monitoring for EKS

  • Metrics: Prometheus (or AMP) + Grafana; kube-state-metrics, node-exporter; app RED metrics (rate, errors, duration); four golden signals — latency, traffic, errors, saturation.
  • Logs: Fluent Bit DaemonSet → CloudWatch Logs/OpenSearch/Loki; structured JSON logs with request IDs; retention policies; control plane audit logs.
  • Traces: OpenTelemetry → X-Ray/Tempo/Jaeger.
  • Alerting: SLO-based burn-rate alerts routed to on-call (PagerDuty/Opsgenie) with runbook links; dashboards per service.

Multi-account AWS foundation

AWS Organizations with OUs (security, infrastructure, workloads prod/non-prod, sandbox), accounts per environment/team, SCPs as guardrails, IAM Identity Center SSO, centralised CloudTrail and Config in a log-archive account, GuardDuty/Security Hub delegated to a security account, networking hub with Transit Gateway and centralised egress/endpoints, IPAM for non-overlapping CIDRs, Terraform pipelines with per-account roles.

Backup and DR

Define RPO (acceptable data loss) and RTO (acceptable downtime) per service first. Then: RDS automated backups/PITR, snapshots copied cross-region, Aurora Global Database for low RPO; S3 versioning and cross-region replication; EKS rebuilt from Terraform + Flux (cluster state is code); Velero for persistent volumes; Route 53 failover or ARC routing controls; regular restore and failover tests — an untested backup isn't a backup.

Enlarge diagramdisasterRPO: data you can loseRTO: time until service is backlast good backup ←→ service restored
Figure 2. For backup and DR questions, start by agreeing RPO and RTO; they decide everything else.
Swipe sideways to see all of it, or tap to open it full screen.

Secrets for 50 services

Central secret manager (Secrets Manager), per-service IAM access via IRSA/Pod Identity, External Secrets Operator syncing into namespaces, automatic rotation (RDS integration, Lambda rotators), no secrets in Git (SOPS for the few that must be), secret scanning in CI, audit via CloudTrail.

Capacity planning for a sale event

Load test with expected peak × safety factor, pre-scale (HPA min replicas, Karpenter limits, RDS instance size, read replicas), check service quotas (EC2 vCPUs, ALB, Lambda concurrency), CDN caching for static and cacheable API responses, queue-based buffering for spikes, feature flags to shed non-critical features, a war-room runbook and on-call plan.

Improving developer experience as a platform engineer

Self-service golden paths (service templates with CI/CD, observability and security built in), clear docs, fast CI, ephemeral preview environments, a developer portal (Backstage), clear ownership, and measuring developer satisfaction and lead time.