Opscraft
GitOps with Flux & Argo CD · Lesson 5 of 6 · 4 min read

Secrets, security and operations

Secrets in GitOps

Option How Notes
External Secrets Operator ExternalSecret in Git references AWS Secrets Manager/SSM; ESO syncs a K8s Secret using IRSA Secret values never in Git; rotation handled upstream — a common default on AWS
SOPS Encrypt values in YAML with KMS/age; Flux decrypts at apply Encrypted in Git; needs key management
Sealed Secrets Encrypt with the controller's public key; only the cluster can decrypt Cluster-specific; key backup needed
CSI Secrets Store driver Mount secrets from Secrets Manager into pods No K8s Secret needed (optionally synced)
Enlarge diagramExternal SecretsGit holds a reference; value in Secrets ManagerSOPSencrypted in Git with KMS; Flux decryptsSealed Secretsonly this cluster can decryptCSI Secrets Storemounted straight into pods
Figure 1. Keeping secrets out of plain Git: the common options.
Swipe sideways to see all of it, or tap to open it full screen.

Security of the GitOps pipeline

  • Git is now the deployment control plane: branch protection, required reviews, CODEOWNERS per path (platform owns infrastructure/, teams own their app dirs), signed commits where required.
  • Flux can verify commit signatures (GPG) and OCI artifact signatures (cosign) before applying.
  • Read-only deploy keys for clusters; CI gets write access only to the paths it updates (or uses PRs).
  • Least-privilege controller RBAC and tenant impersonation; admission policies to block risky manifests even if they're merged.
  • Disaster recovery: a new cluster bootstraps from Git and converges — state that isn't in Git (PV data, secrets in external stores) is backed up separately (Velero, snapshots).
Enlarge diagramBranch protectionreviews, CODEOWNERS per pathSigned commitsFlux can verify themRead-only deploy keysclusters only readScoped CI writesonly its paths, or PRsAdmission policiesblock risky manifests anywayBackupsdata not in Git: Velero
Figure 2. Git is now the control plane for production, so protect it like one.
Swipe sideways to see all of it, or tap to open it full screen.

Troubleshooting

Symptom Check
Change merged but not deployed flux get sources git — did the source fetch the new revision? Auth/deploy key? Path ignored? Then the Kustomization status/events
Kustomization False with build error kustomize build the path locally; missing resource, bad patch target
Apply fails on CRD kind dependsOn order — controller/CRDs not yet installed
Health check timeout kubectl describe the workload — image pull errors, failing probes, insufficient resources; increase timeout only if genuinely slow
HelmRelease failed flux get hr -A, kubectl describe hr, helm history — values errors, hooks failing; remediation settings
Resource keeps reverting That's drift correction — change it in Git, or exclude the field (HPA replicas: omit replicas from manifests)
Accidentally deleted manifests from Git Prune deleted the resources — revert the commit; protect critical objects with prune-disabled annotation; PVC reclaim policies
Argo app OutOfSync forever Controller-mutated fields — ignoreDifferences, ServerSideApply, or normalise manifests
Enlarge diagramSource fetched?flux get sources git:new revision?Build OK?kustomize build thepath locallyApplied?Kustomization Ready,dependsOn orderHealthy?describe the workload:image, probes1234
Figure 3. "I merged it but nothing changed": follow the chain from Git to the pod.
Swipe sideways to see all of it, or tap to open it full screen.