GitOps with Flux & Argo CD · Lesson 5 of 6 · 4 min read
Secrets, security and operations
Secrets in GitOps
| Option | How | Notes |
|---|---|---|
| External Secrets Operator | ExternalSecret in Git references AWS Secrets Manager/SSM; ESO syncs a K8s Secret using IRSA |
Secret values never in Git; rotation handled upstream — a common default on AWS |
| SOPS | Encrypt values in YAML with KMS/age; Flux decrypts at apply | Encrypted in Git; needs key management |
| Sealed Secrets | Encrypt with the controller's public key; only the cluster can decrypt | Cluster-specific; key backup needed |
| CSI Secrets Store driver | Mount secrets from Secrets Manager into pods | No K8s Secret needed (optionally synced) |
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Security of the GitOps pipeline
- Git is now the deployment control plane: branch protection, required reviews, CODEOWNERS per path (platform owns
infrastructure/, teams own their app dirs), signed commits where required. - Flux can verify commit signatures (GPG) and OCI artifact signatures (cosign) before applying.
- Read-only deploy keys for clusters; CI gets write access only to the paths it updates (or uses PRs).
- Least-privilege controller RBAC and tenant impersonation; admission policies to block risky manifests even if they're merged.
- Disaster recovery: a new cluster bootstraps from Git and converges — state that isn't in Git (PV data, secrets in external stores) is backed up separately (Velero, snapshots).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Troubleshooting
| Symptom | Check |
|---|---|
| Change merged but not deployed | flux get sources git — did the source fetch the new revision? Auth/deploy key? Path ignored? Then the Kustomization status/events |
Kustomization False with build error |
kustomize build the path locally; missing resource, bad patch target |
| Apply fails on CRD kind | dependsOn order — controller/CRDs not yet installed |
| Health check timeout | kubectl describe the workload — image pull errors, failing probes, insufficient resources; increase timeout only if genuinely slow |
| HelmRelease failed | flux get hr -A, kubectl describe hr, helm history — values errors, hooks failing; remediation settings |
| Resource keeps reverting | That's drift correction — change it in Git, or exclude the field (HPA replicas: omit replicas from manifests) |
| Accidentally deleted manifests from Git | Prune deleted the resources — revert the commit; protect critical objects with prune-disabled annotation; PVC reclaim policies |
| Argo app OutOfSync forever | Controller-mutated fields — ignoreDifferences, ServerSideApply, or normalise manifests |
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.