Opscraft
Helm & Kustomize · Lesson 4 of 7 · 5 min read

Releases, hooks, tests and operations

How upgrades work

Helm 3 uses a three-way strategic merge: it compares the old manifest, the new manifest and the live state, so manual kubectl edit changes to fields Helm manages are overwritten, and fields Helm never set are preserved. Helm 4 can instead use server-side apply, where the API server records which tool owns each field and reports conflicts.

Enlarge diagramold manifestreplicas: 3new manifestreplicas: 4live objectreplicas: 10 (kubectl edit)helm upgradethree-way mergeresultreplicas: 4
Figure 1. Helm's three-way merge: it looks at the old chart output, the new one and the live object. A manual edit to a field the chart owns is overwritten.
Swipe sideways to see all of it, or tap to open it full screen.

Release states: deployed, superseded, failed, pending-install, pending-upgrade, pending-rollback, uninstalled.

Stuck "another operation is in progress" — a previous helm process died mid-upgrade, leaving the release pending-upgrade. Fix: helm history, then helm rollback <release> <last-good-revision>; as a last resort delete the latest pending release Secret (sh.helm.release.v1.<name>.v<N>).

Hooks

Annotate resources (usually Jobs) to run at lifecycle points:

metadata:
  annotations:
    "helm.sh/hook": pre-upgrade,pre-install
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded

Hook types: pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, post-delete, pre-rollback, post-rollback, test. Typical use: DB migrations before upgrade. Hook resources aren't managed as part of the release (not deleted on uninstall unless the policy says so). With GitOps controllers, Argo CD maps Helm hooks to its own sync hooks; Flux's helm-controller runs them normally.

helm test <release> runs pods annotated helm.sh/hook: test — smoke tests after deploy.

Enlarge diagramhelm upgradepre-upgradehookmigration JobapplymanifestsDeployment rollspost-upgradehooknotify, warmcachehelm testsmoke-test pods
Figure 2. Hooks run at fixed points in a release. The classic use: a database migration Job before the new pods start.
Swipe sideways to see all of it, or tap to open it full screen.

CRDs

Charts can ship CRDs in a crds/ directory: installed on first install, never upgraded or deleted by Helm (to avoid data loss). Many operators therefore install CRDs separately or via a template with a flag. Know this — it's a common upgrade surprise.

Enlarge diagramcrds/ folderWhat teams doInstalled on first installNever upgraded by helm upgradeNever deleted (protects data)Install CRDs separately firstOr a chart flag to template themFlux: CRDs in their own Kustomization
Figure 3. CRDs in a chart's crds/ folder are installed once and never upgraded or deleted by Helm.
Swipe sideways to see all of it, or tap to open it full screen.

OCI registries

helm package ./orders                         # orders-1.6.0.tgz
aws ecr get-login-password --region ap-south-1 | \
  helm registry login --username AWS --password-stdin 123456789012.dkr.ecr.ap-south-1.amazonaws.com
helm push orders-1.6.0.tgz oci://123456789012.dkr.ecr.ap-south-1.amazonaws.com/charts
helm install orders oci://123456789012.dkr.ecr.ap-south-1.amazonaws.com/charts/orders --version 1.6.0

OCI is the modern default — same registry, auth and lifecycle as images; charts can be signed (cosign) and scanned.

Enlarge diagramhelm packageorders-1.6.0.tgzhelm pushoci://…/chartsECRcharts besideimageshelm install /Flux--version 1.6.0
Figure 4. Charts in an OCI registry use the same registry, login and lifecycle as container images.
Swipe sideways to see all of it, or tap to open it full screen.

Testing and quality in CI

  • helm lint and helm template ... | kubeconform -strict (schema validation against the target Kubernetes version).
  • chart-testing (ct lint, ct install) in a kind cluster for changed charts; enforces version bumps.
  • helm-unittest for template unit tests.
  • Policy checks on rendered output (Conftest/OPA, Kyverno CLI) and security scanners (Checkov, kube-score, Trivy config).
  • values.schema.json to reject bad values early.
  • helm diff in PRs to show exactly what changes.
Enlarge diagramlint + schemahelm lint, values.schema.jsonrender + validatehelm template | kubeconformunit testshelm-unittestpolicyConftest, Kyverno CLIinstall testct install in kinddiffhelm diff on the PR123456
Figure 5. Checks for charts in a pull request, from cheap to thorough.
Swipe sideways to see all of it, or tap to open it full screen.

Secrets with Helm

Don't put secrets in values files in Git. Options: External Secrets Operator (chart creates an ExternalSecret pointing at AWS Secrets Manager), Sealed Secrets, SOPS (helm-secrets plugin or Flux's native SOPS decryption), or CSI Secrets Store driver.