Opscraft
Kubernetes in depth · Lesson 10 of 11 · 3 min read

Extending Kubernetes and operating clusters

CRDs and operators

  • CustomResourceDefinitions add new API types (Certificate, Kustomization, HelmRelease, Rollout, ExternalSecret), with OpenAPI schema validation, versions and conversion.
  • An operator is a custom controller that reconciles custom resources using domain knowledge — e.g. a Postgres operator that creates StatefulSets, configures replication, handles failover and backups. Flux itself is a set of controllers reconciling GitRepository, Kustomization and HelmRelease resources.
  • Built with Kubebuilder/controller-runtime or Operator SDK.
Enlarge diagramPostgresClustercustom resource: 3replicasoperatorwatches andreconcilesStatefulSet + PVCsreplication + failoverbackups to S3
Figure 1. An operator is a controller for your own resource type. You declare a Postgres cluster; the operator does what a DBA would.
Swipe sideways to see all of it, or tap to open it full screen.

Upgrades and version skew

  • Kubernetes releases three minor versions a year; each gets about 14 months of upstream patch support.
  • Upgrade one minor version at a time: control plane first, then nodes.
  • Version skew policy: kubelets may be up to three minor versions older than the API server (since 1.28) but never newer; kubectl within ±1.
  • Before upgrading: check for removed/deprecated APIs in manifests and Helm charts (kubent, pluto, the API server's deprecated-API metrics), add-on compatibility, PDBs that block drains, and test in a non-prod cluster.
  • Node upgrades: cordon, drain (respecting PDBs), replace or upgrade, uncordon — or let managed node groups/Karpenter roll nodes.
Enlarge diagramCheck APIsremoved APIs:kubent, plutoAdd-onscompatibleversionsControl plane1.32 → 1.33Nodescordon, drain(PDBs), replaceVerifyworkloadshealthy12345
Figure 2. Upgrading a cluster: one minor version at a time, control plane first, then the nodes.
Swipe sideways to see all of it, or tap to open it full screen.

Backup and disaster recovery

  • etcd snapshots for self-managed control planes.
  • Velero backs up Kubernetes objects and persistent volume snapshots (EBS snapshots) to S3, with namespace-level restore — useful on EKS where you don't access etcd.
  • GitOps is itself a DR strategy: cluster state can be recreated from Git; data still needs backups.
Enlarge diagramConfig: from GitData: from backupsGitOps repo holds every manifestNew cluster → Flux syncs it allNo etcd restore neededVelero: objects + EBS snapshots to S3Databases: their own backupsTest restores regularly
Figure 3. Disaster recovery for Kubernetes: Git restores what should run; backups restore the data.
Swipe sideways to see all of it, or tap to open it full screen.

Certificates

kubeadm clusters use certificates for every component (API server, kubelet client/serving, etcd, front-proxy) — valid 1 year by default, renewed on upgrade or with kubeadm certs renew. Expired certs are a classic cause of a cluster suddenly refusing connections. Managed services like EKS handle control-plane certificates.