Kubernetes in depth · Lesson 10 of 11 · 3 min read
Extending Kubernetes and operating clusters
CRDs and operators
- CustomResourceDefinitions add new API types (
Certificate,Kustomization,HelmRelease,Rollout,ExternalSecret), with OpenAPI schema validation, versions and conversion. - An operator is a custom controller that reconciles custom resources using domain knowledge — e.g. a Postgres operator that creates StatefulSets, configures replication, handles failover and backups. Flux itself is a set of controllers reconciling
GitRepository,KustomizationandHelmReleaseresources. - Built with Kubebuilder/controller-runtime or Operator SDK.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Upgrades and version skew
- Kubernetes releases three minor versions a year; each gets about 14 months of upstream patch support.
- Upgrade one minor version at a time: control plane first, then nodes.
- Version skew policy: kubelets may be up to three minor versions older than the API server (since 1.28) but never newer; kubectl within ±1.
- Before upgrading: check for removed/deprecated APIs in manifests and Helm charts (
kubent,pluto, the API server's deprecated-API metrics), add-on compatibility, PDBs that block drains, and test in a non-prod cluster. - Node upgrades: cordon, drain (respecting PDBs), replace or upgrade, uncordon — or let managed node groups/Karpenter roll nodes.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Backup and disaster recovery
- etcd snapshots for self-managed control planes.
- Velero backs up Kubernetes objects and persistent volume snapshots (EBS snapshots) to S3, with namespace-level restore — useful on EKS where you don't access etcd.
- GitOps is itself a DR strategy: cluster state can be recreated from Git; data still needs backups.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Certificates
kubeadm clusters use certificates for every component (API server, kubelet client/serving, etcd, front-proxy) — valid 1 year by default, renewed on upgrade or with kubeadm certs renew. Expired certs are a classic cause of a cluster suddenly refusing connections. Managed services like EKS handle control-plane certificates.