Opscraft
Kubernetes in depth · Lesson 2 of 11 · 7 min read

Control plane components in depth

kube-apiserver

The only component that talks to etcd, and the front door for everyone: kubectl, controllers, the scheduler, kubelets. It's stateless and horizontally scalable behind a load balancer.

Every request goes through a pipeline:

Enlarge diagramAuthenticationwho are you? fails with 401AuthorizationRBAC: may you? fails with 403Mutating admissiondefaults, injected sidecarsSchema validationis the object valid?Validating admissionpolicies: Pod Security, KyvernoSaved to etcdwatchers are notified123456
Figure 1. Every request to the API server passes the same checkpoints before anything is saved. The error code tells you which one stopped it.
Swipe sideways to see all of it, or tap to open it full screen.
  1. Authentication — who are you? Client certificates, bearer tokens (ServiceAccount JWTs), OIDC tokens, webhook token authenticators (EKS uses the AWS IAM authenticator / access entries). Failure → 401.
  2. Authorization — may you do this verb on this resource? Usually RBAC (plus Node authorizer for kubelets, webhooks). Failure → 403.
  3. Mutating admission — may modify the object: defaulting, injecting sidecars (Istio), adding labels, the ServiceAccount admission plugin, MutatingAdmissionWebhooks (Gatekeeper mutation, Kyverno).
  4. Schema validation of the (mutated) object.
  5. Validating admission — accept/reject only: ValidatingAdmissionWebhook (Gatekeeper/Kyverno), ValidatingAdmissionPolicy (CEL, built-in), PodSecurity, ResourceQuota, LimitRanger.
  6. Persist to etcd; notify watchers.

Other features: API aggregation (metrics-server's metrics.k8s.io is an aggregated API), API Priority and Fairness (protects the server from noisy clients), audit logging, server-side apply with field ownership.

etcd

A distributed, strongly consistent key-value store holding all cluster state (/registry/... keys). It uses the Raft consensus algorithm:

  • One leader handles writes and replicates log entries to followers; an entry is committed once a majority acknowledges it.
  • Quorum = ⌊N/2⌋+1 — run 3 or 5 members (odd), spread across failure domains. 3 tolerates 1 failure, 5 tolerates 2.
  • Lose quorum → the cluster becomes read-only/unavailable for writes: no new pods, no scaling, no changes. Running containers keep running (kubelets keep what they have) but nothing can be managed.

Performance depends on disk fsync latency (use fast SSD; on cloud, provisioned IOPS) and low network latency between members. Symptoms of a slow etcd: high API latency, leader elections, "etcdserver: request timed out". The default database size quota is 2 GB (often raised to 8 GB); old revisions must be compacted and the DB defragmented or it hits the quota and goes read-only ("mvcc: database space exceeded").

Enlarge diagrametcd-1leaderetcd-2followeretcd-3followerQuorumA write is committed when2 of 3 have it. Quorum =N/2 + 1, so use 3 or 5members.replicate
Figure 2. etcd keeps all cluster state and needs a majority to accept writes. With 3 members, one can fail; lose two and the cluster stops changing.
Swipe sideways to see all of it, or tap to open it full screen.

Backup and restore (self-managed clusters):

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
etcdutl snapshot status /backup/etcd-2026-10-25.db
etcdutl snapshot restore /backup/etcd-2026-10-25.db --data-dir /var/lib/etcd-restored
# then point the etcd static pod manifest at the new data dir

Losing etcd with no backup means losing every object and Secret — you'd rebuild the cluster from Git (one more reason for GitOps). Secrets are stored unencrypted (base64) unless you configure encryption at rest with an EncryptionConfiguration (aescbc/aesgcm/KMS provider). On EKS, AWS runs etcd for you and you enable KMS envelope encryption for Secrets.

kube-scheduler

Watches for pods with no spec.nodeName and assigns each to a node. For each pod it runs the scheduling framework:

  1. Filtering (predicates) — remove nodes that can't run the pod: insufficient allocatable CPU/memory for the pod's requests, node selector/affinity mismatch, untolerated taints, port conflicts, volume topology (EBS in another AZ), max pods per node, pod (anti-)affinity violations.
  2. Scoring (priorities) — rank the remaining nodes: least/most allocated (spreading vs bin-packing), balanced resource use, image locality (image already present), topology spread, preferred affinities.
  3. Binding — writes the chosen node into the pod (a Binding object). The kubelet on that node takes over.

If no node passes filtering, the pod stays Pending with an event like 0/6 nodes are available: 3 Insufficient memory, 3 node(s) had untolerated taint {dedicated: gpu}. Preemption may evict lower-priority pods to make room for a higher-priority one. Cluster Autoscaler/Karpenter watch for unschedulable pods and add nodes.

Enlarge diagramnew podrequests 1 CPU,2Ginode-anot enough memorynode-btaint dedicated=gpunode-cfits · score 71node-dfits · image cached · score 88filtered outfiltered outchosen
Figure 3. How the scheduler places one pod: remove nodes that cannot run it, score the rest, bind to the best.
Swipe sideways to see all of it, or tap to open it full screen.

The scheduler only considers requests, never actual usage — a node can be "full" while nearly idle, or overloaded while "half empty" if requests are unrealistic.

kube-controller-manager

One binary running many controllers, each a reconciliation loop: Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, Namespace (cleanup), Node (heartbeat monitoring, taints for unreachable nodes), EndpointSlice, ServiceAccount and token, PersistentVolume binder and expansion, garbage collector, HPA, ResourceQuota, certificate signing, TTL-after-finished. Controllers use leader election so only one active instance runs in an HA control plane.

cloud-controller-manager

Cloud-specific loops split out of core Kubernetes: the node controller (initialising nodes with cloud metadata, deleting Node objects when instances are terminated), the route controller, and the service controller (creating cloud load balancers for type: LoadBalancer). On EKS, the AWS Load Balancer Controller (an add-on you install) handles ALBs/NLBs with much more capability.

High availability

A production control plane runs 3+ API servers behind a load balancer, 3 or 5 etcd members (stacked on control-plane nodes or external), and scheduler/controller-manager replicas with leader election. EKS does all of this for you across multiple AZs.