Opscraft
Kubernetes in depth · Lesson 8 of 11 · 5 min read

Scheduling and resource management

Requests, limits and QoS

  • Requests — what the scheduler reserves; the basis for HPA utilisation.
  • Limits — the cgroup ceiling. CPU over the limit → throttled. Memory over the limit → OOMKilled (exit 137).
QoS class Rule Eviction priority under node pressure
Guaranteed Every container has requests = limits for CPU and memory Last
Burstable At least one request/limit set, not Guaranteed Middle — those using more than their requests first
BestEffort No requests or limits First

Common senior positions: always set memory requests and limits (memory isn't compressible); set CPU requests but consider no CPU limits for latency-sensitive services to avoid throttling (requests still guarantee fair share); Guaranteed QoS for critical components.

Enlarge diagramGuaranteedrequests = limits · evicted lastBurstablesome set · evicted middleBestEffortnone set · evicted firstmemoryrequest 512Miup to limit 1GiOOMKilled (137)cpurequest 500mup to limit 1throttled
Figure 1. Requests are what the scheduler reserves; limits are the ceiling. Above the memory limit the container is killed; above the CPU limit it is slowed.
Swipe sideways to see all of it, or tap to open it full screen.

Node pressure and eviction

The kubelet evicts pods when node resources cross thresholds (memory.available, nodefs.available, imagefs.available, pid.available), ranking by QoS and usage over requests. The node gets conditions like MemoryPressure/DiskPressure and taints so new pods avoid it. Separately, the kernel OOM killer can kill processes when memory runs out faster than kubelet reacts. Reserve resources for system daemons with kubeReserved/systemReserved.

Steering placement

spec:
  nodeSelector: { karpenter.sh/capacity-type: on-demand }
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - { key: kubernetes.io/arch, operator: In, values: [arm64] }
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector: { matchLabels: { app: orders } }
          topologyKey: kubernetes.io/hostname
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector: { matchLabels: { app: orders } }
  tolerations:
  - { key: dedicated, operator: Equal, value: payments, effect: NoSchedule }
  priorityClassName: business-critical
  • nodeSelector / node affinity — pull pods toward nodes (required = hard, preferred = soft). "IgnoredDuringExecution" means running pods aren't moved if labels change.
  • Pod affinity / anti-affinity — co-locate with or separate from other pods (per node, per zone).
  • Taints and tolerations — taints repel pods from nodes unless pods tolerate them. Effects: NoSchedule, PreferNoSchedule, NoExecute (also evicts running pods without a toleration; tolerationSeconds delays it). A toleration allows but doesn't attract — combine with affinity to dedicate nodes. Kubernetes itself taints nodes not-ready/unreachable (NoExecute), and pods tolerate those for 300 s by default — which is why pods on a dead node are rescheduled after about 5 minutes.
  • Topology spread constraints — spread replicas evenly across zones/nodes (maxSkew), the modern way to get AZ resilience.
  • PriorityClass and preemption — higher-priority pods can evict lower-priority ones when the cluster is full; system pods use system-cluster-critical/system-node-critical.
Enlarge diagramZONE AZONE BZONE CnodeSelector / affinitypull pods toward nodes (arm64, on-demand)taints + tolerationskeep pods off nodes unless they toleratepod anti-affinityreplicas on different nodestopology spreadeven across AZs: maxSkew 1orders podorders podorders pod
Figure 2. Tools for steering pods: attract them to nodes, keep them off nodes, and spread them across zones.
Swipe sideways to see all of it, or tap to open it full screen.

Guardrails per namespace

  • ResourceQuota — caps totals per namespace: requests.cpu, limits.memory, pods, persistentvolumeclaims, services.loadbalancers, object counts. With a CPU/memory quota, every pod must declare requests/limits (or get defaults).
  • LimitRange — default and min/max requests/limits per container in a namespace.
  • PodDisruptionBudget — minAvailable/maxUnavailable for voluntary disruptions (drains, upgrades, Karpenter consolidation). A PDB with maxUnavailable: 0 (or minAvailable equal to replicas) blocks drains forever — a common upgrade blocker.
Enlarge diagramResourceQuotatotal CPU, memory, pods pernamespaceLimitRangedefault and max percontainerPodDisruptionBudgethow many may be downduring drains
Figure 3. Namespace guardrails: cap the total, set sensible defaults, and protect availability during maintenance.
Swipe sideways to see all of it, or tap to open it full screen.