Interview questions and model answers
"What happens when you run kubectl apply -f deployment.yaml?"
kubectl sends the object to the API server, which authenticates and authorises the request, runs mutating admission, validates the schema, runs validating admission (PodSecurity, Gatekeeper/Kyverno), and stores it in etcd. The Deployment controller sees it via a watch and creates a ReplicaSet; the ReplicaSet controller creates pods; the scheduler filters and scores nodes based on requests, affinity, taints and topology, and binds each pod. The kubelet on that node pulls images through containerd, the CNI assigns an IP, volumes mount, init containers run, then the app starts. Once readiness passes, the pod's IP goes into EndpointSlices, kube-proxy updates rules on every node and CoreDNS serves the Service name.
"Why does etcd need an odd number of members, and what happens if it loses quorum?"
Raft commits writes only when a majority agrees. With 3 members you can lose 1; adding a 4th raises quorum to 3 without increasing tolerance. If quorum is lost, the API server can't write, so nothing can be created, scaled or rescheduled, though existing containers keep running. Recovery is restoring members or restoring from a snapshot. etcd performance depends mostly on disk fsync latency.
"How does a Service actually route traffic?"
The Service gets a virtual ClusterIP. The EndpointSlice controller tracks ready pod IPs matching its selector. kube-proxy on every node writes iptables or IPVS rules so connections to ClusterIP:port are DNATed to a chosen pod IP:port, with conntrack keeping the connection on that backend. DNS from CoreDNS resolves the Service name to the ClusterIP. Because balancing is per connection, long-lived connections like gRPC don't rebalance — you need L7 balancing or client-side balancing.
"Explain requests vs limits and QoS classes."
Requests are reserved capacity used by the scheduler and HPA; limits are cgroup ceilings — CPU is throttled at the limit, memory is OOM-killed. Requests equal to limits gives Guaranteed QoS, evicted last; partial settings are Burstable; none is BestEffort, evicted first under node pressure. I always set memory requests and limits and CPU requests, and avoid CPU limits for latency-sensitive services because of CFS throttling.
"StatefulSet vs Deployment?"
A Deployment's pods are interchangeable, with random names and shared or no storage. A StatefulSet gives each pod a stable ordinal name, a stable DNS name through a headless Service, and its own PVC from volumeClaimTemplates that follows it across rescheduling, with ordered startup and updates and a partition option for canaries. It's for clustered stateful systems — and I'd still prefer an operator or a managed AWS service for databases.
"A pod is Pending. How do you reason about it?"
Pending means unscheduled or not yet started. kubectl describe pod shows scheduler events: insufficient CPU/memory relative to requests, untolerated taints, affinity or topology constraints, PVCs unbound or in a different AZ, too many pods per node, or quotas. If scheduled but pending, it's image pulls or volume attachment. The fixes follow: adjust requests, add capacity or let Karpenter provision, add tolerations, fix the StorageClass binding mode.
"How do NetworkPolicies work?"
By default all pods can talk to each other. Once a policy selects a pod for ingress or egress, only traffic allowed by some policy is permitted in that direction; policies are additive allow-lists. I start with default-deny per namespace, then allow specific flows by pod and namespace selectors and ports, remembering to allow DNS egress to CoreDNS. They only work if the CNI enforces them — Calico, Cilium, or the VPC CNI with network policy enabled.
"How do you secure a cluster?"
Identity: OIDC/IAM for humans, bound ServiceAccount tokens, IRSA or Pod Identity for AWS access. Least-privilege RBAC — careful with secrets, exec and pod creation. Pod Security Admission at restricted for app namespaces and a policy engine for registries, limits and labels. securityContext with non-root, read-only root filesystem, dropped capabilities and seccomp. NetworkPolicies with default deny. Secrets encrypted with KMS and sourced from Secrets Manager. Audit logs to CloudWatch, image scanning and signing, a private API endpoint, and regular upgrades.
"What are operators and CRDs?"
CRDs extend the API with new resource types and schemas. An operator is a controller that watches those resources and reconciles real state, encoding operational knowledge — creating StatefulSets, handling failover, backups and upgrades for something like Postgres or Kafka. Flux, cert-manager, External Secrets and Karpenter are all CRD-plus-controller systems.