Opscraft
Amazon EKS · Lesson 3 of 11 · 6 min read

Networking with the VPC CNI

How the Amazon VPC CNI works

The CNI plugin (aws-node DaemonSet) gives each pod a real VPC IP address:

  1. On each node it attaches additional ENIs to the instance.
  2. Each ENI gets secondary private IPs from the node's subnet.
  3. When a pod starts, the CNI assigns one of those IPs to the pod's network namespace.

Benefits: pods are first-class VPC citizens — no overlay, native VPC performance, security groups and Flow Logs see pod IPs, ALBs can target pods directly (target-type: ip).

Enlarge diagramEC2 NODE M5.LARGE IN SUBNET 10.20.32.0/19ENI 010 IPs: 1 primary + 9for podspod10.20.30.11pod10.20.30.12pod10.20.30.13ENI 110 IPs: 1 primary + 9for podspod10.20.31.11pod10.20.31.12pod10.20.31.13max pods = ENIs × (IPs per ENI − 1) + 2 → m5.large: 3 × 9 + 2 = 29
Figure 1. The Amazon VPC CNI: the node gets extra network interfaces, each with spare VPC IPs, and every pod gets one of them. No overlay.
Swipe sideways to see all of it, or tap to open it full screen.

Costs: pods consume subnet IPs, and the number of pods per node is limited by the instance's ENI and IP capacity:

max pods = ENIs × (IPv4 addresses per ENI − 1) + 2
m5.large: 3 × (10 − 1) + 2 = 29      c5.4xlarge: 8 × (30 − 1) + 2 = 234

The CNI keeps a warm pool of IPs/ENIs per node for fast pod startup (WARM_ENI_TARGET, WARM_IP_TARGET, MINIMUM_IP_TARGET) — which consumes even more subnet addresses.

Running out of IPs

Symptoms: pods stuck in ContainerCreating with failed to assign an IP address to container, aws-node logs about no available IPs, subnets showing near-zero available addresses.

Solutions:

  1. Prefix delegation (ENABLE_PREFIX_DELEGATION=true) — assigns /28 prefixes (16 IPs) to ENI slots instead of single IPs; raises max pods per node (e.g. m5.large to 110) and speeds allocation. Requires contiguous free /28 blocks in the subnet (fragmented subnets can fail — use subnet CIDR reservations).
  2. Secondary CIDR + custom networking — add a non-routable range like 100.64.0.0/16 (CG-NAT space) to the VPC, create pod subnets from it, and define ENIConfig objects per AZ so pod ENIs use those subnets while node primary interfaces stay in the original subnets. Pods' outbound traffic is SNATed to the node IP for traffic leaving the VPC.
  3. Bigger subnets for new node groups (plan /19–/20 per AZ from the start).
  4. IPv6 clusters — every pod gets an IPv6 address from a /80 per node; IPv4 exhaustion disappears (with ecosystem caveats).
  5. Tune warm pool settings and remove unused ENIs.
Enlarge diagramPrefix delegation/28 (16 IPs) per slot: m5.large to 110 podsSecondary CIDR100.64.0.0/16 pod subnets + ENIConfigBigger subnetsplan /19 or /20 per AZIPv6 clusterno IPv4 exhaustion at all
Figure 2. Running out of pod IPs: the options, from quickest to most involved.
Swipe sideways to see all of it, or tap to open it full screen.

Security groups for pods and network policies

  • Security groups for pods (ENABLE_POD_ENI=true, supported Nitro instances): attach EC2 security groups to specific pods via a SecurityGroupPolicy — e.g. only orders pods may reach the RDS security group. Pods get branch ENIs.
  • Network policies: the VPC CNI now enforces Kubernetes NetworkPolicies natively (eBPF-based, enabled via the add-on configuration). Alternatives: Calico (policy-only mode) or Cilium.
Enlarge diagramorders podssg-orders(SecurityGroupPolicy)web podsnode security groupRDS orders-dballows sg-orders on 5432allowedblocked
Figure 3. Security groups for pods: only the orders pods carry the security group that the database allows.
Swipe sideways to see all of it, or tap to open it full screen.

Load balancing with the AWS Load Balancer Controller

Install the controller (Helm or EKS add-on in Auto Mode) with an IAM role. It watches:

  • Ingress (class alb) → creates/updates an Application Load Balancer.
  • Service type LoadBalancer (with service.beta.kubernetes.io/aws-load-balancer-type: external) → creates a Network Load Balancer.
  • Also supports Gateway API.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orders
  namespace: shop
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing        # or internal
    alb.ingress.kubernetes.io/target-type: ip                # register pod IPs directly
    alb.ingress.kubernetes.io/group.name: shop-public        # share one ALB across Ingresses
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-south-1:123456789012:certificate/abc
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
  rules:
  - host: orders.example.com
    http:
      paths:
      - { path: /, pathType: Prefix, backend: { service: { name: orders, port: { number: 80 } } } }
  • target-type ip registers pod IPs (faster, no NodePort hop, required for Fargate); instance routes to NodePorts on nodes.
  • IngressGroup (group.name) merges many Ingresses into one ALB — 30 services don't need 30 ALBs (big cost saving). Watch the per-ALB rule limits and blast radius.
  • Subnet discovery uses tags: kubernetes.io/role/elb=1 on public subnets, kubernetes.io/role/internal-elb=1 on private.
  • Pod readiness gates (elbv2.k8s.aws/pod-readiness-gate-inject: enabled namespace label) make rollouts wait until new pods are healthy in the ALB target group.
  • scheme: internal for VPC-only services.
Enlarge diagramLB Controllerwatches the APIIngress (class alb)Service type LBexternalALBHTTP(S), one per groupNLBTCP / UDP
Figure 4. The AWS Load Balancer Controller turns Kubernetes objects into AWS load balancers. target-type ip sends traffic straight to pod IPs.
Swipe sideways to see all of it, or tap to open it full screen.