Networking with the VPC CNI
How the Amazon VPC CNI works
The CNI plugin (aws-node DaemonSet) gives each pod a real VPC IP address:
- On each node it attaches additional ENIs to the instance.
- Each ENI gets secondary private IPs from the node's subnet.
- When a pod starts, the CNI assigns one of those IPs to the pod's network namespace.
Benefits: pods are first-class VPC citizens — no overlay, native VPC performance, security groups and Flow Logs see pod IPs, ALBs can target pods directly (target-type: ip).
Costs: pods consume subnet IPs, and the number of pods per node is limited by the instance's ENI and IP capacity:
max pods = ENIs × (IPv4 addresses per ENI − 1) + 2
m5.large: 3 × (10 − 1) + 2 = 29 c5.4xlarge: 8 × (30 − 1) + 2 = 234
The CNI keeps a warm pool of IPs/ENIs per node for fast pod startup (WARM_ENI_TARGET, WARM_IP_TARGET, MINIMUM_IP_TARGET) — which consumes even more subnet addresses.
Running out of IPs
Symptoms: pods stuck in ContainerCreating with failed to assign an IP address to container, aws-node logs about no available IPs, subnets showing near-zero available addresses.
Solutions:
- Prefix delegation (
ENABLE_PREFIX_DELEGATION=true) — assigns /28 prefixes (16 IPs) to ENI slots instead of single IPs; raises max pods per node (e.g. m5.large to 110) and speeds allocation. Requires contiguous free /28 blocks in the subnet (fragmented subnets can fail — use subnet CIDR reservations). - Secondary CIDR + custom networking — add a non-routable range like 100.64.0.0/16 (CG-NAT space) to the VPC, create pod subnets from it, and define
ENIConfigobjects per AZ so pod ENIs use those subnets while node primary interfaces stay in the original subnets. Pods' outbound traffic is SNATed to the node IP for traffic leaving the VPC. - Bigger subnets for new node groups (plan /19–/20 per AZ from the start).
- IPv6 clusters — every pod gets an IPv6 address from a /80 per node; IPv4 exhaustion disappears (with ecosystem caveats).
- Tune warm pool settings and remove unused ENIs.
Security groups for pods and network policies
- Security groups for pods (
ENABLE_POD_ENI=true, supported Nitro instances): attach EC2 security groups to specific pods via aSecurityGroupPolicy— e.g. onlyorderspods may reach the RDS security group. Pods get branch ENIs. - Network policies: the VPC CNI now enforces Kubernetes NetworkPolicies natively (eBPF-based, enabled via the add-on configuration). Alternatives: Calico (policy-only mode) or Cilium.
Load balancing with the AWS Load Balancer Controller
Install the controller (Helm or EKS add-on in Auto Mode) with an IAM role. It watches:
- Ingress (class
alb) → creates/updates an Application Load Balancer. - Service type LoadBalancer (with
service.beta.kubernetes.io/aws-load-balancer-type: external) → creates a Network Load Balancer. - Also supports Gateway API.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: orders
namespace: shop
annotations:
alb.ingress.kubernetes.io/scheme: internet-facing # or internal
alb.ingress.kubernetes.io/target-type: ip # register pod IPs directly
alb.ingress.kubernetes.io/group.name: shop-public # share one ALB across Ingresses
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-south-1:123456789012:certificate/abc
alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
ingressClassName: alb
rules:
- host: orders.example.com
http:
paths:
- { path: /, pathType: Prefix, backend: { service: { name: orders, port: { number: 80 } } } }
- target-type
ipregisters pod IPs (faster, no NodePort hop, required for Fargate);instanceroutes to NodePorts on nodes. - IngressGroup (
group.name) merges many Ingresses into one ALB — 30 services don't need 30 ALBs (big cost saving). Watch the per-ALB rule limits and blast radius. - Subnet discovery uses tags:
kubernetes.io/role/elb=1on public subnets,kubernetes.io/role/internal-elb=1on private. - Pod readiness gates (
elbv2.k8s.aws/pod-readiness-gate-inject: enablednamespace label) make rollouts wait until new pods are healthy in the ALB target group. scheme: internalfor VPC-only services.