Crisp answer: Work through layers: pod-to-pod, pod-to-service, and pod-to-external. Use a debug pod with networking tools. Check security groups, NACLs, VPC CNI health, and CoreDNS for DNS issues.
Toolbox — run a debug pod:
kubectl run netdebug \
--image=nicolaka/netshoot \
--restart=Never \
-it --rm \
-- bash
# Inside netshoot: curl, wget, dig, nslookup, nc, ping, tcpdump, ss, traceroute all available
Pod-to-pod connectivity:
# Can pod A reach pod B directly?
# Get pod B's IP:
kubectl get pod pod-b -o wide
# 10.0.1.45
# From pod A:
kubectl exec -it pod-a -- curl http://10.0.1.45:8080
kubectl exec -it pod-a -- nc -zv 10.0.1.45 8080
# If failing, check:
# 1. Security groups on both pods (if using security groups for pods)
# 2. Network policies blocking traffic
# 3. VPC CNI health
Pod-to-Service connectivity:
# Does DNS resolve?
kubectl exec -it pod-a -- nslookup api.production.svc.cluster.local
# Should return: 10.96.x.x (ClusterIP)
# If DNS fails, check CoreDNS:
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
# Does the ClusterIP respond?
kubectl exec -it pod-a -- curl http://api.production.svc.cluster.local:80
# Check Endpoints — if empty, selector doesn't match:
kubectl get endpoints api -n production
Pod-to-external (internet or AWS services):
# Can the pod reach the internet? (should work if NAT Gateway configured)
kubectl exec -it pod-a -- curl https://example.com
# Can the pod reach an AWS service? (should work via VPC endpoint or NAT)
kubectl exec -it pod-a -- curl https://s3.eu-west-2.amazonaws.com
# If VPC endpoint configured, check it's routing correctly:
aws ec2 describe-vpc-endpoints --region eu-west-2 \
--filters Name=vpc-id,Values=vpc-xxx \
--query 'VpcEndpoints[].{Service:ServiceName,State:State}'
Common EKS networking issues:
Issue: Pod gets no IP address, stays in ContainerCreating
→ VPC CNI exhausted IP pool
→ Check: kubectl describe pod → "failed to assign IP"
→ Fix: check subnet has free IPs, check ENI limits, enable prefix delegation
Issue: DNS resolution fails from pod
→ CoreDNS not running or overloaded
→ Check: kubectl get pods -n kube-system -l k8s-app=kube-dns
→ Check: kubectl logs -n kube-system coredns-xxx
→ Fix: scale up CoreDNS replicas, check NodeLocal DNSCache
Issue: LoadBalancer Service stuck in pending
→ Load Balancer Controller not running
→ IAM role for LB controller missing permissions
→ Subnet not tagged correctly
→ Check: kubectl describe service my-service → Events
Issue: ALB returns 502 bad gateway
→ Target pods are unhealthy
→ Check: ALB target group health in AWS console
→ Check: pod readiness probe passing?
→ Check: pod listening on correct port?
Issue: Cross-AZ traffic causing high latency
→ Pod talking to service endpoints in different AZ
→ Fix: enable topology-aware routing on the Service
→ kubectl annotate service api service.kubernetes.io/topology-mode=Auto
VPC Flow Logs for network debugging:
# Enable VPC flow logs to CloudWatch
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-xxx \
--traffic-type ALL \
--log-destination-type cloud-watch-logs \
--log-group-name /aws/vpc/flow-logs \
--deliver-logs-permission-arn arn:aws:iam::123:role/flow-logs-role
# Query in CloudWatch Logs Insights:
# fields @timestamp, srcAddr, dstAddr, dstPort, action
# | filter srcAddr like "10.0.1.45"
# | filter action = "REJECT"
# | sort @timestamp desc
What to say in the interview:
"I debug EKS networking in layers. First confirm pod-to-pod by curling the pod IP directly. If that works but service DNS fails, I check CoreDNS. If the pod can't reach external services, I check the NAT Gateway route and VPC endpoints. For persistent issues I enable VPC Flow Logs and query them in CloudWatch Logs Insights to find REJECT actions — that shows exactly which traffic is being blocked and by what. The netshoot image is my go-to debug container: it has every networking tool I need and I can run it as a temporary pod in any namespace."