Crisp answer: Start by understanding why — is it the kubelet, the network, or the underlying host? Cordon the node immediately to stop new scheduling, investigate, then either recover it or drain and replace it.
Step 1 — Immediate triage
# See the node condition
kubectl get nodes
# NAME STATUS ROLES AGE
# worker-2 NotReady <none> 10d ← problem here
kubectl describe node worker-2
# Look at Conditions section:
# Type Status Reason Message
# Ready False KubeletNotReady container runtime is not responding
# MemoryPressure False
# DiskPressure False
# PIDPressure False
# Also look at Events at the bottom — recent node events
Step 2 — Cordon the node
Prevent new pods being scheduled here while you investigate:
kubectl cordon worker-2
# node/worker-2 cordoned
# The node now shows SchedulingDisabled in kubectl get nodes
Existing pods continue running (cordoning does not evict). Only new scheduling is blocked.
Step 3 — Check the kubelet
SSH onto the node (or use kubectl debug if the node is still partially accessible):
# On the node:
systemctl status kubelet
# Active: failed (Result: exit-code)
journalctl -u kubelet -n 100 --no-pager
# Common messages:
# "failed to run Kubelet: misconfiguration: kubelet cgroup driver \"systemd\" is different from docker cgroup driver \"cgroupfs\""
# "Unable to read config path" — missing kubeconfig or cert expired
# "Error syncing pod" — container runtime issue
# Restart kubelet if config is fine but it crashed:
systemctl restart kubelet
systemctl enable kubelet # ensure it starts on reboot
Step 4 — Check the container runtime
# containerd (most clusters including EKS):
systemctl status containerd
journalctl -u containerd -n 50
# Test containerd directly:
ctr version
crictl info
# If containerd is wedged:
systemctl restart containerd
# then restart kubelet
Step 5 — Check disk and memory
Kubernetes marks nodes with DiskPressure or MemoryPressure — these can cascade into NotReady:
df -h # Is any filesystem full?
df -i # Inode exhaustion?
free -m # Memory available?
dmesg | grep -i "oom\|killed" # OOM kills?
# Kubernetes eviction thresholds:
# imagefs.available < 15% → evict pods, then NotReady
# memory.available < 100Mi → evict pods
Step 6 — Check networking/CNI
NotReady often means the kubelet cannot reach the API server, or the CNI plugin has crashed:
# Can the node reach the API server?
curl -k https://<api-server-endpoint>:443/healthz
# CNI plugin status (Cilium example):
cilium status
cilium connectivity test
# For general CNI:
ls /etc/cni/net.d/ # CNI config present?
ls /opt/cni/bin/ # CNI binaries present?
journalctl -u kubelet | grep "CNI\|network"
Step 7 — Node certificates
EKS and kubeadm clusters rotate node certificates. If a cert expired, the kubelet cannot authenticate to the API server:
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates
# notAfter=Jun 01 00:00:00 2025 GMT ← expired
On EKS: node certificates are rotated automatically by the node bootstrap process. If a node cert expires it usually means the node has been offline too long and needs replacement.
Step 8 — Drain and replace (if unrecoverable)
# Drain: evict all pods gracefully, then cordon
kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
# --ignore-daemonsets: DaemonSet pods will be recreated anyway
# --delete-emptydir-data: pods using emptyDir lose their data
# On EKS: terminate the EC2 instance — ASG will replace it
# On self-managed: provision a new node, let it join, then delete the old one
kubectl delete node worker-2 # Remove from cluster after terminating the instance
EKS-specific:
On EKS, nodes are EC2 instances managed by a node group (ASG). If a node is unhealthy:
# Check EC2 instance status
aws ec2 describe-instance-status --instance-ids i-xxxx --region eu-west-2
# Node group will auto-replace terminated instances
# You can also manually trigger a node group refresh:
aws eks update-nodegroup-config ...
What to say in the interview:
"First step is always cordon — stop new pods landing there while I investigate. Then check the kubelet with systemctl and journalctl. Common causes are: kubelet crashed, containerd hung, disk full, node certs expired, or CNI plugin failed. If the kubelet log shows it can't reach the API server I check the network path and certificate validity. If the node is unrecoverable I drain it — which evicts pods gracefully — then replace it. On EKS that's a matter of terminating the EC2 instance and letting the ASG replace it."