How do you upgrade an EKS cluster safely

Crisp answer: Upgrade in three stages: control plane first, then cluster add-ons, then node groups. Each stage must be healthy before proceeding to the next. Never skip minor versions.

Pre-upgrade checklist:

# 1. Check current cluster version
aws eks describe-cluster --name my-cluster --query cluster.version --output text
# 1.28

# 2. Check if the target version is available
aws eks describe-addon-versions --kubernetes-version 1.29 --region eu-west-2 \
  --query 'addons[].{name:addonName,versions:addonVersions[0].addonVersion}'

# 3. Check for deprecated API versions in your manifests
# Use kubectl-convert or pluto:
pluto detect-all-in-cluster --target-versions k8s=v1.29

# 4. Review the EKS release notes for breaking changes
# https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

# 5. Check PodDisruptionBudgets — node drain during upgrade respects these
kubectl get pdb -A

# 6. Ensure workloads have multiple replicas and proper readiness probes
# Single-replica Deployments will have downtime during node replacement

Stage 1 — Upgrade the control plane:

aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.29 \
  --region eu-west-2

# Monitor progress:
aws eks describe-update \
  --name my-cluster \
  --update-id <update-id> \
  --region eu-west-2

# Or watch in a loop:
watch aws eks describe-cluster --name my-cluster \
  --query cluster.version --output text

# Control plane upgrade takes 10-20 minutes
# Cluster remains available during upgrade (API server has downtime of ~30 seconds)

Stage 2 — Upgrade cluster add-ons:

Add-ons must be compatible with the new Kubernetes version:

# Check current add-on versions
aws eks list-addons --cluster-name my-cluster

# Find the recommended version for each add-on on the new K8s version
aws eks describe-addon-versions \
  --addon-name coredns \
  --kubernetes-version 1.29 \
  --query 'addons[].addonVersions[?compatibilities[?defaultVersion==`true`]].addonVersion'

# Upgrade each add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name coredns \
  --addon-version v1.11.1-eksbuild.4 \
  --resolve-conflicts OVERWRITE

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name kube-proxy \
  --addon-version v1.29.0-eksbuild.1 \
  --resolve-conflicts OVERWRITE

aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.18.1-eksbuild.1 \
  --resolve-conflicts OVERWRITE

# Monitor each add-on update:
aws eks describe-addon \
  --cluster-name my-cluster \
  --addon-name coredns \
  --query addon.status

Stage 3 — Upgrade node groups:

# Trigger a rolling node replacement
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name workers \
  --kubernetes-version 1.29 \
  --region eu-west-2

# EKS will:
# 1. Launch new nodes with the new kubelet version
# 2. Wait for new nodes to become Ready
# 3. Cordon old nodes
# 4. Drain old nodes (respecting PodDisruptionBudgets)
# 5. Terminate old nodes
# 6. Repeat until all nodes are replaced

# Monitor:
aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name workers \
  --query nodegroup.status

# Watch node versions:
kubectl get nodes -o wide

For self-managed node groups:

# 1. Update the Launch Template with new AMI (EKS-optimised AMI for 1.29)
aws ssm get-parameter \
  --name /aws/service/eks/optimized-ami/1.29/amazon-linux-2/recommended/image_id \
  --region eu-west-2 --query Parameter.Value --output text

# 2. Apply the updated Launch Template to the ASG
# 3. Perform a rolling instance refresh on the ASG
aws autoscaling start-instance-refresh \
  --auto-scaling-group-name my-asg \
  --preferences '{"MinHealthyPercentage":90}'

What to say in the interview:

"EKS upgrades are a three-step process: control plane first, add-ons second, nodes third. You cannot skip steps because the kubelet on nodes must be within one minor version of the API server. Before starting I check for deprecated API versions using pluto, verify all workloads have multiple replicas, and check PodDisruptionBudgets so the drain respects minimum availability. The control plane upgrade takes 15-20 minutes and is mostly invisible. Node group upgrade does a rolling replacement — new nodes come up, old nodes get cordoned and drained, then terminated. I have automated four consecutive minor version upgrades on my homelab cluster in a single day using this pattern."