Upgrade a Kubernetes Cluster One Minor Version at a Time

Check version skew and removed APIs, upgrade the control plane before nodes, and validate workloads on a canary node pool.

~45 minkubernetesupgradeoperations

Task

Move a cluster to a supported version without breaking clients that still call APIs removed in the new release.

When to use this

A cluster jumps two minor versions and an Ingress manifest using a removed API version stops reconciling; the controller runs but new routes never apply.

Prerequisites

A managed cluster (EKS, GKE, or AKS) or kubeadm access
A maintenance window and a tested rollback plan
Add-on versions that support the target minor

Steps

1. Find workloads that still call deprecated API versions.

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Expected output: Non-zero counters name the API groups and versions still in use, which must be migrated first.

2. Check that node and control plane versions satisfy the skew policy.

kubectl version -o yaml

Expected output: Node versions are within one minor of the control plane, as the skew policy requires.

3. Upgrade the control plane before any nodes.

gcloud container clusters upgrade payments-prod --master --cluster-version 1.30

Expected output: The control plane reports the new version while existing nodes keep serving workloads.

4. Upgrade one canary node pool and confirm pods are healthy.

gcloud container node-pools upgrade canary --cluster payments-prod --node-pool-version 1.30 && kubectl get nodes -L cloud.google.com/gke-nodepool

Expected output: Canary nodes report the new kubelet version and every workload is Ready before other pools change.

5. Check for removed-API failures after the upgrade.

kubectl get --raw /metrics | grep 'apiserver_request_total{code="410"' | head

Expected output: No sustained 410 responses; any client still calling a removed API fails visibly and gets fixed.

Troubleshooting

Problem

Pods stay Pending after nodes are upgraded

Solution

The new kubelet may reject old resource requests or taints. Check node conditions and compare taints with tolerations.

Problem

CNI, CSI, or ingress breaks after the upgrade

Solution

Upgrade those add-ons to versions that support the target minor before upgrading nodes, not after.

Limitations

Managed control plane upgrades cannot be rolled back, although node pools can be replaced.
Deprecated APIs are removed across several minors, so the inventory must be repeated at every upgrade.

Upgrade the API before the nodes

Kubernetes upgrades have two halves with different risk profiles. The control plane gains new APIs and removes old ones; nodes run kubelet and container runtime. Clients that call a removed API version keep working until the API server no longer serves it, then fail with 404 or 410. That failure often appears in a controller that reconciles an old manifest, not in the application, which is why the API inventory comes first. The version skew policy allows nodes one minor behind the control plane, which is what makes a staged upgrade possible.

Sequence the work

Upgrade the control plane, then a canary node pool, then the remaining pools. Between each step, confirm that pods are Ready and that the add-ons still function. CNI, CSI, and ingress controllers are the usual casualties of a node upgrade because their daemonsets run with elevated privileges and version-specific assumptions. Keep the previous node pool until the new one has served traffic for a full day, so a rollback means draining back rather than rebuilding.

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
gcloud container clusters upgrade payments-prod --master --cluster-version 1.30
gcloud container node-pools upgrade canary --cluster payments-prod --node-pool-version 1.30

Verify after each hop

Check kubectl version for skew, kubectl get nodes for kubelet versions, and API server metrics for 410 responses. Run a deployment and a service change in the canary pool, because read-only checks miss admission and scheduling problems. Finally, update the cluster’s documented supported version and the next upgrade date; clusters that fall three minors behind lose the supported upgrade path entirely.