Kubernetes 升级

1.版本升级规划

K8s 官方文档明确说明：

Skipping MINOR versions when upgrading is unsupported

也就是说，从v1.31.14升级到v1.35.6，不能一步到位，必须逐个版本升级，共经历四跳：

v1.31.14 -> v1.32.x -> v1.33.x -> v1.34.x -> v1.35.6

2.版本升级流程

由于不能跳版本，v1.31 -> v1.35本质上是把单次升级流程重复执行四遍：

以下步骤针对每个控制平面节点：每完成一个版本升级后，所有节点同步升级，再进入下一个版本升级流程。

2.1.腾空节点

只对当前升级的节点进行：

kubectl drain xxx \ --ignore-daemonsets \ --delete-emptydir-data \ --timeout=300s

2.2.备份 ETCD

在所有控制平面节点上执行。

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /root/backup/etcd-$(date +%F-%H%M).db

执行成功后输出一堆info日志，最终看到Snapshot saved at ...就备份成功了：

{"level":"info","ts":"xxxx-xx-xx","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/backup/etcd-xxxx-xx-xx-xxxx.db.part"} {"level":"info","ts":"xxxx-xx-xx","logger":"client","caller":"v3@v3.5.21/maintenance.go:212","msg":"opened snapshot stream; downloading"} {"level":"info","ts":"xxxx-xx-xx","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"xx MB","took":"xx second ago"} {"level":"info","ts":"xxxx-xx-xx","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/backup/etcd-xxxx-xx-xx-xxxx.db"} Snapshot saved at /backup/etcd-x.x.x-xxxx.db

2.3.更新软件源

当前环境使用清华源。每次版本升级，都需要把旧版本源注释或清理，换成新版本源并下载对应的仓库公钥。

在所有控制平面节点更新：

# 下载前先将旧版本公钥改名备份一下,等升级结束后再清理 curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \ | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # 配置新版本源 # $K8S_VERSION: 具体要升级的版本 cat /etc/apt/sources.list.d/kubernetes.list deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/$K8S_VERSION/deb/ / # 更新源 apt-get update # 列出当前可用版本 apt-cache madison kubeadm | head -n 1 # 示例输出 kubeadm | 1.32.13-1.1 | http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.32/deb Packages apt-cache madison kubelet | head -n 1 apt-cache madison kubectl | head -n 1 # 确认组件未被 apt 锁定(hold) apt-mark showhold | grep -Ei "kubeadm|kubelet|kubectl"

2.4.验证升级计划

检查可升级到哪些版本，并验证当前集群是否可升级。该命令只能在存在 kubeconfig 文件admin.conf的控制平面节点上运行。

# 安装目标版本的 kubeadm apt-get install -y kubeadm='1.32.13-1.1' # 确认版本 kubeadm version # 验证升级计划: kubeadm 会列出当前版本/目标版本/各组件变更 kubeadm upgrade plan # 输出内容(忽略输出的版本信息) [preflight] Running pre-flight checks. [upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... [upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: 1.32.13 [upgrade/versions] kubeadm version: v1.33.13 I0622 18:27:10.934338 3715038 version.go:261] remote version is much newer: v1.36.2; falling back to: stable-1.33 [upgrade/versions] Target version: v1.33.13 [upgrade/versions] Latest version in the v1.32 series: v1.32.13 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT NODE CURRENT TARGET kubelet x.x.x v1.32.13 v1.33.13 kubelet x.x.x v1.32.13 v1.33.13 kubelet x.x.x v1.32.13 v1.33.13 Upgrade to the latest stable version: COMPONENT NODE CURRENT TARGET kube-apiserver x.x.x v1.32.13 v1.33.13 kube-apiserver x.x.x v1.32.13 v1.33.13 kube-apiserver x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-proxy 1.32.13 v1.33.13 CoreDNS v1.11.3 v1.12.0 etcd x.x.x 3.5.24-0 3.5.24-0 etcd x.x.x 3.5.24-0 3.5.24-0 etcd x.x.x 3.5.24-0 3.5.24-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.33.13 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no

2.5.关闭 kube-apiserver 连接

因为kube-apiserver/etcd静态 Pod 始终在运行，所以执行 kubeadm 升级时，对服务器正在进行的请求将停滞。可以在运行kubeadm upgrade apply命令前主动停止kube-apiserver进程几秒钟。这样可以能让正在进行的请求完成处理并关闭现有连接，最大限度地减少 etcd 停机的后果：

killall -s SIGTERM kube-apiserver

2.6.集群升级

2.6.1.控制节点升级

该命令只需执行一次，且仅可在具有 kubeconfig 文件admin.conf的控制平面节点上运行。它的作用是：

1. preflight：升级前预检查；

2. control-plane：升级本节点上的控制平面实例；

3. upload-config：将 kubeadm 和 kubelet 配置上传到 ConfigMap（集群级，唯一）；

   3.1. /kubeadm：将 ClusterConfiguration 写入 kubeadm-config ConfigMap；

   3.2. /kubelet：将 kubelet 配置写入 kubelet-config ConfigMap；

4. kubelet-config：升级本节点的 kubelet 配置；

5. bootstrap-token：配置 bootstrap token 和 cluster-info RBAC 规则（集群级，唯一）；

6. addon：升级默认插件（CoreDNS、kube-proxy）；

7. post-upgrade：升级后任务。

# 第一个控制平面节点用 apply kubeadm upgrade apply 1.32.13 # 输出内容 [upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... [upgrade/preflight] Running preflight checks [upgrade] Running cluster health checks [upgrade/preflight] You have chosen to upgrade the cluster version to "v1.32.13" [upgrade/versions] Cluster version: v1.31.14 [upgrade/versions] kubeadm version: v1.32.13 [upgrade] Are you sure you want to proceed? [y/N]: y [upgrade/preflight] Pulling images required for setting up a Kubernetes cluster [upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection # 这里告警提示: 基础镜像版本不一致 # 参考 '问题处理 1.1 基础镜像版本不一致' 解决 W0622 16:00:12 checks.go:843] detected that the sandbox image "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8" of the container runtime is inconsistent with that used by kubeadm. It is recommended to use "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10" as the CRI sandbox image. [upgrade/control-plane] Upgrading your Static Pod-hosted control plane to version "v1.32.13" (timeout: 5m0s)... [upgrade/staticpods] Preparing for "etcd" upgrade [upgrade/staticpods] Renewing etcd-server / etcd-peer / etcd-healthcheck-client certificate [upgrade/staticpods] Moving new manifest to ".../etcd.yaml" and backing up old manifest [upgrade/staticpods] Component "etcd" upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver / apiserver-kubelet-client / front-proxy-client / apiserver-etcd-client certificate [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upgrade/control-plane] The control plane instance for this node was successfully upgraded! [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system [upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded! [upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded! [upgrade/addon] Skipping upgrade of addons because control plane instances [x.x.x x.x.x] have not been upgraded [upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.32.13". [upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.

2.6.2.其余节点升级

在上面某台 master 节点执行 apply 后执行该命令。与apply相比，node没有upload-config和bootstrap-token这两个集群级阶段，这正是它可以在多个节点上重复执行的原因。在control-plane阶段，当前节点是控制平面就升级控制平面实例，是工作节点则跳过，因此该命令可同时用于两种节点：

"Upgrade the control plane instance deployed on this node,if any"

# 其余控制节点/Node 节点 kubeadm upgrade node # 输出内容 [upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... [upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it. [upgrade/preflight] Running pre-flight checks [upgrade/preflight] Pulling images required for setting up a Kubernetes cluster [upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection [upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull' W0622 18:36:03.656485 1715747 checks.go:843] detected that the sandbox image "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10" as the CRI sandbox image. [upgrade/control-plane] Upgrading your Static Pod-hosted control plane instance to version "v1.33.13"... [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests2277431111" [upgrade/staticpods] Preparing for "etcd" upgrade [upgrade/staticpods] Renewing etcd-server certificate [upgrade/staticpods] Renewing etcd-peer certificate [upgrade/staticpods] Renewing etcd-healthcheck-client certificate [upgrade/staticpods] Restarting the etcd static pod and backing up its manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/etcd.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector component=etcd [upgrade/staticpods] Component "etcd" upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Renewing apiserver-etcd-client certificate [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector component=kube-apiserver [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade [upgrade/staticpods] Renewing controller-manager.conf certificate [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/kube-controller-manager.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector component=kube-controller-manager [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! [upgrade/staticpods] Preparing for "kube-scheduler" upgrade [upgrade/staticpods] Renewing scheduler.conf certificate [upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/kube-scheduler.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector component=kube-scheduler [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upgrade/control-plane] The control plane instance for this node was successfully upgraded! ## 此告警是指配置中 controlPlaneEndpoint 与实际 bindPort 不一致 ## 参考 '问题处理 1.3' 处（在本文中是符合预期的） W0622 18:38:43.473778 1715747 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded! W0622 18:38:50.167686 1715747 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config2351329468 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR [upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2351329468/config.yaml [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded! [addons] Applied essential addon: CoreDNS W0622 18:38:50.338251 1715747 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy

2.7.升级 kubelet 与 kubectl

apt-get install -y kubelet='1.32.13-1.1' kubectl='1.32.13-1.1' # 升级后会自动重启 kubelet containerd(可通过日志观察) # systemctl daemon-reload # systemctl restart kubelet/containerd

2.8.恢复节点调度并验证

kubectl uncordon xxx kubectl get node -o wide

确认节点VERSION更新后，登录下一个控制平面节点，重复上述步骤。

注：上述 2.1 ～ 2.8 步要在所有控制节点上都走一遍；且四要分别四次执行版本升级，即整体循环四次...

Containerd 升级

K8s 集群升级后，参考 containerd 官网版本对应关系，计划将 containerd 升到 2.3.2。虽然 Kubernetes test grid 中 1.35 未覆盖 containerd 2.3.x，但二者 CRI Version 一致，且官方升级路径支持 LTS 跨版本升级，因此本次直接从 1.7.27 直接升级到 2.3.2：

Upgrades are supported for sequential minor releases. For example, an upgrade from 2.0 to 2.1 is supported, but an upgrade from 2.0 to 2.2 is not. Patch releases are always backward compatible with their minor version.

In addition to sequential minor release upgrades, direct upgrades between sequential LTS (Long Term Stable) releases are also supported. For example, a direct upgrade from 1.7 (LTS) to 2.3 (LTS) will be tested and supported, but 1.7 (LTS) to 2.6 (LTS, tentatively) will not. This allows users who prefer to stay on LTS releases to have a clear and safe upgrade path.

升级前的状态：

# containerd --version containerd github.com/containerd/containerd v1.7.27 05044ec0a9a75232cad458027ca83437aae3f4da # runc --version runc version 1.4.3 commit: v1.4.3-0-gbb14dabeb spec: 1.3.0 go: go1.25.11 libseccomp: 2.6.0 # kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME x.x.x Ready control-plane 346d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2 x.x.x Ready control-plane 346d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2 x.x.x Ready control-plane 346d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2

1.升级前检查

先查询当前版本是否有废弃配置 / API 提示：

# ctr deprecations list # ctr deprecations list --format json

从 containerd 2.0 开始，官方废弃了cri-containerd-(cni-)<VERSION>-<OS>-<ARCH>.tar.gz全量安装包。他最初是面向 k8s 提供的：把 containerd、runc 和 CNI 打包在一起，无需再分别下载各个组件，便于快速安装。不过自 k8s 1.24 移除 Dockershim 起，containerd 社区也在讨论中决定不再维护全量包。在 2.0 版本后改为各组件由用户按需安装依赖。

因此，从 2.0 开始，containerd 作为 K8s CRI 时，需要分别安装以下三个组件：

2.升级流程

所有 k8s 节点依次完成 2.1 ～ 2.7 中所有操作

2.1.腾空节点

kubectl cordon xxx kubectl drain xxx --ignore-daemonsets --delete-emptydir-data --timeout=300s

2.2.下载新版本组件

wget "https://github.com/containerd/containerd/releases/download/v2.3.2/containerd-2.3.2-linux-amd64.tar.gz" wget "https://github.com/opencontainers/runc/releases/download/v1.4.3/runc.amd64" wget "https://github.com/containernetworking/plugins/releases/download/v1.9.1/cni-plugins-linux-amd64-v1.9.1.tgz"

2.3.停止 containerd

systemctl stop containerd

2.4.升级 runc

# 查询旧版本 # runc --version runc version 1.2.5 commit: v1.2.5-0-g59923ef1 spec: 1.2.0 go: go1.23.7 # 备份旧版本 mv /usr/local/sbin/runc /usr/local/sbin/runc_1.2.5.old # 替换为新版本 mv ./runc.amd64 /usr/local/sbin/runc chmod 755 /usr/local/sbin/runc # 确认版本 runc --version runc version 1.4.3 commit: v1.4.3-0-gbb14dabeb spec: 1.3.0 go: go1.25.11 libseccomp: 2.6.0

2.5.升级 CNI plugins

看下更新前/opt/cni/bin/下的内容：

# ll /opt/cni/bin/ total 215M -rwxr-xr-x 1 root root 4.0M Jul 15 2025 tuning -rwxr-xr-x 1 root root 4.3M Jul 15 2025 portmap -rwxr-xr-x 1 root root 3.9M Jul 15 2025 loopback -rwxr-xr-x 1 root root 3.8M Jul 15 2025 host-local -rwxr-xr-x 1 root root 2.7M Jul 15 2025 flannel -rwxr-xr-x 1 root root 66M Jul 15 2025 calico-ipam -rwxr-xr-x 1 root root 66M Jul 15 2025 calico -rwxr-xr-x 1 root root 4.3M Jul 15 2025 bandwidth -rwxr-xr-x 1 root root 4.3M Aug 30 2024 macvlan -rwxr-xr-x 1 root root 3.8M Aug 30 2024 sbr -rwxr-xr-x 1 root root 4.6M Aug 30 2024 bridge -rwxr-xr-x 1 root root 11M Aug 30 2024 dhcp -rwxr-xr-x 1 root root 4.3M Aug 30 2024 dummy -rwxr-xr-x 1 root root 4.8M Aug 30 2024 firewall -rwxr-xr-x 1 root root 4.2M Aug 30 2024 host-device -rwxr-xr-x 1 root root 4.3M Aug 30 2024 ipvlan -rw-r--r-- 1 root root 12K Aug 30 2024 LICENSE -rwxr-xr-x 1 root root 4.4M Aug 30 2024 ptp -rw-r--r-- 1 root root 2.3K Aug 30 2024 README.md -rwxr-xr-x 1 root root 3.1M Aug 30 2024 static -rwxr-xr-x 1 root root 4.3M Aug 30 2024 tap -rwxr-xr-x 1 root root 4.3M Aug 30 2024 vlan -rwxr-xr-x 1 root root 4.0M Aug 30 2024 vrf

解压新版本到 CNI 目录（会覆盖同名插件）：

tar zxvf ./cni-plugins-linux-amd64-v1.9.1.tgz -C /opt/cni/bin/

查看更新后内容：

# ll /opt/cni/bin/ total 228M -rwxr-xr-x 1 root root 4.2M Mar 16 22:18 host-local -rw-r--r-- 1 root root 12K Mar 16 22:18 LICENSE -rw-r--r-- 1 root root 2.3K Mar 16 22:18 README.md -rwxr-xr-x 1 root root 3.7M Mar 16 22:18 static -rwxr-xr-x 1 root root 14M Mar 16 22:18 dhcp -rwxr-xr-x 1 root root 5.3M Mar 16 22:18 tap -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 vlan -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 ipvlan -rwxr-xr-x 1 root root 4.2M Mar 16 22:18 loopback -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 macvlan -rwxr-xr-x 1 root root 5.3M Mar 16 22:18 ptp -rwxr-xr-x 1 root root 5.5M Mar 16 22:18 bridge -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 dummy -rwxr-xr-x 1 root root 5.0M Mar 16 22:18 host-device -rwxr-xr-x 1 root root 4.9M Mar 16 22:18 portmap -rwxr-xr-x 1 root root 4.4M Mar 16 22:18 sbr -rwxr-xr-x 1 root root 4.2M Mar 16 22:18 tuning -rwxr-xr-x 1 root root 4.5M Mar 16 22:18 vrf -rwxr-xr-x 1 root root 5.5M Mar 16 22:18 firewall -rwxr-xr-x 1 root root 4.9M Mar 16 22:18 bandwidth -rwxr-xr-x 1 root root 2.7M Jul 15 2025 flannel -rwxr-xr-x 1 root root 66M Jul 15 2025 calico-ipam -rwxr-xr-x 1 root root 66M Jul 15 2025 calico