目录

1. 说明

1.1 软硬件环境

1.2 设计目标

2 实现

2.1 准备工作

- 香橙派 (k8s-master-1)

- 树莓派 (k8s-node-1)

- 两派都要干的事

2.2 containerd 安装与设置

2.3 安装 

3 遇到的问题

3.1 k8s-master-1

3.2 k8s-node-1 

4 相关命令

5 Tips

6 参考

---

1. 说明

1.1 软硬件环境

香橙派4LTS: 命名 k8s-master-1 / 192.168.0.106 / Ubuntu 22.04, 4G / 125G SD卡
树莓派4B : 命名 k8s-node-1 / 192.168.0.104 / Raspi OS(Debian 11), 4G / 64G SD卡

1.2 设计目标

• 实现K8s集群 (基于containerd V1.62和K8s V1.27)
• 在其上部署MariaDB Galera Cluster集群

2 实现

2.1 准备工作

- 香橙派 (k8s-master-1)

#加源

cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu-ports/ jammy main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ jammy main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ jammy-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ jammy-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ jammy-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ jammy-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ jammy-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ jammy-backports main restricted universe multiverse 
EOF

 #添加加载的内核模块

tee /etc/modules-load.d/containerd.conf<<EOF
overlay
br_netfilter
EOF

 #加载内核模块

modprobe overlay && modprobe br_netfilter

 #设置并应用内核参数

tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

sysctl --system

- 树莓派 (k8s-node-1)

#加源

cat > /etc/apt/sources.list <<EOF
#将文件内容用以下内容替换，换上科大源
deb https://mirrors.ustc.edu.cn/debian/ buster main contrib non-free
# deb-src http://mirrors.ustc.edu.cn/debian buster main contrib non-free
deb https://mirrors.ustc.edu.cn/debian/ buster-updates main contrib non-free
# deb-src http://mirrors.ustc.edu.cn/debian buster-updates main contrib non-free
deb https://mirrors.ustc.edu.cn/debian-security buster/updates main contrib non-free
# deb-src http://mirrors.ustc.edu.cn/debian-security/ buster/updates main non-free contrib
#将文件内容用以下内容替换，换上清华源(针对aarch64用户)
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-updates main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bullseye-backports main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian-security bullseye-security main contrib non-free
EOF

- 两派都要干的事

  修改/etc/hosts文件

192.168.0.106 k8s-master-1
192.168.0.104 k8s-node-1
199.232.28.133 raw.githubusercontent.com # 以便kubectl apply时能找到

加k8s源 

curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat > /etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

 检查更新及安装更新

apt update
apt upgrade -y

 安装所需附件

apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

2.2 containerd 安装与设置

#启用 docker 存储库

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
# 支持x86架构64位cpu
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# 支持arm64架构cpu
add-apt-repository "deb [arch=arm64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt update && apt install -y containerd.io

#生成containerd的配置文件

containerd config default | tee /etc/containerd/config.toml >/dev/null 2>&1

 #修改cgroup Driver为systemd

sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

 #将镜像源设置为阿里云 google_containers 镜像源

sed -i 's/k8s.gcr.io/registry.aliyuncs.com\/google_containers/g' /etc/containerd/config.toml

systemctl daemon-reload
systemctl start containerd
systemctl enable containerd.service

2.3 安装 

临时关闭Swap, 在我的香橙派中，重启后swap分区又会出来,（试过很多方法都不行），简直是打不死的小强，后期在配置文件（/etc/systemd/system/kubelet.service.d/10-kubeadm.conf）中添加参数--fail-swap-on=false解决，参看遇到的问题一节

# swapoff -a         # 临时关闭
# sed -ri 's/.*swap.*/#&/' /etc/fstab    # 没啥用

apt -y install kubeadm kubelet kubectl # 按最新的来玩

# 固定版本不更新(暂时如此，免得出幺蛾子)
apt-mark hold kubelet kubeadm kubectl 
systemctl enable kubelet.service

# 加入环境变量
echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> /etc/profile
source /etc/profile

master server初始化 (node不需要走init)，这里用了区域镜像，否则等到猴年马月..

kubeadm init --apiserver-advertise-address=192.168.0.106 --pod-network-cidr=10.244.0.0/16 \
 --image-repository registry.aliyuncs.com/google_containers 

node 加入 

# 成功后，会得到与token一起的加入提示命令, 在node1运行之
kubeadm join 192.168.0.106:6443 --token {xxx} \
        --discovery-token-ca-cert-hash sha256:{yyy}


[preflight] Running pre-flight checks
        [WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

  这时的集群都是NotReady状态的

kubectl get nodes
NAME           STATUS     ROLES                  AGE     VERSION
k8s-master-1   NotReady   control-plane,master   8m15s   v1.27.1
k8s-node-1     NotReady   <none>                 2m30s   v1.27.1

在k8s-master-1上安装 Flannel 网络插件

export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

几经周折，完成nodes and pods为running状态，乌拉! 

安装Helm

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

3 遇到的问题

3.1 k8s-master-1

- 如果删除不了swap交换分区，则kubelet服务会启动不来，由于K8s1.21后的版本能支持swap，所以调整参数(--fail-swap-on=false) 即可，设置方法：

cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --fail-swap-on=false

在启动命令末尾加上： --fail-swap-on=false ，然后reload配置
systemctl daemon-reload
systemctl start kubelet

-  "The connection to the server localhost:8080 was refused - did you specify the right host or port?" 

cd /etc/kubernetes/

查看到有个文件：kubelet.conf, 执行命令
echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> /etc/profile
source /etc/profile

再次查看 kubectl get pods 已经正常。

原因： kubernetes master没有与本机绑定，集群初始化的时候没有绑定，此时设置在本机的环境变量即可解决问题。

3.2 k8s-node-1 

 - 加入时，遇到提示：CGROUPS_MEMORY: missing,

解决办法：编辑 /boot/cmdline.txt，加入：

cgroup_enable=memory cgroup_memory=1

Node为NotReady状态, 日志提示："Unable to update cni config: No networks found in /etc/cni/net.d"

解决办法: 删除 --network-plugin=cni

nano /var/lib/kubelet/kubeadm-flags.env

# KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.6"
=>
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.6"

 "The following signatures couldn't be verified because the public key is not available: {key}"

解决办法

gpg --keyserver keyserver.ubuntu.com --recv  {key}
gpg --export --armor  {key} | sudo apt-key add -

 "container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

解决办法: CNI is not initialized in k8s v1.16.4 · Issue #1236 · flannel-io/flannel · GitHub

cat <<EOL > /etc/cni/net.d/10-flannel.conflist 
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
EOL

 "Failed to create pod sandbox: open /run/systemd/resolve/resolv.conf: no such file or directory"

解决办法

systemctl enable systemd-resolved.service
systemctl start systemd-resolved

 "failed to pull image \"registry.k8s.io/pause:"

解决方法:

查看日志 journalctl -xeu kubelet
### 生成 containerd 的默认配置文件
containerd config default > /etc/containerd/config.toml 
### 查看 sandbox 的默认镜像仓库在文件中的第几行 
cat /etc/containerd/config.toml | grep -n "sandbox_image"  
### 使用 vim 编辑器 定位到 sandbox_image，将 仓库地址修改成 k8simage/pause:3.6
vim /etc/containerd/config.toml  
sandbox_image = "k8simage/pause:3.6"  
### 重启 containerd 服务  
systemctl daemon-reload  
systemctl restart containerd 

操作时发现当前用户不是 kubernetes-admin@kubernetes, "Error from server (Forbidden): pods "kube-proxy-zvkbq" is forbidden: User "system:node:k8s-master-1" cannot get resource "pods/log" in API group "" in the namespace "kube-system"

export KUBECONFIG=/etc/kubernetes/admin.conf

无厘头的 kube-flannel-ds-xxx 或 kube-proxy-xxx 出现 CrashLooppBackOff, 要检查安装containerd时是否有修改cgroup Driver为systemd

sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

4 相关命令

kubeadm token list  # 查看 tokens
kubeadm token create # 重新生成
kubeadm reset -f # 重置

kubectl logs -n kube-system kube-proxy-zvkbq {pod name} #查日志
kubectl auth can-i create namespace # 查询是否有权做某事

kubectl get nodes
kubectl describe node k8s-node-1 # 查看节点k8s-node-1
kubectl describe nodes # 查看所有节点

kubectl get pods -o wide -A # 查看pods
kubectl describe pod -n kube-system kube-proxy-nxr68 # 查看pod

kubectl delete node <node name>
kubectl delete pod -n kube-flannel <pod  name>

kubectl config current-context # 查看当前上下文(用户)
kubectl config view # 查看配置
kubectl config get-contexts # 上下文(用户)列表
kubectl get role -n kube-public

journalctl -f -u kubelet    # 查看kubelet日志

5 Tips

• 在树莓派中，最好还是装个proxychains，科学找源，避免找不到或解析问题
• 关于/etc/kubernetes/ 目录下的四个文件，其作用是:
  admin.conf kubectl与apiServer打交道的文件
  controller-manager.conf controllerManager与apiServer打交道的文件
  kubelet.conf kubelet与apiServer打交道的文件
  scheduler.conf scheduler与apiServer打交道的文件

6 参考

使用树莓派搭建K8S集群（ARM64架构，附安装脚本）_树莓派集群_NaclChan的博客-CSDN博客

Creating a cluster with kubeadm | Kubernetes

Kubernetes安装与踩坑_--apiserver-advertise-address___walden的博客-CSDN博客

使用Kubeadm(1.13+)快速搭建Kubernetes集群

k8s笔记17--ubuntu & k8s 开启 swap功能_k8s swap_昕光xg的博客-CSDN博客