使用 kubeadm 部署 Kubernetes

Kubernetes 可以通过 kubeadm 进行安装部署。

安装流程

准备工作

  1. 关闭 swap,将 /etc/fstab 中关于 swap 的行注释掉。
  2. 修改主机名:hostnamectl set-hostname <hostname>。
  3. 内核开启 IP 转发:
echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
  1. 安装 containerd、runc。
  2. 你需要在每台机器上安装以下软件包:
    • kubeadm:用来初始化集群的指令。
    • kubelet:在集群中的每个节点上用来启动 Pod 和容器等。
    • kubectl:用来与集群通信的命令行工具。

控制节点与工作节点

  • 控制节点:通过 kubeadm init 初始化集群。
  • 工作节点:通过 kubeadm join 加入已有集群。

kubeadm 安装

安装方法参见:kubeadm 安装文档

sudo apt-get update
# apt-transport-https 可能是一个虚拟包(dummy package);如果是的话,你可以跳过安装这个包
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# 如果 `/etc/apt/keyrings` 目录不存在,则应在 curl 命令之前创建它,请阅读下面的注释。
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# 此操作会覆盖 /etc/apt/sources.list.d/kubernetes.list 中现存的所有配置。
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl   # 锁定其版本

使用 kubeadm 创建集群

首先初始化控制平面节点:kubeadm init,建议机器内存资源要充足(16G 以上)。

# kubeadm 默认从 registry.k8s.io 拉取镜像,但该域名在国内(或某些网络环境)无法访问,因此需要从阿里云镜像仓库拉取。
root@slpc:~# kubeadm config images pull   --image-repository=registry.aliyuncs.com/google_containers
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.12.1
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.10.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.6.5-0

# 通过 kubeadm init 命令初始化集群,并指定使用阿里云镜像仓库,指定 pod 网络地址段(CIDR 格式)为 10.244.0.0/16
root@slpc:~# kubeadm init --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.34.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local slpc] and IPs [10.96.0.1 192.168.232.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost slpc] and IPs [192.168.232.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost slpc] and IPs [192.168.232.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 502.596944ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.232.128:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 1.002103972s
[control-plane-check] kube-scheduler is healthy after 1.195871396s
[control-plane-check] kube-apiserver is healthy after 3.001422611s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node slpc as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node slpc as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: zdd3y8.g6yz49h5q1ztz75y
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.232.128:6443 --token zdd3y8.g6yz49h5q1ztz75y \
	--discovery-token-ca-cert-hash sha256:08377aeb0fa21dfceeecfece817b4263c6eddf11ee1a09e68c6f9f9ca7bc03a7

安装网络插件,这里选择 flannel 网络插件:

# 下载 flannel
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

root@slpc:~# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

要使非 root 用户可以运行 kubectl,请运行以下命令:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

或者,如果是 root 用户,则可以运行:

export KUBECONFIG=/etc/kubernetes/admin.conf

然后通过 kubectl 查看集群状态:

root@slpc:~# kubectl get pods -A
NAMESPACE      NAME                           READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-lb9mt          1/1     Running   0          58m
kube-system    coredns-7cc97dffdd-5vvvk       1/1     Running   0          62m
kube-system    coredns-7cc97dffdd-bzhww       1/1     Running   0          62m
kube-system    etcd-slpc                      1/1     Running   8          62m
kube-system    kube-apiserver-slpc            1/1     Running   9          62m
kube-system    kube-controller-manager-slpc   1/1     Running   0          62m
kube-system    kube-proxy-68bfc               1/1     Running   0          62m
kube-system    kube-scheduler-slpc            1/1     Running   10         62m

到目前为止,我们仅有一个节点,我们需要添加工作节点。在添加工作节点之前,需要准备如下:

  • 每个要加入的工作节点都已安装 kubeadm 中所需的组件,例如 kubeadm、kubelet 和容器运行时。
  • 一个正在运行的、由 kubeadm init 命令所创建的 kubeadm 集群,且该集群的创建遵循使用 kubeadm 创建集群文档中所给的步骤。
  • 你需要对节点拥有超级用户权限。

可参考 添加 Linux 工作节点

在控制节点执行生成工作节点加入 Kubernetes 集群的命令:

root@slpc:~# kubeadm token create --print-join-command
kubeadm join 192.168.232.128:6443 --token faro80.iwwif9f0duu8uv0k --discovery-token-ca-cert-hash sha256:08377aeb0fa21dfceeecfece817b4263c6eddf11ee1a09e68c6f9f9ca7bc03a7

工作节点需要安装并正确配置 containerd:

# 生成默认配置
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# 启用 systemd cgroup 驱动(必须与 kubelet 一致)
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

# 重启 containerd
sudo systemctl restart containerd
sudo systemctl enable containerd

验证 CRI 是否正常:

crictl info   # 正常情况返回 json 信息。

在工作节点执行如下:

root@node1:~# kubeadm join 192.168.232.128:6443 --token faro80.iwwif9f0duu8uv0k --discovery-token-ca-cert-hash sha256:08377aeb0fa21dfceeecfece817b4263c6eddf11ee1a09e68c6f9f9ca7bc03a7
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 502.295181ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

镜像拉取失败问题(被墙限制)

# 拉取镜像(使用国内镜像加速,避免 registry.k8s.io 失败)
docker pull registry.aliyuncs.com/google_containers/pause:3.10
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.34.3
docker pull docker.io/flannel/flannel:v0.25.6
docker pull docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel2

# 重命名 tag 为 Kubernetes 期望的名称(关键!)
docker tag registry.aliyuncs.com/google_containers/pause:3.10 registry.k8s.io/pause:3.10.1
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.34.3 registry.k8s.io/kube-proxy:v1.34.3

# 打包
docker save -o k8s-node-images.tar \
  registry.k8s.io/pause:3.10.1 \
  registry.k8s.io/kube-proxy:v1.34.3 \
  docker.io/flannel/flannel:v0.25.6 \
  docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel2

scp k8s-node-images.tar root@192.168.232.137:/root/   # node1
scp k8s-node-images.tar root@192.168.232.138:/root/   # node2

# 在 node1 和 node2 上分别执行
sudo ctr -n=k8s.io images import /root/k8s-node-images.tar

# 重启 kubelet
sudo systemctl restart kubelet

查看集群节点信息:

root@slpc:~# kubectl get nodes
NAME    STATUS   ROLES           AGE     VERSION
node1   Ready    <none>          18h     v1.34.3
node2   Ready    <none>          3h15m   v1.34.3
slpc    Ready    control-plane   3d17h   v1.34.2

安装 Kubernetes Dashboard

  1. 创建命名空间并部署:
# 部署 Dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
  1. 验证安装:
kubectl get pods -n kubernetes-dashboard
kubectl get svc -n kubernetes-dashboard
  1. 如果镜像拉取有问题,可编辑 Deployment,将镜像地址改为国内镜像地址:
# 修改 dashboard 镜像
kubectl set image deployment/kubernetes-dashboard \
  kubernetes-dashboard=registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0 \
  -n kubernetes-dashboard

# 修改 metrics-scraper 镜像
kubectl set image deployment/dashboard-metrics-scraper \
  dashboard-metrics-scraper=registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8 \
  -n kubernetes-dashboard

kubectl set image 会自动触发滚动更新,旧 Pod 被删除,新 Pod 使用新镜像启动。

  1. 启动 Kubernetes 代理:
kubectl proxy --address='0.0.0.0' --port=8001
  1. 使用 Token 登录 Dashboard:
# 创建一个名为 admin-user 的账号,并赋予其 cluster-admin 全局管理员权限。
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

获取登录 Token:

root@slpc:~# kubectl -n kubernetes-dashboard create token admin-user
eyJhbGciOiJSUzI1NiIsImtpZCI6ImI0NmpHT3NnaUx2a0xVbno3Q2QtZklMdDdmNU1DYmVYVFRYRVQ2NjdZdFkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzY1ODk5OTI4LCJpYXQiOjE3NjU4OTYzMjgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiN2EwNDFkMzQtMzBmNi00NjZjLTg3MGQtMzhjNDMyNzMwZmUzIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiY2Y3MDFlYzgtMzA2Yy00OWU2LTk1NWEtMWY1MWZmN2E5YzIxIn19LCJuYmYiOjE3NjU4OTYzMjgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.bcwIpuhFDDK8fWlRv7yqT9MTJINq1tLujcRAjyjNdmU7ctJeUYy0WL54m152w8KcIc0TJ0IfXa8ZoE1ygqTtoR3DDrgkdPIMA08h5l11I3vQNpqUNptsjCLTqsTmE8FkoXoXfD2Aov3--obOiewykXsU72ppRHc0Fb0KlKDPLHJVzmObrzvLGngmfyiUU-D6qkAEtVpo6-DEZYMFn2KCZrboqgOztEpLH9hsEhOQkztC5QFMpqljwreVGqG3Px89GpQ97Zg3a3Qkkmjq_DBkoXbdV_2Qi8ypFB3F6LaDh7V1n_Szhr8e-ydjD8-7QFpJFRsHP0lxyj3b8k3tjS7Cqg
  1. 访问 Dashboard,在浏览器中打开:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

输入上面的 token,点击登录。

安装 Metrics Server

Metrics Server 是 Kubernetes 官方推荐的轻量级资源指标聚合器(Resource Metrics API 实现),用于从集群中的节点(Node)和 Pod 收集 CPU 和内存使用率等核心资源指标。

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 更改镜像地址为国内镜像
image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.8.0

# 编辑 Deployment,添加 --kubelet-insecure-tls 参数,跳过 TLS 证书校验
- --kubelet-insecure-tls

kubectl apply -f components.yaml

验证:

postgres@slpc:$ kubectl top nodes
NAME    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
node1   36m          0%       616Mi           33%
node2   29m          0%       1003Mi          55%
slpc    198m         2%       6877Mi          88%

root@slpc:~# kubectl top pods -A
NAMESPACE              NAME                                      CPU(cores)   MEMORY(bytes)
kube-flannel           kube-flannel-ds-6lt2r                     8m           12Mi
kube-flannel           kube-flannel-ds-7pncp                     7m           13Mi
kube-flannel           kube-flannel-ds-plldn                     7m           21Mi
kube-system            coredns-7cc97dffdd-5vvvk                  3m           17Mi
kube-system            coredns-7cc97dffdd-bzhww                  4m           15Mi
kube-system            etcd-slpc                                 33m          47Mi
kube-system            kube-apiserver-slpc                       57m          254Mi
kube-system            kube-controller-manager-slpc              28m          52Mi
kube-system            kube-proxy-68bfc                          1m           15Mi
kube-system            kube-proxy-854cx                          1m           26Mi
kube-system            kube-proxy-b6jzz                          1m           13Mi
kube-system            kube-scheduler-slpc                       9m           29Mi
kube-system            metrics-server-587957767f-2wg97           5m           20Mi
kubernetes-dashboard   dashboard-metrics-scraper-76c8db4-46f52   1m           12Mi
kubernetes-dashboard   kubernetes-dashboard-7547fc74b4-nls72     1m           17Mi