Deploying Kubernetes with kubeadm

Kubernetes can be installed and deployed using kubeadm.

Installation flow

Preparation

Disable swap by commenting out the swap line in /etc/fstab.
Set the hostname: hostnamectl set-hostname <hostname>.
Enable IP forwarding in the kernel:

echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Install containerd and runc.
You need to install the following packages on every machine:
- kubeadm: the command used to initialize the cluster.
- kubelet: runs on every node in the cluster and starts Pods and containers, etc.
- kubectl: the command-line tool used to talk to the cluster.

Control plane and worker nodes

Control plane node: initialize the cluster with kubeadm init.
Worker nodes: join an existing cluster with kubeadm join.

Installing kubeadm

See the kubeadm install documentation for details.

sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip installing it
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# If the `/etc/apt/keyrings` directory does not exist, create it before the curl command. See the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This operation overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list.
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl   # pin their versions

Creating a cluster with kubeadm

First initialize the control-plane node: kubeadm init. Make sure the machine has ample memory (16G or more is recommended).

# kubeadm pulls images from registry.k8s.io by default, but that domain is unreachable in China (or some network environments), so we pull from the Aliyun mirror registry instead.
root@slpc:~# kubeadm config images pull   --image-repository=registry.aliyuncs.com/google_containers
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.34.3
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.12.1
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.10.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.6.5-0

# Initialize the cluster with kubeadm init, specifying the Aliyun mirror registry and a pod network CIDR of 10.244.0.0/16
root@slpc:~# kubeadm init --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.34.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local slpc] and IPs [10.96.0.1 192.168.232.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost slpc] and IPs [192.168.232.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost slpc] and IPs [192.168.232.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 502.596944ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.232.128:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 1.002103972s
[control-plane-check] kube-scheduler is healthy after 1.195871396s
[control-plane-check] kube-apiserver is healthy after 3.001422611s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node slpc as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node slpc as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: zdd3y8.g6yz49h5q1ztz75y
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.232.128:6443 --token zdd3y8.g6yz49h5q1ztz75y \
	--discovery-token-ca-cert-hash sha256:08377aeb0fa21dfceeecfece817b4263c6eddf11ee1a09e68c6f9f9ca7bc03a7

Install a network plugin; here we choose the flannel network plugin:

# Download flannel
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

root@slpc:~# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

To let a non-root user run kubectl, run the following commands:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Or, if you are the root user, run:

export KUBECONFIG=/etc/kubernetes/admin.conf

Then check the cluster status with kubectl:

root@slpc:~# kubectl get pods -A
NAMESPACE      NAME                           READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-lb9mt          1/1     Running   0          58m
kube-system    coredns-7cc97dffdd-5vvvk       1/1     Running   0          62m
kube-system    coredns-7cc97dffdd-bzhww       1/1     Running   0          62m
kube-system    etcd-slpc                      1/1     Running   8          62m
kube-system    kube-apiserver-slpc            1/1     Running   9          62m
kube-system    kube-controller-manager-slpc   1/1     Running   0          62m
kube-system    kube-proxy-68bfc               1/1     Running   0          62m
kube-system    kube-scheduler-slpc            1/1     Running   10         62m

So far we only have a single node; we need to add worker nodes. Before adding a worker node, prepare the following:

Every worker node to be joined has already installed the components required by kubeadm, such as kubeadm, kubelet, and a container runtime.
A running kubeadm cluster created by the kubeadm init command, following the steps given in the “Creating a cluster with kubeadm” documentation.
You need superuser privileges on the node.

See Adding Linux worker nodes for reference.

On the control-plane node, generate the command that worker nodes will use to join the Kubernetes cluster:

root@slpc:~# kubeadm token create --print-join-command
kubeadm join 192.168.232.128:6443 --token faro80.iwwif9f0duu8uv0k --discovery-token-ca-cert-hash sha256:08377aeb0fa21dfceeecfece817b4263c6eddf11ee1a09e68c6f9f9ca7bc03a7

Worker nodes need containerd installed and configured correctly:

# Generate the default configuration
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# Enable the systemd cgroup driver (must match kubelet)
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

# Restart containerd
sudo systemctl restart containerd
sudo systemctl enable containerd

Verify that the CRI is working:

crictl info   # returns json info on success

Run the following on the worker node:

root@node1:~# kubeadm join 192.168.232.128:6443 --token faro80.iwwif9f0duu8uv0k --discovery-token-ca-cert-hash sha256:08377aeb0fa21dfceeecfece817b4263c6eddf11ee1a09e68c6f9f9ca7bc03a7
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 502.295181ms
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Image-pull failures (blocked by the GFW)

# Pull images (use a domestic mirror to avoid registry.k8s.io failures)
docker pull registry.aliyuncs.com/google_containers/pause:3.10
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.34.3
docker pull docker.io/flannel/flannel:v0.25.6
docker pull docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel2

# Re-tag to the names Kubernetes expects (critical!)
docker tag registry.aliyuncs.com/google_containers/pause:3.10 registry.k8s.io/pause:3.10.1
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.34.3 registry.k8s.io/kube-proxy:v1.34.3

# Package them
docker save -o k8s-node-images.tar \
  registry.k8s.io/pause:3.10.1 \
  registry.k8s.io/kube-proxy:v1.34.3 \
  docker.io/flannel/flannel:v0.25.6 \
  docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel2

scp k8s-node-images.tar root@192.168.232.137:/root/   # node1
scp k8s-node-images.tar root@192.168.232.138:/root/   # node2

# Run on node1 and node2 respectively
sudo ctr -n=k8s.io images import /root/k8s-node-images.tar

# Restart kubelet
sudo systemctl restart kubelet

Check the cluster node information:

root@slpc:~# kubectl get nodes
NAME    STATUS   ROLES           AGE     VERSION
node1   Ready    <none>          18h     v1.34.3
node2   Ready    <none>          3h15m   v1.34.3
slpc    Ready    control-plane   3d17h   v1.34.2

Installing the Kubernetes Dashboard

Create the namespace and deploy:

# Deploy the Dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

Verify the installation:

kubectl get pods -n kubernetes-dashboard
kubectl get svc -n kubernetes-dashboard

If there is an image-pull problem, edit the Deployment and change the image address to a domestic mirror:

# Change the dashboard image
kubectl set image deployment/kubernetes-dashboard \
  kubernetes-dashboard=registry.cn-hangzhou.aliyuncs.com/google_containers/dashboard:v2.7.0 \
  -n kubernetes-dashboard

# Change the metrics-scraper image
kubectl set image deployment/dashboard-metrics-scraper \
  dashboard-metrics-scraper=registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-scraper:v1.0.8 \
  -n kubernetes-dashboard

kubectl set image automatically triggers a rolling update: the old Pod is deleted and a new Pod starts with the new image.

Start the Kubernetes proxy:

kubectl proxy --address='0.0.0.0' --port=8001

# Create an account named admin-user and grant it the cluster-admin global admin role.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

Get the login Token:

root@slpc:~# kubectl -n kubernetes-dashboard create token admin-user
eyJhbGciOiJSUzI1NiIsImtpZCI6ImI0NmpHT3NnaUx2a0xVbno3Q2QtZklMdDdmNU1DYmVYVFRYRVQ2NjdZdFkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzY1ODk5OTI4LCJpYXQiOjE3NjU4OTYzMjgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiN2EwNDFkMzQtMzBmNi00NjZjLTg3MGQtMzhjNDMyNzMwZmUzIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiY2Y3MDFlYzgtMzA2Yy00OWU2LTk1NWEtMWY1MWZmN2E5YzIxIn19LCJuYmYiOjE3NjU4OTYzMjgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.bcwIpuhFDDK8fWlRv7yqT9MTJINq1tLujcRAjyjNdmU7ctJeUYy0WL54m152w8KcIc0TJ0IfXa8ZoE1ygqTtoR3DDrgkdPIMA08h5l11I3vQNpqUNptsjCLTqsTmE8FkoXoXfD2Aov3--obOiewykXsU72ppRHc0Fb0KlKDPLHJVzmObrzvLGngmfyiUU-D6qkAEtVpo6-DEZYMFn2KCZrboqgOztEpLH9hsEhOQkztC5QFMpqljwreVGqG3Px89GpQ97Zg3a3Qkkmjq_DBkoXbdV_2Qi8ypFB3F6LaDh7V1n_Szhr8e-ydjD8-7QFpJFRsHP0lxyj3b8k3tjS7Cqg

Open the Dashboard in a browser:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Enter the token above and click Log in.

Installing Metrics Server

Metrics Server is the official, lightweight resource metrics aggregator (a Resource Metrics API implementation) recommended by Kubernetes. It collects core resource metrics such as CPU and memory usage from the Nodes and Pods in the cluster.

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Change the image address to a domestic mirror
image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.8.0

# Edit the Deployment and add the --kubelet-insecure-tls flag to skip TLS certificate verification
- --kubelet-insecure-tls

kubectl apply -f components.yaml

Verification:

postgres@slpc:$ kubectl top nodes
NAME    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
node1   36m          0%       616Mi           33%
node2   29m          0%       1003Mi          55%
slpc    198m         2%       6877Mi          88%

root@slpc:~# kubectl top pods -A
NAMESPACE              NAME                                      CPU(cores)   MEMORY(bytes)
kube-flannel           kube-flannel-ds-6lt2r                     8m           12Mi
kube-flannel           kube-flannel-ds-7pncp                     7m           13Mi
kube-flannel           kube-flannel-ds-plldn                     7m           21Mi
kube-system            coredns-7cc97dffdd-5vvvk                  3m           17Mi
kube-system            coredns-7cc97dffdd-bzhww                  4m           15Mi
kube-system            etcd-slpc                                 33m          47Mi
kube-system            kube-apiserver-slpc                       57m          254Mi
kube-system            kube-controller-manager-slpc              28m          52Mi
kube-system            kube-proxy-68bfc                          1m           15Mi
kube-system            kube-proxy-854cx                          1m           26Mi
kube-system            kube-proxy-b6jzz                          1m           13Mi
kube-system            kube-scheduler-slpc                       9m           29Mi
kube-system            metrics-server-587957767f-2wg97           5m           20Mi
kubernetes-dashboard   dashboard-metrics-scraper-76c8db4-46f52   1m           12Mi
kubernetes-dashboard   kubernetes-dashboard-7547fc74b4-nls72     1m           17Mi