Kubernetes : ajouter un noeud master
Bonjour,
Aujourd'hui, un billet portant sur l'ajout d'un node master a un cluste existant.
Context : cluster avec 1 master sous ubuntu 22 et 3 nodes worker sous Flatcar.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
flc-node1.adm.securmail.fr Ready <none> 124d v1.30.1
flc-node2.adm.securmail.fr Ready <none> 124d v1.30.1
flc-node3.adm.securmail.fr Ready <none> 5d18h v1.30.1
k8s-master.adm.securmail.fr Ready control-plane 316d v1.30.0
En premier lieu, il faut récuper la configuration du cluster :
kubectl get cm kubeadm-config -n kube-system -o yaml
apiVersion: v1
data:
ClusterConfiguration: |
apiServer:
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: k8s-master.adm.securmail.fr:6443
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.30.0
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
kind: ConfigMap
metadata:
creationTimestamp: "2023-09-05T08:34:24Z"
name: kubeadm-config
namespace: kube-system
resourceVersion: "38704805"
uid: 0c53a531-2cdf-4b52-984e-a987bacf9c8f
On ne prend que la partie après ClusterConfiguration :
apiServer:
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: k8s-master.adm.securmail.fr:6443
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.30.0
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
Il faut ensuite exécuter une commande pour upload les certificats du control-plane et récupérer la clé qui sera utilisée lors du join :
kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
531c03b66e0f2b21575dd4d5401e3bed0d23a48309f6b0c968b482d49bd8ee7c
Création de la commande pour join un node :
kubeadm token create --print-join-command
kubeadm join k8s-master.adm.securmail.fr:6443 --token 2jw15k.4jk2abk1lspb1tdk --discovery-token-ca-cert-hash sha256:acac9c8XXXXX
Et on rajoute à la fin la partie control plane/certificat vu qu'il s'agit d'un master (en utilisant la clé récupéré plus haut) :
--control-plane --certificate-key 531c03XXXXX
Ce qui donne :
kubeadm join k8s-master.adm.securmail.fr:6443 --token 2jw15k.4jk2abk1lspb1tdk --discovery-token-ca-cert-hash sha256:acac9c8XXXXX --control-plane --certificate-key 531c03XXXXX
Création du fichier d'installation :
variant: flatcar
version: 1.0.0
passwd:
users:
- name: gizmo
ssh_authorized_keys:
- ssh-rsa AAAAB3Nzblabla
groups: [ sudo, docker ]
storage:
links:
- target: /opt/extensions/kubernetes/kubernetes-v1.30.1-x86-64.raw
path: /etc/extensions/kubernetes.raw
hard: false
files:
- path: /etc/hostname
contents:
inline: |
flc-master1.adm.securmail.fr
- path: /etc/systemd/network/00-eth0.network
contents:
inline: |
[Match]
Name=eth0
[Network]
Address=192.168.0.37/24
Address=2a0e:f41:0:3::1b/64
Gateway=192.168.0.21
DNS=192.168.0.21
DNS=2a0e:f41:0:3::1
[Route]
Metric=1024
Destination=::/0
Gateway=2a0e:f41:0:3::1
GatewayOnLink=yes
- path: /etc/flatcar/update.conf
overwrite: true
contents:
inline: |
GROUP=stable
SERVER=http://nebraska.adm.securmail.fr/v1/update/
MACHINE_ALIAS="flc-master1.adm.securmail.fr"
- path: /etc/sysupdate.kubernetes.d/kubernetes.conf
contents:
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-v1.30.conf
- path: /etc/sysupdate.d/noop.conf
contents:
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
- path: /opt/extensions/kubernetes/kubernetes-v1.30.1-x86-64.raw
contents:
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-v1.30.1-x86-64.raw
systemd:
units:
- name: systemd-sysupdate.timer
enabled: true
- name: systemd-sysupdate.service
dropins:
- name: kubernetes.conf
contents: |
[Service]
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
ExecStartPost=/usr/bin/sh -c "[[ $(cat /tmp/kubernetes) != $(cat /tmp/kubernetes-new) ]] && touch /run/reboot-required"
- name: kubeadm.service
enabled: true
contents: |
[Unit]
Description=Kubeadm service
Requires=containerd.service
After=containerd.service
[Service]
ExecStart=/usr/bin/kubeadm join k8s-master.adm.securmail.fr:6443 --token 2jw15k.4jk2abk1lspb1tdk --discovery-token-ca-cert-hash sha256:acac9c802744bb0697346e20ad2c1c0dd44e55ad82fc944cc3d531d491ae96c8 --control-plane --certificate-key 531c03b66e0f2b21575dd4d5401e3bed0d23a48309f6b0c968b482d49bd8ee7c
[Install]
WantedBy=multi-user.target
Avec la commande de join custom pour un node master à la fin.
Conversion en version butane utilisable par l'installeur via docker :
cat flatcar-master.bu | docker run --rm -i quay.io/coreos/butane:latest > ignition-master.json
Installation de Flatcar :
curl -L0 http://deploy.adm.securmail.fr/preseed/ignition-master.json > ignition-master.json
sudo flatcar-install -d /dev/vda -i ignition-master.json
sudo reboot -i
Une fois l'installation faite et le reboot, au bout de quelques secondes le nouveau node master apparait :
kubectl get nodes
NAME STATUS ROLES AGE VERSION
flc-master1.adm.securmail.fr Ready control-plane 4m22s v1.30.1
flc-node1.adm.securmail.fr Ready <none> 119d v1.30.1
flc-node2.adm.securmail.fr Ready <none> 119d v1.30.1
flc-node3.adm.securmail.fr Ready <none> 6h17m v1.30.1
k8s-master.adm.securmail.fr Ready control-plane 310d v1.30.0
Il restera à gérer la partie load balancer.
Have fun.