Kubernetes : ajouter un noeud master

Bonjour,

Aujourd'hui, un billet portant sur l'ajout d'un node master a un cluste existant.

Context : cluster avec 1 master sous ubuntu 22 et 3 nodes worker sous Flatcar.

kubectl get nodes
NAME                           STATUS   ROLES           AGE     VERSION
flc-node1.adm.securmail.fr     Ready    <none>          124d    v1.30.1
flc-node2.adm.securmail.fr     Ready    <none>          124d    v1.30.1
flc-node3.adm.securmail.fr     Ready    <none>          5d18h   v1.30.1
k8s-master.adm.securmail.fr    Ready    control-plane   316d    v1.30.0

En premier lieu, il faut récuper la configuration du cluster :

kubectl get cm kubeadm-config -n kube-system -o yaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta3
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controlPlaneEndpoint: k8s-master.adm.securmail.fr:6443
    controllerManager: {}
    dns: {}
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: registry.k8s.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.30.0
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
kind: ConfigMap
metadata:
  creationTimestamp: "2023-09-05T08:34:24Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "38704805"
  uid: 0c53a531-2cdf-4b52-984e-a987bacf9c8f

On ne prend que la partie après ClusterConfiguration :

apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: k8s-master.adm.securmail.fr:6443
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.30.0
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

Il faut ensuite exécuter une commande pour upload les certificats du control-plane et récupérer la clé qui sera utilisée lors du join :

kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
531c03b66e0f2b21575dd4d5401e3bed0d23a48309f6b0c968b482d49bd8ee7c

Création de la commande pour join un node :

kubeadm token create --print-join-command
kubeadm join k8s-master.adm.securmail.fr:6443 --token 2jw15k.4jk2abk1lspb1tdk --discovery-token-ca-cert-hash sha256:acac9c8XXXXX

Et on rajoute à la fin la partie control plane/certificat vu qu'il s'agit d'un master (en utilisant la clé récupéré plus haut) :

--control-plane --certificate-key 531c03XXXXX

Ce qui donne :

kubeadm join k8s-master.adm.securmail.fr:6443 --token 2jw15k.4jk2abk1lspb1tdk --discovery-token-ca-cert-hash sha256:acac9c8XXXXX --control-plane --certificate-key 531c03XXXXX

Création du fichier d'installation :

variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: gizmo
      ssh_authorized_keys:
        - ssh-rsa AAAAB3Nzblabla
      groups: [ sudo, docker ]
storage:
  links:
    - target: /opt/extensions/kubernetes/kubernetes-v1.30.1-x86-64.raw
      path: /etc/extensions/kubernetes.raw
      hard: false
  files:
    - path: /etc/hostname
      contents:
        inline: |
          flc-master1.adm.securmail.fr
    - path: /etc/systemd/network/00-eth0.network
      contents:
        inline: |
          [Match]
          Name=eth0

          [Network]
          Address=192.168.0.37/24
          Address=2a0e:f41:0:3::1b/64
          Gateway=192.168.0.21
          DNS=192.168.0.21
          DNS=2a0e:f41:0:3::1

          [Route]
          Metric=1024
          Destination=::/0
          Gateway=2a0e:f41:0:3::1
          GatewayOnLink=yes
    - path: /etc/flatcar/update.conf
      overwrite: true
      contents:
        inline: |
          GROUP=stable
          SERVER=http://nebraska.adm.securmail.fr/v1/update/
          MACHINE_ALIAS="flc-master1.adm.securmail.fr"
    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-v1.30.conf
    - path: /etc/sysupdate.d/noop.conf
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
    - path: /opt/extensions/kubernetes/kubernetes-v1.30.1-x86-64.raw
      contents:
        source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-v1.30.1-x86-64.raw
systemd:
  units:
    - name: systemd-sysupdate.timer
      enabled: true
    - name: systemd-sysupdate.service
      dropins:
        - name: kubernetes.conf
          contents: |
            [Service]
            ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
            ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
            ExecStartPost=/usr/bin/sh -c "[[ $(cat /tmp/kubernetes) != $(cat /tmp/kubernetes-new) ]] && touch /run/reboot-required"
    - name: kubeadm.service
      enabled: true
      contents: |
        [Unit]
        Description=Kubeadm service
        Requires=containerd.service
        After=containerd.service
        [Service]
        ExecStart=/usr/bin/kubeadm join k8s-master.adm.securmail.fr:6443 --token 2jw15k.4jk2abk1lspb1tdk --discovery-token-ca-cert-hash sha256:acac9c802744bb0697346e20ad2c1c0dd44e55ad82fc944cc3d531d491ae96c8 --control-plane --certificate-key 531c03b66e0f2b21575dd4d5401e3bed0d23a48309f6b0c968b482d49bd8ee7c
        [Install]
        WantedBy=multi-user.target

Avec la commande de join custom pour un node master à la fin.

Conversion en version butane utilisable par l'installeur via docker :

cat flatcar-master.bu | docker run --rm -i quay.io/coreos/butane:latest > ignition-master.json

Installation de Flatcar :

curl -L0 http://deploy.adm.securmail.fr/preseed/ignition-master.json > ignition-master.json
sudo flatcar-install -d /dev/vda -i ignition-master.json
sudo reboot -i

Une fois l'installation faite et le reboot, au bout de quelques secondes le nouveau node master apparait :

kubectl get nodes
NAME                           STATUS   ROLES           AGE     VERSION
flc-master1.adm.securmail.fr   Ready    control-plane   4m22s   v1.30.1
flc-node1.adm.securmail.fr     Ready    <none>          119d    v1.30.1
flc-node2.adm.securmail.fr     Ready    <none>          119d    v1.30.1
flc-node3.adm.securmail.fr     Ready    <none>          6h17m   v1.30.1
k8s-master.adm.securmail.fr    Ready    control-plane   310d    v1.30.0

Il restera à gérer la partie load balancer.

Have fun.