Kubernetes on VirtualBox for fun and profit

Kubernetes, k8s for short, is a vital piece of infrastructure for cloud engineers today. I wanted to dig deep but was always deterred by the lack of a setup that I could freely tinker with, without care of hourly billing rates and without bothering about the specifics of a cloud operating environment. I wanted a setup on my laptop but not a one-node Minikube installation, because I wanted to try real-life multi-node scenarios involving failover of pods running stateful apps. So I set about building my own portable k8s cluster using a laptop and VirtualBox. Along the way I figured I had to do a few ancillary setups (like a local Docker registry) or satisfy some prerequisites (like a usable networking scheme for the VirtualBox VMs). And before I made it repeatable through Ansible or some such mechanism, I wanted to document it in as much detail as possible. This is the first of a clutch of articles that lets you, especially if you’re a k8s newbie in search of a hackable portable setup, get such a setup on your own laptop.

The setup

I have a reasonably beefy laptop — 16 GiB of RAM, 500 GB of non-SSD storage, i5 5th generation quad-core processor with hyperthreading enabled.

I am creating a three-node cluster — on master and two worker nodes. I run CentOS Stream 9 (although if not using Cilium or Calico, CentOS 7.8 should work too) on my three VMs on Oracle VirtualBox 6.1. The Linux kernel version on the VMs is 5.14 (but 3.10 on CentOS 7.8 should work fine if not using Cilium or Calico). They each have 4 GiB of RAM, 2 vCPUs, and around 20 Gigs of hard drive space. With Calico, 3 GiB or RAM on worker nodes should suffice while 4 GiB on the controller nodes is preferred. Please make sure you have at least 2 vCPUs per VM or none of the following would work. I installed CentOS minimal install. Each VM has two NICs — the first one is attached to a common VirtualBox host-only network, and the second one is the NAT network provided by VirtualBox. The first interface allows k8s’ kube API server on the master node to talk to the kubelet service on each node. The second interface allows each VM access to the web for downloading packages and container images. Flipping this order has certain consequences and needs extra handling — refer to the section Flannel minutiae below.

Configuring the VMs

On each node, disable SELinux with the following changes:

$ setenforce 0
$ sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/sysconfig/selinux 
$ sed -i 's/^SELINUX=\([^ ]*\)\(.*\)/SELINUX=disabled # \2/' /etc/sysconfig/selinux

Disable DHCP on the host-only network, or reconfigure your VirtualBox DHCP server to handle a smaller IP range, leaving out some IPs for use as static IPs. Assign static IPs to the corresponding NIC on each VM. You can do this using the nmtui tool if NetworkManager is enabled. If NetworkManager is disabled, you could manually edit the configuration script for your interface under /etc/sysconfig/network-script. In either case, make sure that the gateway is not set for this interface — because this interface does not allow connecting to the web.

Next, edit /etc/hosts on each VM to map the hostnames to the fixed IPs of these nodes. My /etc/hosts has the following lines added on all three nodes, one for each node. (Yes, I have a funky naming scheme for my VMs - short names starting with E):

192.168.219.2  ebro ebro
192.168.219.3  enzo enzo
192.168.219.4  elba elba 

You may need to remove any default gateways pointing to the internal network interface (such as the host-only VirtualBox network).

ip route del default via 192.168.219.1

Installing a container runtime

Run the following command to add the Docker Community Edition repo.

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

On Kubernetes versions prior to 1.24, Docker is supported out of the box. For these versions, to install Docker, run the following command to understand which versions of Docker are available:

yum list docker-ce — showduplicates | sort -r

Finally run the following command to install the latest available versions, or specify version tags of the package too in order to install a specific version of your choice.

$ yum install docker-ce docker-ce-cli containerd.io

NOTE: If you get errors while yum-config-manager, etc. then just run the following (you have less control on versions etc., but this should still be ok).

$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh ./get-docker.sh

[Older instructions below, no longer needed]

The latest and greatest Kubernetes (1.10) doesn’t work with the latest and greatest Docker (18.03). Moreover, k8s 1.10 isn’t all well-rounded yet it seems. So, I chose Kubernetes 1.9 (1.9.7–0) and for that I installed Docker 17.03. If you’re trying to install Docker Community Edition, you’ll end up at this site in all likelihood and the steps there would install Docker 18.03 instead (yes, even when you explicitly specify 17.03*). So instead, you may download the docker 17.03 RPMs manually. The following should work:

$ wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-17.03.1.ce-1.el7.centos.x86_64.rpm
$ wget https://yum.dockerproject.org/repo/main/centos/7/Packages/docker-engine-selinux-17.03.1.ce-1.el7.centos.noarch.rpm
$ rpm -U docker-engine-selinux-17.03.1.ce-1.el7.centos.noarch.rpm docker-engine-17.03.1.ce-1.el7.centos.x86_64.rpm

You might see some errors, like the following, coming from the installation of docker-engine-selinux; you can ignore them:

setsebool: SELinux is disabled.
Re-declaration of type docker_t
Failed to create node
Bad type declaration at /etc/selinux/targeted/tmp/modules/400/docker/cil:1
/usr/sbin/semodule: Failed!

[End of older instructions]

Do the following to configure Docker engine, in particular settings its cgroup driver to systemd instead of the default cgroupfs.

$ mkdir -p /etc/docker
$ cat <<EOF | tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

Don’t set the storage driver to overlay2 if your CentOS / RHEL kernel is older than 3.10.0–514. If it’s a different distribution, it is better to avoid it for pre-4.0 kernels.

Finally, enable docker and start the service:

$ systemctl enable docker
$ systemctl start docker 

On Kubernetes 1.24 and later, we can install the containerd container runtime. Like with Docker, figure the different versions available, choose a recent version and install it.

$ yum list containerd.io --showduplicates | sort -r
$ sudo yum install containerd-<version>

Take a backup of the default containerd configuration, and then overwrite it this way:

$ sudo containerd config default | sudo tee /etc/containerd/config.toml

Open the config.toml file, look for the key systemd_cgroup, and set it to false. And finally:

$ sudo systemctl enable --now containerd

Tweaking your VMs for Kubernetes

The first thing you need to do on all your nodes is turn off swap. Use the following commands:

$ swapoff -a
$ sed -i sed '/.*swap/s/^\(.*\)/# \1/'  /etc/fstab

Next, if dnsmasq is running locally on your system, disable it as kube-dns won’t come up while dnsmasq is running:

$ systemctl disable dnsmasq  # Or your kube-dns won't come up

On newer versions of CentOS (such as CentOS 9 Stream), we need to explicitly enable certain kernel modules.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

Next set a couple of kernel parameters:

$ modprobe br_netfilter
$ sysctl net.bridge.bridge-nf-call-ip6tables=1
$ sysctl net.bridge.bridge-nf-call-iptables=1

Also add the following line to /etc/sysctl.conf:

net.bridge.bridge-nf-call-iptables = 1

Then run:

sudo sysctl -p

Finally, disable firewalld:

$ systemctl disable firewalld && systemctl stop firewalld

Alternatively run the following commands on every node:

firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=4240/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10252/tcp
firewall-cmd --permanent --add-port=8285/udp
firewall-cmd --permanent --add-port=8472/udp
firewall-cmd --add-masquerade --permanent

On the node(s) that you want to run as k8s master, run the following additional commands:

firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10255/tcp

At a later point, if you want to expose your services as k8s NodePort services, you will need to open those ports too — can happen later. Finally, remember to restart the firewalld at the end of it.

sudo systemctl restart firewalld

Installing Kubernetes

Add the appropriate repos first:

$ cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
    https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg 
EOF

If you get signature verification errors, then set gpg_check=0 and skip the showduplicates command below.

Edit /etc/yum.conf, adding the following line at the end.

exclude = kubelet-1.18.* kubelet-1.17.* kubelet-1.16.*

Next, check the versions available:

$ yum list kubeadm --showduplicates | sort -r

Pick a kubeadm version 1.23.0 if you can see it, or the latest 1.23.x version that’s listed. Now install these and other related packages:

$ yum install kubeadm-1.23.17-0 kubelet-1.23.17-0 kubectl-1.23.17-0 kubernetes-cni

The above line should also install two dependencies — kubernetes-cni and socat (~ 1.7.3.2). Also enable the kubelet service (without starting it):

$ systemctl enable kubelet

Next, run the following commands to see that the cgroup driver for the docker installation matches that of kubeadm.

$ docker info | grep systemd
Cgroup Driver: systemd

Check the file /var/lib/kubelet/config.yaml, and change cgroupDriver to systemd in case it is cgroupfs. This should later be done on all nodes, not just the master.

[The following part about editing kubeadm service script is no longer relevant] Now check the contents of the file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and edit it in case it refers to a different cgroup driver. If you see the following line:

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

change it to:

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"  # same as the docker cgroup driver

Also, add an additional line on each node to set the node IP of the local node in the kubelet service’s command-line args.

Environment="KUBELET_EXTRA_ARGS=--node-ip=192.168.219.3"  # the local IP

[End of older instructions]

Then run:

$ systemctl daemon-reload

Followed by:

$ containerd config default | sudo tee /etc/containerd/config.toml
$ sed 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
$ sudo systemctl restart containerd
$ sudo systemctl enable kubelet

At this point, the kubelet service isn’t yet configured to start running. For that, we perform the next step. Assuming you would be running the Flannel CNI plugin, run the following command on the VM that you want to designate as the master, replacing the first IP with that of your master node:

$ kubeadm init --apiserver-advertise-address=192.168.219.2 --pod-network-cidr=10.244.0.0/16 # use the right IP

The --pod-network-cidr option with value 10.244.0.0/16 is needed if you want to use Flannel for your POD network.

In case you want to run Cilium instead, in overlay mode (the default), use the following command to configure kubernetes on the master:

$ kubeadm init --apiserver-advertise-address=192.168.219.2 --control-plane-endpoint=192.168.219.2:6443 --skip-phases=addon/kube-proxy # use the right IP

In case you want to deploy it with native networking instead, use the following alternative:

$ kubeadm init --apiserver-advertise-address=192.168.219.2 --control-plane-endpoint=192.168.219.2:6443 --pod-network-cidr <host-network-cidr> --skip-phases=addon/kube-proxy

And in case Calico is your CNI of choice, then the following is easier to start with.

$ kubeadm init --apiserver-advertise-address=192.168.219.2 --control-plane-endpoint=192.168.219.2:6443 --pod-network-cidr=172.16.0.0/16

Do note that I specified the pod network CIDR as 172.16.0.0/16. This is not the default — which is 192.168.0.0/16. However, because we are using a VM network which overlaps with that range (in this case 192.168.219.0/24), hence we choose a different non-overlapping CIDR range. This however requires some customization later while installing the Calico plugin. If your VM network does not overlap with 192.168.0.0/16, then you should specify that instead of 172.16.0.0/16 for the —pod-network-cidr option.

Refer to the official documentation for other types of POD networks. I have also tried using Weave, but the worker nodes never became ready.

Once kubeadm starts your master node, you should see a line such as the one below in its output. Copy this:

kubeadm join --token 689f6f.1865da38377c3e55 10.0.0.10:6443 --discovery-token-ca-cert-hash sha256:1250d71d8ed1b11de1b156bc23fcac9e80eaafa60395f75ba88ed550c64e42f4 

If you could not copy this for some reason, you need to be able to determine the token and CA cert hash. Here is how you determine the token:

kubeadm token list

And here is the command you run to determine the CA cert hash:

sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2>/dev/null | sha256sum | cut -d' ' -f1

In order to be able to run the kubectl command as a non-root user, do the following on the controller node(s):

mkdir -p ~/.kube/config
sudo cat /etc/kubernetes/admin.conf | tee ~/.kube/config

Run a kubectl command to verify that the nodes have registered:

kubectl get nodes

Flannel for network overlay

[Check the Flannel minutiae section below for more details if you want to use a network interface other than the first one, as was the case with me. Skip this and head over to the Cilium for Network Overlay section later if that’s what you chose.]

On the master node, install flannel for managing the pod network:

$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
## DON'T DO THIS kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
## DON'T DO THISkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml

While the above should be sufficient, depending on your network configuration you may have issues with hairpin NATting. As an alternative to the above, you may want to apply the following:

## DON'T RUN THIS kubectl apply -f https://gist.githubusercontent.com/phagunbaya/2a53519a9427ba0623244f1680a5b5ff/raw/13ada0d6dd92388c8c5aae93bfb1ccaf9c79f60b/flannel-0.9.1.yaml

On the remaining nodes, run from the command line the kubeadm join ... snippet from the kubeadm init output that you just copied. These nodes should now join your cluster. Also, copy over the file /etc/kubernetes/admin.conf from your master node to the same location on your worker nodes. This will allow you to run kubectl commands from your worker nodes. On the master (or a worker node), run the following command to see whether the other nodes have joined:

$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ kubectl get nodes
NAME      STATUS     ROLES     AGE       VERSION
ebro      Ready      master    8h        v1.21.0
elba      Ready      <none>    8h        v1.21.0
enzo      NotReady   <none>    8h        v1.21.0

Also check the status of the pods that Kubernetes creates:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY     STATUS    RESTARTS   AGE
kube-system   etcd-ebro                      1/1       Running   2          8h
kube-system   kube-apiserver-ebro            1/1       Running   2          8h
kube-system   kube-controller-manager-ebro   1/1       Running   2          8h
kube-system   kube-dns-6f4fd4bdf-bxnv2       3/3       Running   6          8h
kube-system   kube-flannel-ds-brfz8          1/1       Running   2          8h
kube-system   kube-flannel-ds-rdh2q          1/1       Running   2          8h
kube-system   kube-flannel-ds-xxxw5          1/1       Running   1          8h
kube-system   kube-proxy-glrdj               1/1       Running   2          8h
kube-system   kube-proxy-m6r2m               1/1       Running   1          8h
kube-system   kube-proxy-tbfnh               1/1       Running   2          8h
kube-system   kube-scheduler-ebro            1/1       Running   2          8h

It may take a minute or so for each node to join the cluster as the flannel images are downloaded on to each node. Once at least one of the worker nodes is ready, you’re ready to deploy your applications as pods.

Flannel minutiae

The VMs in my initial setup had the first NIC connected to the VirtualBox NAT network, and the second NIC connected to a host-only network. This resulted in the pod IPs being inaccessible across nodes — because flannel used the first NIC by default and in my case it was a NAT interface on which the machines could not cross-talk, only reach the web. Thus the pod network did not really work across nodes. An easy way to find out which NIC flannel uses is to look at flannel’s logs:

$ kubectl get pod --all-namespaces | grep flannel
kube-system   kube-flannel-ds-j587n          1/1       Running    2          3h
kube-system   kube-flannel-ds-p6sm7          1/1       Running   1          3h
kube-system   kube-flannel-ds-xn27c          1/1       Running   1          3h
$ kubectl logs  -f kube-flannel-ds-j587n -n kube-system
I0622 14:17:37.841808       1 main.go:487] Using interface with name enp0s3 and address 10.0.2.2
...

In the above case you can see that the flannel pod is listening on the NAT interface and this would not work. The way I fixed it was by:

1. Deleting the flannel daemon set:
   kubectl delete ds kube-flannel-ds -n kube-system
2. Removing the flannel.1 virtual NIC on each node.
   ip link delete flannel.1
3. Recreating the flannel daemonset with a tweak in the flannel configmap. Open the kube-flannel.yamlfile (downloaded from kube-flannel.yaml) and append the following tokens to the args list for thekube-flannel container: --iface and enp0s8 (or whatever is the name of the NIC connected to the host-only network).
4. Finally restarting kubelet and docker on all nodes:
   systemctl stop kubelet && systemctl restart docker && systemctl start kubelet

Cilium for Network Overlay

To install Cilium, perform the following steps only on the k8s master node(s).

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod +x get_helm.sh
./get_helm.sh

The above should install helm. Now configure helm to recognize the cilium helm repository.

helm repo add cilium https://helm.cilium.io/

Finally, run the following command to set Cilium up as the CNI plugin in the cluster — but make sure to first read a bit past the command to understand alternatives to consider.

API_SERVER_IP=192.168.219.2
API_SERVER_PORT=6443
helm install cilium cilium/cilium --version 1.13.4 --namespace kube-system --set kubeProxyReplacement=strict --set k8sServiceHost=${API_SERVER_IP} --set k8sServicePort=${API_SERVER_PORT}

Now by default Cilium would set up an overlay network a la vxlan, distinct from the native network on which the nodes communicate. But if you want to, you can use direct routing over the node network for the pods as well — this might get you better network performance. To try that, check the alternative command with additional configs. If you do not want the vxlan overlay and all the cluster nodes are on the same L2 network, try running the command with additional config options as shown below:

helm install cilium cilium/cilium --version 1.14.0 --namespace kube-system --set kubeProxyReplacement=strict --set k8sServiceHost=${API_SERVER_IP} --set k8sServicePort=${API_SERVER_PORT} --set tunnel="disabled" --set ipv4.enabled=true --set nativeRoutingCIDR=172.31.0.0/16 --set ipv4NativeRoutingCIDR=172.31.0.0/16 --set ipam.operator.clusterPoolIPv4PodCIDR="172.31.0.0/16" --set autoDirectNodeRoutes=true 

The option autoDirectNodeRoutes=true works only if all the cluster nodes are on the same L2 network and uses Linux kernel’s L2 routing capability to route to the specified CIDR without additional encapsulation. Otherwise, additional configuration is required, such as BGP. The option ipam.operator.clusterPoolIPv4PodCIDR does not work with newer Cilium versions and you have to use ipam.operator.clusterPoolIPv4PodCIDRList instead, but you can still specify a single value without the need for additional syntax.

To verify that all Cilium pods are up and running, run:

kubectl -n kube-system get pods -l k8s-app=cilium

Also deploy a test pod and check its IP address to verify that it is indeed in the CIDR you specified for the pod network.

There are more interesting combinations possible with Cilium, such as enabling IPsec for encrypting pod traffic. You can create the Kubernetes secret required for the purpose by following the commands described in this tutorial, but for actually installing Cilium after that, the following is a more robust method:

API_SERVER_IP=192.168.219.2
API_SERVER_PORT=6443
helm install cilium cilium/cilium --version=1.14.0 -n kube-system --set k8sServiceHost="${API_SERVER_IP}" --set k8sServicePort="${API_SERVER_PORT}" --set encryption.type="ipsec" --set encryption.enabled=true --set encryption.ipsec.inter
face="eth0" --set encryption.ipsec.keyFile="key" --set encryption.ipsec.mountPath="/etc/ipsec/keys" --set encryption.ip
sec.secretName="cilium-ipsec-keys"

Last, but not least, it is important sometimes to be able to install the latest Cilium off the main branch and try some recent fixes. To do that, we have to rely on the CI builds for the specific commits we are looking for. We no longer specify a version, and use the cloned Cilium repo for the helm source. The image tag <img-tag> below must be replaced with the actual image tag from the CI pipeline of the particular commit — something you can determine following the method here.

mkdir -p github.com/cilium && cd github.com/cilium
git clone github.com/cilium/cilium
cd -
helm install cilium ./github.com/cilium/cilium -n kube-system \
  --set image.repository=quay.io/cilium/cilium-ci --set image.tag=<img-tag> \
  --set image.useDigest=false --set operator.image.repository=quay.io/cilium/operator \
  --set operator.image.suffix=-ci --set operator.image.tag=<img-tag> \
  --set operator.image.useDigest=false ... remaining args ...

Calico for Network Overlay

While Cilium has been grabbing headlines as the eBPF-enabled CNI, Calico by Tigera too is a formidable CNI plugin with cool capabilities using eBPF, and production-ready, performant features at scale. Enabling Calico isn’t too difficult.

On all controller as well as worker nodes, run the following command:

$ firewall-cmd --permanent --add-port=179/tcp

Then deploy the Calico operator:

$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/tigera-operator.yaml

Finally, create the Calico custom resources:

$ curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/custom-resources.yaml -O 

In case you had to choose a CIDR different from the default 192.168.0.0/16, because the VM network is already on this or an overlapping CIDR, edit the CRD file to have the following attributes defined:

spec:
  calicoNetwork:
    ipPools:
    - cidr: 172.16.0.0/16

Then create the resources:

kubectl create -f custom-resources.yaml

This should get all the nodes ready, and the pods up.

Cleaning up, resetting, and restoring

Setting up Kubernetes by hand isn’t exactly straightforward and you could miss a step here and there that could result in a non-functional cluster. To recreate your cluster without having to go the whole hog, you could try out a few things.

The following should work for resetting Kubernetes on a cluster before starting out afresh. Do them for each node, taking care to drain the master node right at the end:

$ kubectl drain <node>  --delete-local-data --force --ignore-daemonsets
$ kubectl delete node <node>
$ kubeadm reset

If you have to clean up manually because you forgot to run above on the worker nodes, try the following:

$ rm -rf /etc/kubernetes/*
$ pkill kubelet

After this you could rerun the kubeadm init … command on the master node, and then follow the rest of the steps from there, reinstalling flannel on the master, and running the corresponding kubeadm join … commands on the worker nodes. These steps let you iterate through fault installations easily.

There is one specific case that can do with slightly less drastic changes. If your API server certificates expired, you can generate new certificates on the master using the following steps.

$ sudo mkdir /etc/kubernetes/pki.backup
$ sudo mv /etc/kubernetes/pki/apiserver* /etc/kubernetes/pki.backup/
$ sudo mv /etc/kubernetes/pki/front-proxy-client* /etc/kubernetes/pki.backup/
$ sudo kubeadm alpha phase certs apiserver --apiserver-advertise-address <master-IP>
$ sudo kubeadm alpha phase certs apiserver-kubelet-client
$ sudo kubeadm alpha phase certs front-proxy-client
$ sudo mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
$ sudo mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
$ sudo mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old
$ sudo kubeadm alpha phase kubeconfig all --apiserver-advertise-address <master-ip> --node-name <master-node-name>
$ reboot

On more recent k8s versions, the following seems to work:

$ # backup certs
$ sudo kubeadm alpha certs renew all
$ # backup confs
$ sudo kubeadm init phase kubeconfig all --apiserver-advertise-address <master-ip> --node-name <master-node-name>

Following this, check if there are valid tokens available and generate if none is.

$ sudo kubeadm token list
$ sudo kubeadm token create

On each slave node, run the following. You can get the discovery token as described earlier in this article.

$ sudo kubeadm reset
$ sudo kubeadm join --token <...> --discovery-token-ca-cert-hash sha256:...

This should get all your nodes up and running.

Configuring a load balancer

You’d want to enable a load balancer on your k8s cluster. It is really simple to do this.

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml
$ cat <<EOF >metallb-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.56.65-192.168.56.96
EOF
$ kubectl apply -f metallb-config.yaml

If you need more details and caveats, or somehow the above doesn’t work, head to this fantastic video to understand more details. The port 7496 must be opened between the nodes — I didn’t have to do this but not sure why.

Verify your installation

Again, quite easy really:

$ kubectl create deploy nginx — image nginx
$ kubectl expose deploy nginx — port 80 — type LoadBalancer
$ kubectl get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1      <none>          443/TCP        70m
nginx        LoadBalancer   10.97.33.236   192.168.56.65   80:31752/TCP   8s

Now point your browser to the IP address listed under the external IP column. If you expose the service on a non-standard load balancer port, or on one of the node ports, then run the firewall-cmd to open it:

firewall-cmd --permanent --add-port=32106/tcp