Credit where credit is due, most of the steps here are taken from the Kubernetes documentation pages:<\/p>\n\n\n\n
- Installing kubeadm | Kubernetes<\/a><\/li>
- Creating Highly Available clusters with kubeadm | Kubernetes<\/a><\/li><\/ul>\n\n\n\n
With that, let’s get going:<\/p>\n\n\n\n
Setting up the required Azure infrastructure<\/h2>\n\n\n\n
We need a bit of Azure infrastructure to build a highly available Kubernetes cluster on Azure. What we’ll create:<\/p>\n\n\n\n
- VNET<\/li>
- Network Security Group<\/li>
- 2 master VMs<\/li>
- 2 worker VMs<\/li>
- A load balancer for the master VMs<\/li><\/ul>\n\n\n\n
Let’s create all of this using the Azure CLI:<\/p>\n\n\n\n
First up, all the networking elements (VNET and NSG)<\/p>\n\n\n\n
az group create -n kubeadm -l westus2\n\naz network vnet create \\\n --resource-group kubeadm \\\n --name kubeadm \\\n --address-prefix 192.168.0.0\/16 \\\n --subnet-name kube \\\n --subnet-prefix 192.168.0.0\/16\n\naz network nsg create \\\n --resource-group kubeadm \\\n --name kubeadm\n\naz network nsg rule create \\\n --resource-group kubeadm \\\n --nsg-name kubeadm \\\n --name kubeadmssh \\\n --protocol tcp \\\n --priority 1000 \\\n --destination-port-range 22 \\\n --access allow\n\naz network nsg rule create \\\n --resource-group kubeadm \\\n --nsg-name kubeadm \\\n --name kubeadmWeb \\\n --protocol tcp \\\n --priority 1001 \\\n --destination-port-range 6443 \\\n --access allow\n\naz network vnet subnet update \\\n -g kubeadm \\\n -n kube \\\n --vnet-name kubeadm \\\n --network-security-group kubeadm<\/code><\/pre>\n\n\n\nThen we can add the 4 virtual machines:<\/p>\n\n\n\n
az vm create -n kube-master-1 -g kubeadm \\\n--image UbuntuLTS \\\n--vnet-name kubeadm --subnet kube \\\n--admin-username nilfranadmin \\\n--ssh-key-value @~\/.ssh\/id_rsa.pub \\\n--size Standard_D2ds_v4 \\\n--nsg kubeadm \\\n--public-ip-sku Standard --no-wait\n\naz vm create -n kube-master-2 -g kubeadm \\\n--image UbuntuLTS \\\n--vnet-name kubeadm --subnet kube \\\n--admin-username nilfranadmin \\\n--ssh-key-value @~\/.ssh\/id_rsa.pub \\\n--size Standard_D2ds_v4 \\\n--nsg kubeadm \\\n--public-ip-sku Standard --no-wait\n\naz vm create -n kube-worker-1 -g kubeadm \\\n--image UbuntuLTS \\\n--vnet-name kubeadm --subnet kube \\\n--admin-username nilfranadmin \\\n--ssh-key-value @~\/.ssh\/id_rsa.pub \\\n--size Standard_D2ds_v4 \\\n--nsg kubeadm \\\n--public-ip-sku Standard --no-wait\n\naz vm create -n kube-worker-2 -g kubeadm \\\n--image UbuntuLTS \\\n--vnet-name kubeadm --subnet kube \\\n--admin-username nilfranadmin \\\n--ssh-key-value @~\/.ssh\/id_rsa.pub \\\n--size Standard_D2ds_v4 \\\n--nsg kubeadm \\\n--public-ip-sku Standard<\/code><\/pre>\n\n\n\nAnd then finally, we can create the load balancer:<\/p>\n\n\n\n
\naz network public-ip create \\\n --resource-group kubeadm \\\n --name controlplaneip \\\n --sku Standard \\\n --dns-name nilfrankubeadm\n\n az network lb create \\\n --resource-group kubeadm \\\n --name kubemaster \\\n --sku Standard \\\n --public-ip-address controlplaneip \\\n --frontend-ip-name controlplaneip \\\n --backend-pool-name masternodes \n\naz network lb probe create \\\n --resource-group kubeadm \\\n --lb-name kubemaster \\\n --name kubemasterweb \\\n --protocol tcp \\\n --port 6443 \n\naz network lb rule create \\\n --resource-group kubeadm \\\n --lb-name kubemaster \\\n --name kubemaster \\\n --protocol tcp \\\n --frontend-port 6443 \\\n --backend-port 6443 \\\n --frontend-ip-name controlplaneip \\\n --backend-pool-name masternodes \\\n --probe-name kubemasterweb \\\n --disable-outbound-snat true \\\n --idle-timeout 15 \\\n --enable-tcp-reset true\n\naz network nic ip-config address-pool add \\\n --address-pool masternodes \\\n --ip-config-name ipconfigkube-master-1 \\\n --nic-name kube-master-1VMNic \\\n --resource-group kubeadm \\\n --lb-name kubemaster\n\naz network nic ip-config address-pool add \\\n --address-pool masternodes \\\n --ip-config-name ipconfigkube-master-2 \\\n --nic-name kube-master-2VMNic \\\n --resource-group kubeadm \\\n --lb-name kubemaster<\/code><\/pre>\n\n\n\nAnd then, let’s get the public IP of all the nodes so we can connect to them and set up what’s needed:<\/p>\n\n\n\n
MASTER1IP=`az vm list-ip-addresses -g kubeadm -n kube-master-1 \\\n--query \"[].virtualMachine.network.publicIpAddresses[0].ipAddress\" --output tsv`\nMASTER2IP=`az vm list-ip-addresses -g kubeadm -n kube-master-2 \\\n--query \"[].virtualMachine.network.publicIpAddresses[0].ipAddress\" --output tsv`\nWORKER1IP=`az vm list-ip-addresses -g kubeadm -n kube-worker-1 \\\n--query \"[].virtualMachine.network.publicIpAddresses[0].ipAddress\" --output tsv`\nWORKER2IP=`az vm list-ip-addresses -g kubeadm -n kube-worker-2 \\\n--query \"[].virtualMachine.network.publicIpAddresses[0].ipAddress\" --output tsv`<\/code><\/pre>\n\n\n\nAnd now, we can setup the first master node:<\/p>\n\n\n\n
Setup first master node<\/h2>\n\n\n\n
Now we can start setting up the first master node. Much of the configuration for the first master node will be repeated on the other nodes; as most of this is laying the plumbing for Kubernetes to work. <\/p>\n\n\n\n
We need to do a couple things:<\/p>\n\n\n\n
- Install the right tools, like kubeadm, kubectl and kubelet.<\/li>
- Configure networking, container runtime.<\/li>
- Use kubeadm to setup node.<\/li>
- Setup a CNI.<\/li><\/ul>\n\n\n\n
Let’s start with the first step:<\/p>\n\n\n\n
ssh nilfranadmin@$MASTER1IP\n\nsudo apt update\nsudo apt -y install curl apt-transport-https;\n\ncurl -s https:\/\/packages.cloud.google.com\/apt\/doc\/apt-key.gpg | sudo apt-key add -\necho \"deb https:\/\/apt.kubernetes.io\/ kubernetes-xenial main\" | sudo tee \/etc\/apt\/sources.list.d\/kubernetes.list\n\nsudo apt update\nsudo apt -y install vim git curl wget kubelet kubeadm kubectl containerd;\n\nsudo apt-mark hold kubelet kubeadm kubectl\n\nkubectl version --client && kubeadm version<\/code><\/pre>\n\n\n\nThat final command should confirm kubectl and kubeadm are installed and return something similar to the following screenshot:<\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-1024x166.png\")
Output showing kubectl and kubeadm successfully installed<\/figcaption><\/figure>\n\n\n\n Next up, let’s configure networking and containerd on the node:<\/p>\n\n\n\n
cat <<EOF | sudo tee \/etc\/modules-load.d\/containerd.conf\noverlay\nbr_netfilter\nEOF\n\nsudo modprobe overlay\nsudo modprobe br_netfilter\n\n# Setup required sysctl params, these persist across reboots.\ncat <<EOF | sudo tee \/etc\/sysctl.d\/99-kubernetes-cri.conf\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nEOF\n\n# Apply sysctl params without reboot\nsudo sysctl --system\n\nsudo mkdir -p \/etc\/containerd\ncontainerd config default | sudo tee \/etc\/containerd\/config.toml\n\nsudo systemctl restart containerd\n\ncat <<EOF | sudo tee \/etc\/modules-load.d\/k8s.conf\nbr_netfilter\nEOF\n\ncat <<EOF | sudo tee \/etc\/sysctl.d\/k8s.conf\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1\nEOF\nsudo sysctl --system<\/code><\/pre>\n\n\n\nAnd with the node prepared, we can setup Kubernetes using kubeadm. Although all the steps to this point seemed like a ton of work; setting up Kubernetes using kubeadm is a single command:<\/p>\n\n\n\n
sudo kubeadm init --control-plane-endpoint \"nilfrankubeadm.westus2.cloudapp.azure.com:6443\" --upload-certs<\/code><\/pre>\n\n\n\nThe output of this command will show you three things:<\/p>\n\n\n\n
- Where to get the kubeconfig to configure kubectl to work with this cluster.<\/li>
- How to join additional control plane nodes<\/li>
- How to join worker nodes<\/li><\/ol>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-1-1024x556.png\")
Instructions showing you how to interact with the cluster.<\/figcaption><\/figure>\n\n\n\n Make sure to copy the two kubeadm join commands, as we’ll need these to bootstrap the other nodes.<\/p>\n\n\n\n
Let’s do the first command shown in the screenshot above to make the kubeconfig file available:<\/p>\n\n\n\n
mkdir -p $HOME\/.kube\nsudo cp -i \/etc\/kubernetes\/admin.conf $HOME\/.kube\/config\nsudo chown $(id -u):$(id -g) $HOME\/.kube\/config<\/code><\/pre>\n\n\n\nAnd then we can interact with the cluster, and e.g. execute a get nodes:<\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-2.png\")
After copying the kubeconfig file, we can use kubectl to see the node<\/figcaption><\/figure>\n\n\n\n The node shows not ready, because a CNI wasn’t setup yet. We can do this using the following command:<\/p>\n\n\n\n
kubectl apply -f \"https:\/\/cloud.weave.works\/k8s\/net?k8s-version=$(kubectl version | base64 | tr -d '\\n')\"<\/code><\/pre>\n\n\n\nAnd if you wait a couple of seconds, you will see the node transition into a ready state:<\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-3.png\")
After setting up the CNI, we can see the node get into a ready state.<\/figcaption><\/figure>\n\n\n\n Now that that’s done, we can exit out of this node since we need to configure the other nodes now.<\/p>\n\n\n\n
Setting up the second control node<\/h2>\n\n\n\n
Now we have to prepare the second control node. Here, we’ll do a couple things:<\/p>\n\n\n\n
- Creating Highly Available clusters with kubeadm | Kubernetes<\/a><\/li><\/ul>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-4-1024x113.png\")
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-5.png\")
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-6.png\")
![\"\"](\"\/wp-content\/uploads\/2021\/10\/image-7-1024x780.png\")