CKAD series part 5: Observability

nillsf CKAD series

This is part 5 in a multi-series on my CKAD study efforts. You can find previous entries here:

This is part 5 in this CKAD series, covering the 4th exam topic out of 7. If you made it this far, we have covered half of the topics for the exam, 3 more to go!

This section will cover observability. We’ll dive into the following 4 topics:

  • Understand LivenessProbes and ReadinessProbes
  • Understand container logging
  • Understand how to monitor applications in Kubernetes
  • Understand debugging in Kubernetes

Let’s go ahead and start with the first topic:

Understand LivenessProbes and ReadinessProbes

Kubernetes uses LivenessProbes and ReadinessProbes to monitor the availability of your applications. Each probe serves a different purpose:

  • A LivenessProbe monitors the availability of an application while it is running. If a LivenessProbe fails, Kubernetes will restart your pod. This could be useful to catch deadlocks, infinite loops, or just a ‘stuck’ application.
  • A ReadinessProbe monitors when you application becomes available. As long as a ReadinessProbe fails, Kubernetes will not send any traffic to unready pods. This is useful if your application has to go through some configuration before it becomes available.

LivenessProbes and ReadinessProbes don’t need to be served from the same endpoint in your application. If you have a smart application, that application could take itself out of rotation (meaning, no more traffic is sent to the application), while still being healthy. To achieve this, you would have the ReadinessProbe fail, but have the LivenessProbe remain active.

Why not try out this last scenario is a quick test? Let’s try and do the following:

  • Create 2 nginx pods, each with a distinct page;
  • Create a service that load balances between these 2 pods;
  • Have a seperate LivenessProbe and ReadinessProbe;
  • Start of with everything healthy;
  • Have the ReadinessProbe fail on 1, but not the other;
  • Have the ReadinessProbe fail on both;
  • Recover a ReadinessProbe;
  • Have the LivenessProbe fail on 1.

In part 4 during the ambassador pattern, we created two pods to which we used an ambassador to do load balancing. We’ll re-use those pods to have 2 distinct web pages, and then create a service on top of them. Just for the fun of things, let’s also create a new namespace for this part; and set the kubectl default namespace. 

kubectl create namespace observability
kubectl config set-context $(kubectl config current-context) --namespace=observability

Let’s now recreate our HTML pages as a configmap. 

<!DOCTYPE html>
<html>
<head>
    <title>Server 1</title>
</head>
<body>
Server 1
</body>
</html>
<!DOCTYPE html>
<html>
<head>
    <title>Server 2</title>
</head>
<body>
Server 2
</body>
</html>
kubectl create configmap server1 --from-file=index1.html
kubectl create configmap server2 --from-file=index2.html

Let’s also add a health pages. Nothing fancy here, just a health page.

<!DOCTYPE html>
<html>
<head>
    <title>All is fine here</title>
</head>
<body>
OK
</body>
</html>
kubectl create configmap healthy --from-file=healthy.html

Let’s now go ahead and create our two pods, and add a LivenessProbe and ReadinessProbe to them. (bare with me, this is a lot of yaml)

apiVersion: v1
kind: Pod
metadata:
  name: server1
  labels:
    app: web-server #we'll give both servers the same label, so the service will load balance
spec:
  containers:
    - name: nginx-1
      image: nginx
      ports:
        - containerPort: 80
      livenessProbe:
        httpGet:
          path: /healthy.html
          port: 80
        initialDelaySeconds: 3
        periodSeconds: 3
      readinessProbe:
        httpGet:
          path: /index.html
          port: 80
        initialDelaySeconds: 3
        periodSeconds: 3
      volumeMounts:
        - name: html
          mountPath: /usr/share/nginx/html
  initContainers:
    - name: prep
      image: busybox
      volumeMounts:
        - name: index
          mountPath: /tmp/index.html
          subPath: index.html
        - name: healthy
          mountPath: /tmp/healthy.html
          subPath: healthy.html
        - name: html
          mountPath: /usr/share/nginx/html/
      command: ["/bin/sh", "-c"]
      args: ["cp /tmp/index.html /usr/share/nginx/html/index.html; cp /tmp/healthy.html /usr/share/nginx/html/healthy.html;"]
  volumes:
    - name: index
      configMap:
        name: server1
    - name: healthy
      configMap:
        name: healthy
    - name: html
      emptyDir: {}
---
apiVersion: v1
kind: Pod
metadata:
  name: server2
  labels:
    app: web-server #we'll give both servers the same label, so the service will load balance
spec:
  containers:
    - name: nginx-1
      image: nginx
      ports:
        - containerPort: 80
      livenessProbe:
        httpGet:
          path: /healthy.html
          port: 80
        initialDelaySeconds: 3
        periodSeconds: 3
      readinessProbe:
        httpGet:
          path: /index.html
          port: 80
        initialDelaySeconds: 3
        periodSeconds: 3
      volumeMounts:
        - name: html
          mountPath: /usr/share/nginx/html
  initContainers:
    - name: prep
      image: busybox
      volumeMounts:
        - name: index
          mountPath: /tmp/index.html
          subPath: index.html
        - name: healthy
          mountPath: /tmp/healthy.html
          subPath: healthy.html
        - name: html
          mountPath: /usr/share/nginx/html/
      command: ["/bin/sh", "-c"]
      args: ["cp /tmp/index.html /usr/share/nginx/html/index.html; cp /tmp/healthy.html /usr/share/nginx/html/healthy.html;"]
  volumes:
    - name: index
      configMap:
        name: server2
    - name: healthy
      configMap:
        name: healthy
    - name: html
      emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web-server
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: LoadBalancer

You’ll see in the yaml above, that we have a seperate LivenessProbe and a seperate ReadinessProbe. We can then go to our service and get returns from server 1 and 2. The load balancing isn’t always fully round-robin, so let’s do a couple of curls and check the occurrences of server 1 and server 2:

kubectl get service # get the service ip here
for i in {1..50}; do curl --silent 52.191.138.86 | sed -n '7p' >> output.txt; done
echo 'server 1';grep '1' output.txt | wc;echo 'server 2';grep '2' output.txt | wc

The above commands will do 50 curls to our IP address, and the echo after that will check how often a certain server appears.

Let’s now go in, and make the ReadinessProbe of server 1 fail.

#first, we'll go into the container and move the index file away
kubectl exec -it server1 sh
cd /usr/share/nginx/html
mv index.html index.html.fail
#this will make our readiness probe fail, and make our container unready:
exit #to go out of the container shell
kubectl get pods --watch

As we see our pod go in a unready state, it won’t get any more traffic, but it remains running. Let’s run our curls again:

rm output.txt
for i in {1..50}; do curl --silent 52.191.138.86 | sed -n '7p' >> output.txt; done
echo 'server 1';grep '1' output.txt | wc;echo 'server 2';grep '2' output.txt | wc

If things go well, you should now see all traffic go to server 2, and no traffic to server 1.

Let’s now also fail server 2, and see what gives:

#first, we'll go into the container and move the index file away
kubectl exec -it server2 sh
cd /usr/share/nginx/html
mv index.html index.html.fail
#this will make our readiness probe fail, and make our container unready:
exit #to go out of the container shell
kubectl get pods --watch

We don’t need to do 50 curls now, we just need 1 which will eventually timeout:

curl 52.191.138.86 --connect-timeout 2

And this will timeout after those 2 seconds.

We can then go into one of our pods, and also have the LivenessProbe fail. This will initiate a restart of the container, so if you watch this via kubectl get pods --watch; you’ll see the restart happen. 

kubectl exec -it server2 sh
mv /usr/share/nginx/html/healthy.html /usr/share/nginx/html/healthy.html.fail

Now, this is where it gets fun! Do a kubectl get pods --watch and see what happens. Your pod gets into a continuous loop of running – crashloopbackoff. No way to repair the damage easily. (this where you typically throw away the pod, and redeploy your app).

If you followed along in part 2, we did something similar in removing the page a LivenessProbe refers to. There, the page would automatically come back from the container image. In this case however, our health page is stored on the volume we mount into the container. And the LivenessProbe will fail after 3 seconds (as per our pod definition), meaning we have a 3 second window to apply our changes. Which is too short, believe me, I tried. AAAAND: The LivenessProbe is implemented at the container level, not the pod level (there’s a github issue talking about this), so the initcontainer won’t reapply our files.

This led me to want to go into the node and move the files around in the actual emptydir (the dir exists somewhere you know…). What follows is a bit of mental masturbation – as I want to get in the node – which is actually harder than it looked for a VMSS based AKS-cluster. Long story short, don’t do. Just delete the pod and carry on with your live. (I did a lot of back and forth with the load balancer, NAT pools, and eventually gave up and just ran a SSH pod in my cluster, which I describe in what follows).

Step 1 is to enable SSH into your cluster nodes. This can be done with the following script using AZ CLI:

rgname="akschallenge"
aksname="aksworkshop"
CLUSTER_RESOURCE_GROUP=$(az aks show --resource-group $rgname --name $aksname --query nodeResourceGroup -o tsv)
SCALE_SET_NAME=$(az vmss list --resource-group $CLUSTER_RESOURCE_GROUP --query [0].name -o tsv)
az vmss extension set  \
    --resource-group $CLUSTER_RESOURCE_GROUP \
    --vmss-name $SCALE_SET_NAME \
    --name VMAccessForLinux \
    --publisher Microsoft.OSTCExtensions \
    --version 1.4 \
    --protected-settings "{\"username\":\"azureuser\", \"ssh_key\":\"$(cat ~/.ssh/id_rsa.pub)\"}"

az vmss update-instances --instance-ids '*' \
    --resource-group $CLUSTER_RESOURCE_GROUP \
    --name $SCALE_SET_NAME

With SSH enabled on the nodes, we can then try to SSH into them. First get the following info: your private key and the IP of the node hosting your failed pod. Afterwards we’ll need the pod ID of the failed pod.

kubectl get pod server2 -o wide #remember the node
kubectl get pod server2 -o yaml | grep id #copy this ID
kubectl get nodes -o wide #copy the IP address of the node
cat ~/.ssh/id_rsa #copy your private key
kubectl run -it --rm aks-ssh --image=debian
#wait a couple seconds for the pod to come live
apt-get update && apt-get install openssh-client -y
mkdir ~/.ssh
echo "PASTE IN YOUR PRIVATE KEY" > ~/.ssh/id_rsa
chmod 0600 ~/.ssh/id_rsa 
ssh azureuser@THE_IP_OF_YOUR_NODE 
sudo su #to become root

So, now we’re finally in our node. We can then access that emptydir we created.

Navigate to this folder, and you can move around the file to get the Probes to work again:

cd /var/lib/kubelet/pods/cdfdea76-b3b6-11e9-ad8a-46d847be6880/volumes/kubernetes.io~empty-dir/html/ #replace the ID with your ID
mv index.html.fail index.html #this will repair our LivenessProbe page
cp index.html healthy.html #this will repair our ReadinessProbe

If you then do a kubectl get pods --watch – you’ll notice (after a while) that your pod will stop being rebooted. #SUCCESS

Understand container logging

There’s a good kubernetes documentation article that describes logging in kubernetes.

For our certification purposes, I’m going to keep this one short; we’ll just access logs of a container and of a multi-container pod. But please, if you’re going to bring this to production, think about your logging infrastructure. Kubernetes doesn’t maintain logs if a container crashes, a pod is evicted or a node dies – so you’ll want to have an external solution for your logs. Think either Elasticsearch and Kibana (the open source default), StackDriver (the GKE default) or Azure Monitor (guess what, the Azure default. Which is actually quiet nice, since it combines logging and metrics in one.)

Ok, let’s start with a single container pod – and try to get it’s logs.

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

We can create this one using kubectl create -f counter.yaml – and we can access the logs of this container using kubectl logs counter. You can also attach your terminal to follow these logs using kubectl logs counter --follow.

Pro-tip for the exam, you might be asked (I expect this) to export logs to a file and copy this file to a certain location. Learn about output redirection if you don’t already. kubectl logs counter > logfile.log will do the trick for you.

Let’s now try the same with a multi-container pod.

apiVersion: v1
kind: Pod
metadata:
  name: counter-web
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
  - name: web-server
    image: nginx

Let’s create this using kubectl create -f counter-web.yaml . To get web-server logs, we’ll need to make at least one request to our web server and then we can get to our logs:

kubectl exec -it counter-web --container count wget localhost
kubectl logs counter-web web-server # to get web logs
kubectl logs counter-web count # to get count logs

Understand how to monitor applications in Kubernetes

Monitoring comes with the same remark as the previous topic around logging. For a production cluster, you’ll want to think this one through. You’ll want to get good information about both the health and utilization of your cluster and your pods. For the purpose of our certification focus here, we’ll just focus on 1 command here: kubectl top

To get metric information about your nodes, you can run kubectl top node. This will show you CPU/Memory utilization across your cluster.

To get metric information about your pods, you can run kubectl top pods to show info on your pods. If you want to go a little more granular and also show to containers in your pods, you can execute kubectl top pods --containers.

Understand debugging in Kubernetes

There’s a large section of the Kubernetes docs dedicated to debugging. The docs discuss debugging Init Containers, debugging Pods and ReplicationControllers, debugging Services, debugging a StatefulSet and debugging a node.

The best way to deal with debugging is running a lot of examples, and hitting errors while you’re doing so (I certainly have run into a couple of issues along the way that have taken me down deep rabbit holes).

Before we jump into a couple of examples, let’s look at the most popular debugging commands:

  • kubectl get pod: This will show you basic info about your pod. You can get more output via the -o wide flag, and even more output with -o yaml or -o json. With --watch you can keep an eye on potential changes.
  • kubectl describe pod: This command describes the pod for you (in some detail) and also shows you the latest events related to your pod.
  • kubectl logs *podname* *containername*: We discussed this before, but this way you can get the logs from your pods and the containers in your pods
  • kubectl get service: This will give you service information. You can get more output via the -o wide flag, and even more output with -o yaml or -o json. With --watch you can keep an eye on potential changes.
  • kubectl describe endpoints: this will give you information about the endpoints used by services. This will also show you healthy and unhealthy endpoints.

When you are debugging issues with resources apperently not existing, don’t forget about namespaces. You can append the --all-namespaces flag to most commands in kubectl, this can help. In service debugging, please remember that the nameresolution for services works with the servicename within the same namespace, but needs the FQDN for cross-namespace communication (service.namespace.svc.cluster.local).

Let’s have a look at two examples, and where things can go wrong. We’ll start with a very simple busybox container that does a counter, with one little mistake in there.

apiVersion: v1
kind: Pod
metadata:
  name: counter-wrong
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while treu; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

(challenge, can you spot the issue with the naked eye?)

Let’s create this and see what happens:

kubectl create -f counter-wrong.yaml
kubectl get pods --watch

In watching our pods, we’ll see the container complete a couple times and then enter the CrashLoopBackOff state.

Let’s have a look at the describe for this pod kubectl describe pod counter-wrong. If you look at the events here you just see the container starting and then restarting when it fails. This doesn’t help much does it?

Maybe the logs will show us something: kubectl logs counter-wrong

/bin/sh: treu: not found

Now, this tells me something. We misspelled true as treu. That’s why our while loop was failing.

Let’s try another example, now running an alpine container:

apiVersion: v1
kind: Pod
metadata:
  name: alpine-error
spec:
  containers:
  - name: alpine
    image: apline

Let’s create this and look at the pods:

kubectl create -f alpine.yaml
kubectl get pods --watch

This will show us a new state for pods we haven’t seen before:

alpine-error    0/1     ErrImagePull       0          31s
alpine-error    0/1     ImagePullBackOff   0          43s

Let’s have a look at a describe to see if this shows us something more:

Events:
  Type     Reason     Age                From                                        Message
  ----     ------     ----               ----                                        -------
  Normal   Scheduled  79s                default-scheduler                           Successfully assigned observability/alpine-error to aks-nodepool1-14406582-vmss000001
  Normal   Pulling    37s (x3 over 78s)  kubelet, aks-nodepool1-14406582-vmss000001  Pulling image "apline"
  Warning  Failed     36s (x3 over 77s)  kubelet, aks-nodepool1-14406582-vmss000001  Failed to pull image "apline": rpc error: code = Unknown desc = Error response from daemon: pull access denied for apline, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed     36s (x3 over 77s)  kubelet, aks-nodepool1-14406582-vmss000001  Error: ErrImagePull
  Normal   BackOff    11s (x4 over 77s)  kubelet, aks-nodepool1-14406582-vmss000001  Back-off pulling image "apline"
  Warning  Failed     11s (x4 over 77s)  kubelet, aks-nodepool1-14406582-vmss000001  Error: ImagePullBackOff

Do you see what we did there? We misspelled apline, and we couldn’t pull our container.

Summary

In this part of the CKAD series we look at observability. We played around with LivenessProbes and ReadinessProbes, briefly touched on logging and monitoring and looked into debugging Kubernetes. For the debugging part I recommend you play around a lot, and learn to work with Kubernetes. The best way to learn is by doing. Nothing beats experience.

That being said, we’re over half way our study progress. Are you ready for part 6?

Leave a Reply