Don’t use environment variables in Kubernetes to consume secrets

Managing secrets is a complicated endeavor. Kubernetes has a native secrets implementation, that allows you to store and access secrets from your deployments.

A while ago, I read a short free book on Kubernetes Security, by Liz Rice and Michael Hausenblas (apparently O’Reilly calls it a report, but I actually have a hard copy that I call a book). One of the most memorable passages in there was around secret management. I wanted to try out their recommendations for myself.

So, in this post I’m planning to explain why passing secrets via environment variables is a bad idea. I’ll explain two reasons why you shouldn’t use environment variables for secrets:

• They are passed plain text to Docker
• They leak to monitoring tools

In this post I do not aim to tackle the limited security of the secrets implementation itself (meaning, base64 encoded values in etcd), but rather how you consume these secrets in your application.

Setting up a demo application

To demonstrate the findings, we’ll create a quick and dirty demo application, running on top of an AKS cluster in Azure. We’ll create a new secret, and then consume that secret as an environment variable in our pod.

Let’s first create the secret:

kubectl create secret generic supersecret --from-literal=secret=supersecret-value

We will use a pod running nginx to consume the secret:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-env
spec:
  containers:
  - name: nginx-env
    image: nginx
    env:
    - name: SUPERSECRET
      valueFrom:
        secretKeyRef:
          name: supersecret
          key: secret

To create this pod, you can use the following command:

kubectl create -f nginx-env.yaml

To consume the secret, we can exec into the pod and echo the value:

kubectl exec -it nginx-env sh
echo $SUPERSECRET

This, as is to be expected, will show the value of the secret:

Issue 1: Docker run-time outputs environment variables in plain text

The first issue with environment variables is that the docker run-time stores their values in plain text, and they can be retrieved by anybody with access to Docker on the cluster. Let me show you:

To start, we’ll figure out which host is running our pod :

kubectl get pods nginx-env -o wide

Then, we’ll need to get the docker ID of the running container. We can get that this way:

kubectl get pods nginx-env -o yaml | grep containerID

We will then do a docker inspect command to get a full description of our running container. Instead of setting up SSH access to the nodes, I’ll show you how you can use invoke command using the Azure CLI to execute commands on the nodes. You’ll need the resource group name and the name of the VMSS hosting your cluster for this.

INSTANCE=1
VMSS="aks-agentpool-39838025-vmss"
RGNAME="MC_handsonaks_handsonaks_westus2"
CONTAINERID="9a0ba590d702daa4d818cfd380805da573e3d71b3137c1a36bf1a9280a9e5f79"
az vmss run-command invoke -g $RGNAME -n $VMSS --command-id RunShellScript --instance-id $INSTANCE --scripts "docker inspect $CONTAINERID" -o yaml | grep super

As you can see, we now have access to the value of the secret:

This means that anybody with access to Docker on the machines hosting your cluster, can read the secrets you consume as an environment variable. Not only the Azure admins that have access via Azure CLI can execute these command, anybody with access to Docker on those VMs can execute this and read the secrets.

Issue 2: environment variables leak to logging tools

In the case of Azure monitor for containers, the solution stores any environment variables linked to a container. This also stores the secrets in plain text. Let’s have a look.

In the blade of your AKS cluster, head on over to the Insights tab and select containers.

Look for the container we just created. Select that container, and expand the environment variables. There you will see the plain text version of your environment variable

Recommended solutions

The recommendations I’m making here are my own personal recommendations. These are not Microsoft-specific, nor are they Microsoft endorsed.

There are a couple of solutions to avoid the environment variable solution:

• Mount secrets as files, rather than environment variables. Kubernetes natively supports mounting secrets in the container itself as a file rather than an environment variable. This is typically regarded as more secure.
• Use an external key store, such as Azure Key Vault or Hashicorp vault. Both have plugins into Kubernetes to then mount the secrets. e.g. For keyvault there is a FlexVolume driver.
• Use a commercial solution, such as Aqua, to mount secrets into a container. Aqua even has a solution to securely pass environment variables to your containers.

Summary

In this quick post I wanted to highlight that consuming secrets via environment variables in Kubernetes can be risky business. Anybody with access to the Docker hosts or to a monitoring system can get access to the value of the secrets.