So, in this post I’m planning to explain why passing secrets via environment variables is a bad idea. I’ll explain two reasons why you shouldn’t use environment variables for secrets:<\/p>\n\n\n\n
- They are passed plain text to Docker<\/li>
- They leak to monitoring tools<\/li><\/ul>\n\n\n\n
In this post I do not aim to tackle the limited security of the secrets implementation itself (meaning, base64 encoded values in etcd), but rather how you consume these secrets in your application. <\/p>\n\n\n\n
Setting up a demo application<\/h2>\n\n\n\n
To demonstrate the findings, we’ll create a quick and dirty demo application, running on top of an AKS cluster in Azure. We’ll create a new secret, and then consume that secret as an environment variable in our pod.<\/p>\n\n\n\n
Let’s first create the secret:<\/p>\n\n\n\n
kubectl create secret generic supersecret --from-literal=secret=supersecret-value<\/code><\/pre>\n\n\n\nWe will use a pod running nginx to consume the secret:<\/p>\n\n\n\n
apiVersion: v1\nkind: Pod\nmetadata:\n name: nginx-env\nspec:\n containers:\n - name: nginx-env\n image: nginx\n env:\n - name: SUPERSECRET\n valueFrom:\n secretKeyRef:\n name: supersecret\n key: secret<\/code><\/pre>\n\n\n\nTo create this pod, you can use the following command:<\/p>\n\n\n\n
kubectl create -f nginx-env.yaml<\/code><\/pre>\n\n\n\nTo consume the secret, we can exec into the pod and echo the value:<\/p>\n\n\n\n
kubectl exec -it nginx-env sh\necho $SUPERSECRET<\/code><\/pre>\n\n\n\nThis, as is to be expected, will show the value of the secret:<\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2020\/02\/image-49.png\")
Execing in the pod shows us the value of the secret. As is expected.<\/figcaption><\/figure>\n\n\n\n Issue 1: Docker run-time outputs environment variables in plain text<\/h2>\n\n\n\n
The first issue with environment variables is that the docker run-time stores their values in plain text, and they can be retrieved by anybody with access to Docker on the cluster. Let me show you:<\/p>\n\n\n\n
To start, we’ll figure out which host is running our pod :<\/p>\n\n\n\n
kubectl get pods nginx-env -o wide<\/code><\/pre>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/02\/image-53-1024x63.png\")
My pod is running on instance 1<\/figcaption><\/figure>\n\n\n\n Then, we’ll need to get the docker ID of the running container. We can get that this way:<\/p>\n\n\n\n
kubectl get pods nginx-env -o yaml | grep containerID<\/code><\/pre>\n\n\n\n![\"\"](\"\/wp-content\/uploads\/2020\/02\/image-54.png\")
Note down the container ID (without the docker:\/\/)<\/figcaption><\/figure>\n\n\n\n We will then do a
docker inspect<\/code> command to get a full description of our running container. Instead of setting up SSH access to the nodes, I’ll show you how you can use invoke command using the Azure CLI to execute commands on the nodes. You’ll need the resource group name and the name of the VMSS hosting your cluster for this.<\/p>\n\n\n\nINSTANCE=1\nVMSS=\"aks-agentpool-39838025-vmss\"\nRGNAME=\"MC_handsonaks_handsonaks_westus2\"\nCONTAINERID=\"9a0ba590d702daa4d818cfd380805da573e3d71b3137c1a36bf1a9280a9e5f79\"\naz vmss run-command invoke -g $RGNAME -n $VMSS --command-id RunShellScript --instance-id $INSTANCE --scripts \"docker inspect $CONTAINERID\" -o yaml | grep super<\/code><\/pre>\n\n\n\nAs you can see, we now have access to the value of the secret:<\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2020\/02\/image-56.png\")
The value of the secret of available plaintext in the output of a docker inspect command.<\/figcaption><\/figure>\n\n\n\n This means that anybody with access to Docker on the machines hosting your cluster, can read the secrets you consume as an environment variable. Not only the Azure admins that have access via Azure CLI can execute these command, anybody with access to Docker on those VMs can execute this and read the secrets.<\/p>\n\n\n\n
Issue 2: environment variables leak to logging tools<\/h2>\n\n\n\n
In the case of Azure monitor for containers, the solution stores any environment variables linked to a container. This also stores the secrets in plain text. Let’s have a look.<\/p>\n\n\n\n
In the blade of your AKS cluster, head on over to the Insights tab and select containers. <\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2020\/02\/image-57-1024x194.png\")
Head over to the container view in container Insights<\/figcaption><\/figure>\n\n\n\n Look for the container we just created. Select that container, and expand the environment variables. There you will see the plain text version of your environment variable<\/p>\n\n\n\n
![\"\"](\"\/wp-content\/uploads\/2020\/02\/image-58-1024x282.png\")
Container insights logs the environment variable in plain text<\/figcaption><\/figure>\n\n\n\n Recommended solutions<\/h2>\n\n\n\n
The recommendations I’m making here are my own personal recommendations. These are not Microsoft-specific, nor are they Microsoft endorsed.<\/em><\/p>\n\n\n\n
There are a couple of solutions to avoid the environment variable solution:<\/p>\n\n\n\n
- Mount secrets as files, rather than environment variables. Kubernetes natively supports mounting secrets in the container itself as a file rather than an environment variable. This is typically regarded as more secure.<\/li>
- Use an external key store, such as Azure Key Vault or Hashicorp vault. Both have plugins into Kubernetes to then mount the secrets. e.g. For keyvault there is a FlexVolume driver<\/a>. <\/li>
- Use a commercial solution, such as Aqua<\/a>, to mount secrets into a container. Aqua even has a solution to securely pass environment variables<\/a> to your containers. <\/li><\/ul>\n\n\n\n
Summary<\/h2>\n\n\n\n
In this quick post I wanted to highlight that consuming secrets via environment variables in Kubernetes can be risky business. Anybody with access to the Docker hosts or to a monitoring system can get access to the value of the secrets. <\/p>\n","protected":false},"excerpt":{"rendered":"
Managing secrets is a complicated endeavor. Kubernetes has a native secrets implementation, that allows you to store and access secrets from your deployments. A while ago, I read a short free book on Kubernetes Security, by Liz Rice and Michael Hausenblas (apparently O’Reilly calls it a report, but I actually have a hard copy that […]<\/p>\n","protected":false},"author":1,"featured_media":801,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3,58,85],"tags":[86,18,50],"class_list":["post-798","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-kubernetes","category-security","tag-docker","tag-kubernetes","tag-security"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2020\/02\/image-49.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/798","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=798"}],"version-history":[{"count":3,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/798\/revisions"}],"predecessor-version":[{"id":812,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/798\/revisions\/812"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/801"}],"wp:attachment":[{"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=798"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=798"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=798"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Use a commercial solution, such as Aqua<\/a>, to mount secrets into a container. Aqua even has a solution to securely pass environment variables<\/a> to your containers. <\/li><\/ul>\n\n\n\n