How to handle changes to k8s secrets - Thu, Jul 28, 2022
How to handle changes k8s to secrets
How to handle changes to k8s secrets
In my current project we recently discussed how to process changes to secrets like database credentials. And we came to the conclusion that the application should immediately use the new password when it’s corresponding secret was updated. This blog post looks at how updates to secrets are handled in a pod and looks at kustomize to offer a way of automatically redeploying a deployment on a config map change.
Usage of secrets in pods and containers
There are two main ways of using secrets in containers:
- As environment variables
- As mounted files in the file system
Both ways are illustrated in the code snippet below:
spec: containers: - name: file-watchdog env: - name: TEST_SECRET valueFrom: secretKeyRef: name: file-watchdog-secret key: TEST_SECRET ... volumeMounts: - name: secrets mountPath: "/tmp/secrets" readOnly: true volumes: - name: secrets secret: secretName: file-watchdog-secret
The first part of the snippet makes the content of a secret available in the environment variable
TEST_SECRET. The volume mount makes the secret available in the file
/tmp/secrets/TEST_SECRET. For security reasons the later approach is preferred since environment variables may <em>leak</em> to logging frameworks
. Both ways also differ in the way they behave to changes to a secret.
What happens when a secret is changed
Secrets or config maps mounted as files are changed shortly after the corresponding secret has been changed. The following output of a file watchdog shows what happens when the secret is updated:
2022-07-28 13:44:40 - Created directory: /tmp/secrets/..2022_07_28_13_44_40.640096036 2022-07-28 13:44:40 - Modified directory: /tmp/secrets 2022-07-28 13:44:40 - Created file: /tmp/secrets/..2022_07_28_13_44_40.640096036/TEST_SECRET 2022-07-28 13:44:40 - Modified directory: /tmp/secrets/..2022_07_28_13_44_40.640096036 2022-07-28 13:44:40 - Created file: /tmp/secrets/..data_tmp 2022-07-28 13:44:40 - Modified directory: /tmp/secrets 2022-07-28 13:44:40 - Moved file: from /tmp/secrets/..data_tmp to /tmp/secrets/..data 2022-07-28 13:44:40 - Modified directory: /tmp/secrets 2022-07-28 13:44:40 - Deleted file: /tmp/secrets/..2022_07_28_13_43_10.908686010/TEST_SECRET 2022-07-28 13:44:40 - Modified directory: /tmp/secrets/..2022_07_28_13_43_10.908686010 2022-07-28 13:44:40 - Deleted directory: /tmp/secrets/..2022_07_28_13_43_10.908686010 2022-07-28 13:44:40 - Modified directory: /tmp/secrets
First the new secret file is created and the link to
/tmp/secrets/TEST_SECRET is updated. Then the old secret is deleted.
In contrast the environment variable containing the same secret is not updated, since environment variables are set only once when the pod is created. Thus an application is only capable of detecting changes to secret if the secret is mounted in the file system.
Although this difference seems significant, it does not really help the application picking up the changes. Most applications read configurations from the file system only during startup and reloading configuration files is also not quiet common.
So we still need a way of restarting the application once the secret has been changed. Luckily kustomize offers a way to do exactly that.
Using kustomize for secrets
Kustomize is a tool to customize applications during deployment. A very common use case is the addition of a namespace to all manifests. Kustomize uses a declarative approach that describes all changes to be done in files named
kustomization.yaml. One of the features of kustomize is that is it offers so called
generators to generate config maps and secrets from literals or files. These config maps get unique names every time kustomize is run. In addition kustomize offers the ability to replace reference to the secrets base name in other manifests.
This is how the kustomization file looks for our use case:
resources: - deployment.yaml secretGenerator: - name: file-watchdog-secret literals: - TEST_SECRET=12345678901
The complete example can be found here
secretGenerator generates a secret with a key value
TEST_SECRET=12345678901. The name of the secret is always different. For example if I run
kubectl apply -k ., the generated config map is named
file-watchdog-secret-886hdb7mk5 (not the hash as a suffix). We still need to tell kustomize to replace any reference to a secret with the name
file-watchdog-secret-886hdb7mk5. This is done in these lines:
resources: - deployment.yaml
Kustomize searches the manifest
deployment.yaml for reference to
file-watchdog-secret and replaces them with
file-watchdog-secret-886hdb7mk5. So while the original manifest looks like this:
env: - name: TEST_SECRET valueFrom: secretKeyRef: name: file-watchdog-secret key: TEST_SECRET
The customized manifest looks like this:
env: - name: TEST_SECRET valueFrom: secretKeyRef: name: file-watchdog-secret-886hdb7mk5 key: TEST_SECRET
Since the secret has changed, kubernetes will notice that change to the deployment and do an automatic rollout thus restarting the application. Problem solved.
While replacement for standard resources works out of the box, kustomize can also be used to replace the value in any manifest field. The following example would replace the base name of the secret with the generated one in an environment variable value:
resources: - deployment.yaml secretGenerator: - name: file-watchdog-kustomization-secret literals: - TEST_KUST_SECRET=12345678901 configurations: - kustomizeconfig.yaml
Notice the reference to the file
kustomizeconfig.yaml. The content of this file looks like this:
nameReference: - kind: Secret fieldSpecs: - path: spec/template/spec/containers/env/value kind: Deployment
When a kubernetes secret is mounted as a file in a container it will be updated when the secret is changes. If the secret is used as an environment variable, it is not updated since these variables are set only once when the pod is started. For application configuration this usually means that they need to be restarted after a secret change. Luckily kustomize offers a way of automatically doing that.