How to handle changes to k8s secrets

How to handle changes to k8s secrets

In my current project we recently discussed how to process changes to secrets like database credentials. And we came to the conclusion that the application should immediately use the new password when it’s corresponding secret was updated. This blog post looks at how updates to secrets are handled in a pod and looks at kustomize to offer a way of automatically redeploying a deployment on a config map change.

Usage of secrets in pods and containers

There are two main ways of using secrets in containers:

• As environment variables
• As mounted files in the file system

Both ways are illustrated in the code snippet below:

spec:
    containers:
    - name: file-watchdog
      env:
        - name: TEST_SECRET
          valueFrom:
            secretKeyRef:
            name: file-watchdog-secret
            key: TEST_SECRET
...
      volumeMounts:
      - name: secrets
        mountPath: "/tmp/secrets"
        readOnly: true
    volumes:
    - name: secrets
      secret:
        secretName: file-watchdog-secret

The first part of the snippet makes the content of a secret available in the environment variable TEST_SECRET. The volume mount makes the secret available in the file /tmp/secrets/TEST_SECRET. For security reasons the later approach is preferred since environment variables may <em>leak</em> to logging frameworks . Both ways also differ in the way they behave to changes to a secret.

What happens when a secret is changed

Secrets or config maps mounted as files are changed shortly after the corresponding secret has been changed. The following output of a file watchdog shows what happens when the secret is updated:

2022-07-28 13:44:40 - Created directory: /tmp/secrets/..2022_07_28_13_44_40.640096036
2022-07-28 13:44:40 - Modified directory: /tmp/secrets
2022-07-28 13:44:40 - Created file: /tmp/secrets/..2022_07_28_13_44_40.640096036/TEST_SECRET
2022-07-28 13:44:40 - Modified directory: /tmp/secrets/..2022_07_28_13_44_40.640096036
2022-07-28 13:44:40 - Created file: /tmp/secrets/..data_tmp
2022-07-28 13:44:40 - Modified directory: /tmp/secrets
2022-07-28 13:44:40 - Moved file: from /tmp/secrets/..data_tmp to /tmp/secrets/..data
2022-07-28 13:44:40 - Modified directory: /tmp/secrets
2022-07-28 13:44:40 - Deleted file: /tmp/secrets/..2022_07_28_13_43_10.908686010/TEST_SECRET
2022-07-28 13:44:40 - Modified directory: /tmp/secrets/..2022_07_28_13_43_10.908686010
2022-07-28 13:44:40 - Deleted directory: /tmp/secrets/..2022_07_28_13_43_10.908686010
2022-07-28 13:44:40 - Modified directory: /tmp/secrets

First the new secret file is created and the link to /tmp/secrets/TEST_SECRET is updated. Then the old secret is deleted.
In contrast the environment variable containing the same secret is not updated, since environment variables are set only once when the pod is created. Thus an application is only capable of detecting changes to secret if the secret is mounted in the file system.
Although this difference seems significant, it does not really help the application picking up the changes. Most applications read configurations from the file system only during startup and reloading configuration files is also not quiet common.
So we still need a way of restarting the application once the secret has been changed. Luckily kustomize offers a way to do exactly that.

Using kustomize for secrets

Kustomize is a tool to customize applications during deployment. A very common use case is the addition of a namespace to all manifests. Kustomize uses a declarative approach that describes all changes to be done in files named kustomization.yaml. One of the features of kustomize is that is it offers so called generators to generate config maps and secrets from literals or files. These config maps get unique names every time kustomize is run. In addition kustomize offers the ability to replace reference to the secrets base name in other manifests.
This is how the kustomization file looks for our use case:

resources:
  - deployment.yaml

secretGenerator:
- name: file-watchdog-secret
  literals:
  - TEST_SECRET=12345678901

The complete example can be found here . The secretGenerator generates a secret with a key value TEST_SECRET=12345678901. The name of the secret is always different. For example if I run kubectl apply -k ., the generated config map is named file-watchdog-secret-886hdb7mk5 (not the hash as a suffix). We still need to tell kustomize to replace any reference to a secret with the name file-watchdog-secret with file-watchdog-secret-886hdb7mk5. This is done in these lines:

resources:
  - deployment.yaml

Kustomize searches the manifest deployment.yaml for reference to file-watchdog-secret and replaces them with file-watchdog-secret-886hdb7mk5. So while the original manifest looks like this:

env:
    - name: TEST_SECRET
      valueFrom:
        secretKeyRef:
        name: file-watchdog-secret
        key: TEST_SECRET

The customized manifest looks like this:

env:
    - name: TEST_SECRET
      valueFrom:
        secretKeyRef:
        name: file-watchdog-secret-886hdb7mk5
        key: TEST_SECRET

Since the secret has changed, kubernetes will notice that change to the deployment and do an automatic rollout thus restarting the application. Problem solved.
While replacement for standard resources works out of the box, kustomize can also be used to replace the value in any manifest field. The following example would replace the base name of the secret with the generated one in an environment variable value:

resources:
  - deployment.yaml

secretGenerator:
- name: file-watchdog-kustomization-secret
  literals:
  - TEST_KUST_SECRET=12345678901

configurations:
- kustomizeconfig.yaml

Notice the reference to the file kustomizeconfig.yaml. The content of this file looks like this:

nameReference:
- kind: Secret
  fieldSpecs:
  - path: spec/template/spec/containers/env/value
    kind: Deployment

Conclusion

When a kubernetes secret is mounted as a file in a container it will be updated when the secret is changes. If the secret is used as an environment variable, it is not updated since these variables are set only once when the pod is started. For application configuration this usually means that they need to be restarted after a secret change. Luckily kustomize offers a way of automatically doing that.

Back to Home