Posted on 12 August 2019, updated on 21 December 2023.
I encountered an issue with the container orchestrator, Kubernetes, where I couldn’t set up a secret as an environment variable: my service only supported passing this variable as a single configuration file along with non-sensitive data. To address this issue, I called Helm to the rescue.
Issue: Kubernetes can't natively merge Secret and ConfigMap in a single file
Let’s say we have this standard yaml configuration file:
| parameters: | |
| database_username: username | |
| database_password: $DATABASE_PASSWORD | |
| database_host: 127.0.01 | |
| database_port: 3456 |
Applying Kubernetes concepts we should only store the database password in a Secret and the rest should be in a ConfigMap.
But we wouldn’t be able to merge those two data sources in a single file while using those ConfigMap and Secret as a volume:
| volumes: | |
| - name: tmpldirs | |
| configMap: | |
| name: configmap | |
| - name: configdir | |
| secret: | |
| secretName: secret |
Well Kubernetes doesn’t support it and does not aim at supporting it: https://github.com/kubernetes/kubernetes/issues/30716
But with the help of Helm and a little bash trick you can achieve it!
Setup: Creating a Helm chart
First use Helm to create your chart:
helm create templating-inception
Let’s say your config files are at the root directory of your repository here’s the directory setup you’ll need:
| ├── config | |
| │ └── parameters.yaml | |
| ├── src | |
| └── templating-inception | |
| ├── charts | |
| ├── Chart.yaml | |
| ├── config -> ../config | |
| ├── templates | |
| │ ├── deployment.yaml | |
| │ ├── _helpers.tpl | |
| │ ├── ingress.yaml | |
| │ ├── NOTES.txt | |
| │ ├── service.yaml | |
| │ └── tests | |
| │ └── test-connection.yaml | |
| └── values.yaml |
Nothing too exciting except for this symbolic link, we’ll come back to that later but it’s a requirement if you want to keep your configuration files that way.
Helm: Templating the template
We are going to use the templating function of Helm during the templating process of our ConfigMap, here’s the configmap.yaml file you should create:
| apiVersion: v1 | |
| kind: ConfigMap | |
| metadata: | |
| name: {{ .Release.Name }}-configmap | |
| data: | |
| {{- tpl ((.Files.Glob "config/*.tmpl").AsConfig) . | nindent 2 }} |
The instruction in this template reads all files suffixed by .tmpl and set them up as filename:filecontent in this ConfigMap.
As you can see we will have to rename our parameters.yaml file parameters.yaml.tmpl and use some Helm templating:
This step isn’t required but it helps when working with multiple environments.
| parameters: | |
| database_username: {{ .Values.configs.databaseUsername }} | |
| database_password: ${{ .Values.secrets.databasePassword }} | |
| database_host: {{ .Values.configs.databaseHost }} | |
| database_port: {{ .Values.configs.databasePort }} |
Let’s see add our values to the values.yaml file:
| ... | |
| configs: | |
| databaseUsername: username | |
| databaseHost: 127.0.0.1 | |
| databasePort: 3456 | |
| secrets: | |
| databasePassword: DATABASE_PASSWORD | |
| ... |
We’re close achieve our goal but for now, we want Helm to template our parameters.yaml file as follow:
| parameters: | |
| database_username: username | |
| database_password: $DATABASE_PASSWORD | |
| database_host: 127.0.01 | |
| database_port: 3456 |
NB: The symlink you created before was only there because Helm uses server-side rendering on Tiller to compute the resources it has to deploy and only the chart directory is uploaded to Tiller. For now, Helm doesn’t support referencing paths higher in the hierarchy but with the 3.0.0 version currently in alpha and the disappearance of Tiller this should be achievable.
Bash: Substitutor of environments
The last part is a mix of the envsubst command and the concept of initContainer in Kubernetes.
The description of the envsubst command is as clear as its name: “Substitutes the values of environment variables.”
Let’s use it in our deployment.yaml file:
| apiVersion: apps/v1 | |
| kind: Deployment | |
| metadata: | |
| name: {{ include "templating-inception.fullname" . }} | |
| labels: | |
| app.kubernetes.io/name: {{ include "templating-inception.name" . }} | |
| helm.sh/chart: {{ include "templating-inception.chart" . }} | |
| app.kubernetes.io/instance: {{ .Release.Name }} | |
| app.kubernetes.io/managed-by: {{ .Release.Service }} | |
| spec: | |
| replicas: {{ .Values.replicaCount }} | |
| selector: | |
| matchLabels: | |
| app.kubernetes.io/name: {{ include "templating-inception.name" . }} | |
| app.kubernetes.io/instance: {{ .Release.Name }} | |
| template: | |
| metadata: | |
| labels: | |
| app.kubernetes.io/name: {{ include "templating-inception.name" . }} | |
| app.kubernetes.io/instance: {{ .Release.Name }} | |
| spec: | |
| initContainers: | |
| - name: config | |
| image: "{{ .Values.initContainer.image }}" | |
| command: ['sh', '-c', 'for file in $(ls -A /tmpl/ | grep tmpl); do newfile=${file%".tmpl"}; envsubst < /tmpl/$file > /app/config/$newfile; done'] | |
| volumeMounts: | |
| - name: tmpldir | |
| mountPath: /tmpl | |
| - name: configdir | |
| mountPath: /config | |
| envFrom: | |
| - secretRef: | |
| name: {{ .Values.secretName }} | |
| containers: | |
| - name: {{ .Chart.Name }} | |
| image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" | |
| imagePullPolicy: {{ .Values.image.pullPolicy }} | |
| ports: | |
| - name: http | |
| containerPort: 80 | |
| protocol: TCP | |
| livenessProbe: | |
| httpGet: | |
| path: / | |
| port: http | |
| readinessProbe: | |
| httpGet: | |
| path: / | |
| port: http | |
| resources: | |
| {{- toYaml .Values.resources | nindent 12 }} | |
| volumeMounts: | |
| - name: configdir | |
| mountPath: /config | |
| volumes: | |
| - name: tmpldir | |
| configMap: | |
| name: {{ .Release.Name }}-configmap | |
| - name: configdir | |
| emptyDir: {} |
Here’s the shell command launched by the initContainer:
sh -c for file in $(ls -A /tmpl/ | grep tmpl); do newfile=${file%".tmpl"}; envsubst < /tmpl/$file > /app/config/$newfile; done'
The $DATABASE_PASSWORD environment variable will be substituted with its value in the parameters.yaml.
Using this template you’ll have to setup those values in your values.yaml file as well:
| ... | |
| secretName: db # Secret name containing your database password setup as DATABASE_PASSWORD:password | |
| initContainer: | |
| image: image # You'll need an image containing the envsubst command | |
| tag: tag | |
| ... |
Here’s your service getting its configuration /config/paramaeters.yaml file with both values from a Secret and a ConfigMap.
This setup will work for any file added to the config directory.
Merging a ConfigMap and a Secret in a single file is a complex process that requires knowledge of Helm and Kubernetes if you have any questions or suggestions, don't hesitate to contact us to share your experience with us.
Also, check out our article about Kubernetes productivity tips and tricks to go further on Kubernetes usage.