Safeguard Application Secrets Efficiently in Kubernetes using AWS Secrets Manager

It’s no wonder that we all have some secrets in our lives. However, maintaining them can be exhausting sometimes! Likewise, the diverse range of applications and systems in the IT ecosystem consists of secrets, which makes it arduous for enterprises to securely store, transmit, and audit them.

Kubernetes Secrets let you store and manage sensitive information, such as passwords, authentication tokens, and ssh keys. However, a lot of production-grade systems avoid using Kubernetes Secrets for storing secrets as it does not include an option for strong encryptions but only base64 encodings. Further, it is also recommended to externalize secrets to secret management tools like AWS Secret Manager or Vault. 

AWS ECS has out-of-the- box integration with AWS Secrets Manager where one could externalize secrets to AWS Secret Manager whereas no such direct integration exists with AWS EKS.

This blog showcases how we can externalize application secrets in Kubernetes to AWS Secret Manager with a sample implementation where your application secrets are injected into your application container using an init-container.

Working Mechanism

The implementation contains a simple python based init-container for fetching secrets from AWS Secrets Manager to Kubernetes pods. It fetches the provided key from AWS Secrets Manager and stores it in pods' volume as emptyDir at a specified location so that the main containers can use them in the application.

Implementation Process

The init-secret container requires these env variables to work-

• SECRET_FILE_PATH - Specifies the location in emptyDir volume to store the credentials. For example - /secret/secret.env
• SM_DB - The key to the secrets manager to be fetched. For example - demo-database-secret which would contain our secrets as key-value pairs.
• AWS_REGION - The AWS region where the secrets have been put in the secrets manager. For example - ap-southeast-1

Before you implement, consider these important points-

• The name of the env should be in the format *SM_<.env-prefix>*. For eg.: SM_DB or SM_SECRETS
• All the keys fetched from the secret manager will be converted to uppercase and prefixed with env-prefix. For example - if the env var for the init-container is SM_DB then the key for password would be DB_PASSWORD.
• The volume secret-volume should be an emptyDir and mounted to both init-container and main containers.
• Make sure the pods have access to the AWS Secret Manager. This implementation assumes that the required permissions have been granted to the EC2 Instance Profile (add permissions to Amazon EKS node IAM role in case of EKS) to nodes where pods would be spinning up, instead of using AWS Access Keys and Secret Keys. An example policy for the minimal permissions required can be found in examples/secrets-policy.json.

AWS Secrets Manager

Create a secret in AWS Secret Manager as shown below. Further, follow this AWS guide for creating a basic secret.

Build Your Docker Image

To build your docker image for init-container simply clone this repository and run below Docker commands:

➜  ~ docker build -t srijanlabs/init-secret:latest .

➜  ~ docker push srijanlabs/init-secret:latest

Kubernetes Implementation

➜  ~ kubectl get nodes

NAME                                                STATUS   ROLES    AGE   VERSION
ip-192-168-36-74.ap-southeast-1.compute.internal    Ready    <none>   72m   v1.16.13-eks-2ba888
ip-192-168-48-236.ap-southeast-1.compute.internal   Ready    <none>   72m   v1.16.13-eks-2ba888
ip-192-168-73-240.ap-southeast-1.compute.internal   Ready    <none>   71m   v1.16.13-eks-2ba888

• A basic kubernetes deployment yaml looks like below and can be found in examples/k8s-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-manager
  labels:
    app: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      volumes:
      - name: secret-volume
        emptyDir: {}
      initContainers:
      - name: init-secret
        image: srijanlabs/init-secret:latest
        imagePullPolicy: Always
        command: ["python"]
        args:
          - "secret.py"

       env:
        - name: SECRET_FILE_PATH
          value: "/secret/secret.env"
        - name: SM_DB
          value: "demo-database-secret"
        - name: AWS_REGION
          value: "ap-southeast-1"
        volumeMounts:
          - mountPath: /secret
            name: secret-volume
      containers:
      - name: main-app
        image: busybox
        imagePullPolicy: Always
        command: ["/bin/sh", "-c"]

        # Run your application command after sourcing env file or use the env file to your convenience

        args:
          - "source /secret/secret.env && while true; do echo '\n\n$DB_PASSWORD = '$DB_PASSWORD; echo '\nContents of secret.env file:'; cat /secret/secret.env;  sleep 2; done"

        env:
          - name: SECRET_FILE_PATH
            value: "/secret/secret.env"
        volumeMounts:
          - mountPath: /secret
            name: secret-volume

• Apply the yaml to create a deployment:

• Check the status of the created pods, here Init:0/1 indicates that zero of the one Init Containers has completed successfully:

➜  ~ kubectl get pods

NAME                              READY   STATUS     RESTARTS   AGE
secret-manager-6685f778c7-lmk9g   0/1     Init:0/1   0          2s
secret-manager-6685f778c7-vggxn   0/1     Init:0/1   0          2s

• After a while if the init-container runs successfully pods goes in PodInitializing state which indicates our main app container is being created now:

➜  ~ kubectl get pods

NAME                              READY   STATUS            RESTARTS   AGE
secret-manager-6685f778c7-lmk9g   0/1     PodInitializing   0          5s
secret-manager-6685f778c7-vggxn   0/1     PodInitializing   0          5s

• As soon as the main app container is up and running, the pod status becomes Running :

➜  ~ kubectl get pods

NAME                              READY   STATUS    RESTARTS   AGE
secret-manager-6685f778c7-lmk9g   1/1     Running   0          10s
secret-manager-6685f778c7-vggxn   1/1     Running   0          10s

• To check the logs of our init-secret container, simply append -c init-secret to the kubectl logs command :

➜  ~ kubectl logs secret-manager-6685f778c7-lmk9g -c init-secret

Running init container script
Saving demo-database-secret secrets to /secret/secret.env
Done fetching secrets demo-database-secret
Exiting init container script

• To check the logs of the main container, use the following command :

➜  ~ kubectl logs secret-manager-6685f778c7-lmk9g

$DB_PASSWORD = supersecretpassword
Contents of secret.env file:
DB_USERNAME='demo'
DB_PASSWORD='supersecretpassword'
DB_ENGINE='mysql'
DB_HOST='127.0.0.1'
DB_PORT='3306'
DB_DBNAME='demo'

• As we can see, the secrets have been injected properly in our main container which can easily be used as required, without having to create any other Kubernetes resources. The values of the secrets in secret.env is exactly the same as we have created in our AWS Secret Manager.
• To clean up the deployment, we have created -

➜  ~ kubectl delete -f examples/k8s-deployment.yaml

Bottom Line

As security remains a major concern across all industries and domains, it would be vitally important for enterprises to capitalize on a potential secrets management tool to avoid any business disruptions. Externalizing concerns like secrets to secret management tools is an ideal recommendation and this sample implementation sheds light on how this can  be achieved using init-containers in Kubernetes.