It’s no wonder that we all have some secrets in our lives. However, maintaining them can be exhausting sometimes! Likewise, the diverse range of applications and systems in the IT ecosystem consists of secrets, which makes it arduous for enterprises to securely store, transmit, and audit them.
Kubernetes Secrets let you store and manage sensitive information, such as passwords, authentication tokens, and ssh keys. However, a lot of production-grade systems avoid using Kubernetes Secrets for storing secrets as it does not include an option for strong encryptions but only base64 encodings. Further, it is also recommended to externalize secrets to secret management tools like AWS Secret Manager or Vault.
AWS ECS has out-of-the- box integration with AWS Secrets Manager where one could externalize secrets to AWS Secret Manager whereas no such direct integration exists with AWS EKS.
This blog showcases how we can externalize application secrets in Kubernetes to AWS Secret Manager with a sample implementation where your application secrets are injected into your application container using an init-container.
Working Mechanism
The implementation contains a simple python based init-container for fetching secrets from AWS Secrets Manager to Kubernetes pods. It fetches the provided key from AWS Secrets Manager and stores it in pods' volume as emptyDir at a specified location so that the main containers can use them in the application.
Implementation Process
The init-secret container requires these env variables to work-
- SECRET_FILE_PATH - Specifies the location in emptyDir volume to store the credentials. For example - /secret/secret.env
- SM_DB - The key to the secrets manager to be fetched. For example - demo-database-secret which would contain our secrets as key-value pairs.
- AWS_REGION - The AWS region where the secrets have been put in the secrets manager. For example - ap-southeast-1
Before you implement, consider these important points-
- The name of the env should be in the format *SM_<.env-prefix>*. For eg.: SM_DB or SM_SECRETS
- All the keys fetched from the secret manager will be converted to uppercase and prefixed with env-prefix. For example - if the env var for the init-container is SM_DB then the key for password would be DB_PASSWORD.
- The volume secret-volume should be an emptyDir and mounted to both init-container and main containers.
- Make sure the pods have access to the AWS Secret Manager. This implementation assumes that the required permissions have been granted to the EC2 Instance Profile (add permissions to Amazon EKS node IAM role in case of EKS) to nodes where pods would be spinning up, instead of using AWS Access Keys and Secret Keys. An example policy for the minimal permissions required can be found in examples/secrets-policy.json.
AWS Secrets Manager
Create a secret in AWS Secret Manager as shown below. Further, follow this AWS guide for creating a basic secret.
Build Your Docker Image
To build your docker image for init-container simply clone this repository and run below Docker commands:
➜ ~ docker build -t srijanlabs/init-secret:latest .
➜ ~ docker push srijanlabs/init-secret:latest
Kubernetes Implementation
➜ ~ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-36-74.ap-southeast-1.compute.internal Ready <none> 72m v1.16.13-eks-2ba888
ip-192-168-48-236.ap-southeast-1.compute.internal Ready <none> 72m v1.16.13-eks-2ba888
ip-192-168-73-240.ap-southeast-1.compute.internal Ready <none> 71m v1.16.13-eks-2ba888
- A basic kubernetes deployment yaml looks like below and can be found in examples/k8s-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: secret-manager
labels:
app: my-app
spec:
replicas: 2
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
volumes:
- name: secret-volume
emptyDir: {}
initContainers:
- name: init-secret
image: srijanlabs/init-secret:latest
imagePullPolicy: Always
command: ["python"]
args:
- "secret.py"
env:
- name: SECRET_FILE_PATH
value: "/secret/secret.env"
- name: SM_DB
value: "demo-database-secret"
- name: AWS_REGION
value: "ap-southeast-1"
volumeMounts:
- mountPath: /secret
name: secret-volume
containers:
- name: main-app
image: busybox
imagePullPolicy: Always
command: ["/bin/sh", "-c"]
# Run your application command after sourcing env file or use the env file to your convenience
args:
- "source /secret/secret.env && while true; do echo '\n\n$DB_PASSWORD = '$DB_PASSWORD; echo '\nContents of secret.env file:'; cat /secret/secret.env; sleep 2; done"
env:
- name: SECRET_FILE_PATH
value: "/secret/secret.env"
volumeMounts:
- mountPath: /secret
name: secret-volume
- Apply the yaml to create a deployment:
- Check the status of the created pods, here Init:0/1 indicates that zero of the one Init Containers has completed successfully:
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
secret-manager-6685f778c7-lmk9g 0/1 Init:0/1 0 2s
secret-manager-6685f778c7-vggxn 0/1 Init:0/1 0 2s
- After a while if the init-container runs successfully pods goes in PodInitializing state which indicates our main app container is being created now:
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
secret-manager-6685f778c7-lmk9g 0/1 PodInitializing 0 5s
secret-manager-6685f778c7-vggxn 0/1 PodInitializing 0 5s
- As soon as the main app container is up and running, the pod status becomes Running :
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
secret-manager-6685f778c7-lmk9g 1/1 Running 0 10s
secret-manager-6685f778c7-vggxn 1/1 Running 0 10s
- To check the logs of our init-secret container, simply append -c init-secret to the kubectl logs command :
➜ ~ kubectl logs secret-manager-6685f778c7-lmk9g -c init-secret
Running init container script
Saving demo-database-secret secrets to /secret/secret.env
Done fetching secrets demo-database-secret
Exiting init container script
- To check the logs of the main container, use the following command :
➜ ~ kubectl logs secret-manager-6685f778c7-lmk9g
$DB_PASSWORD = supersecretpassword
Contents of secret.env file:
DB_USERNAME='demo'
DB_PASSWORD='supersecretpassword'
DB_ENGINE='mysql'
DB_HOST='127.0.0.1'
DB_PORT='3306'
DB_DBNAME='demo'
- As we can see, the secrets have been injected properly in our main container which can easily be used as required, without having to create any other Kubernetes resources. The values of the secrets in secret.env is exactly the same as we have created in our AWS Secret Manager.
- To clean up the deployment, we have created -
➜ ~ kubectl delete -f examples/k8s-deployment.yaml
Bottom Line
As security remains a major concern across all industries and domains, it would be vitally important for enterprises to capitalize on a potential secrets management tool to avoid any business disruptions. Externalizing concerns like secrets to secret management tools is an ideal recommendation and this sample implementation sheds light on how this can be achieved using init-containers in Kubernetes.
Our Services
Customer Experience Management
- Content Management
- Marketing Automation
- Mobile Application Development
- Drupal Support and Maintanence
Enterprise Modernization, Platforms & Cloud
- Modernization Strategy
- API Management & Developer Portals
- Hybrid Cloud & Cloud Native Platforms
- Site Reliability Engineering