In this tutorial, you will learn how to create and inject secrets into your application running on Kubernetes using Hashicorp Vault. According to the official website HashiCorp Vault, can be defined as:
HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates.
Vault allows you to manage all your secrets in one place, as it takes care of encryption, leasing, and renewal, as well as secrets revocation when required. Implementing any one of these by itself is quite the challenge which is why Vault is a good choice if you are looking for a cloud-agnostic way to manage secrets.
Prerequisites
This tutorial assumes some familiarity with Kubernetes and helm. In addition, you will also need:
Creating a cluster using the Civo CLI
We’ll begin by creating a Kubernetes cluster using the Civo command line:
$ civo kubernetes create vault-experiments --size "g4s.kube.medium" --nodes 2 --wait --save --merge --region NYC1
The above command will create a cluster with two nodes and save your KUBECONFIG so you can access your cluster.
Next, you will need to run the following command to switch your Kubernetes context to your new cluster:
kubectl config use-context vault-experiments
Installing Vault
Hashicorp recomends installing vault in a separate namespace for logical separation and isolation.
Run the following command to create a namespace:
kubectl create namespace vault
Next, add the hashicorp helm repository:
helm repo add hashicorp https://helm.releases.hashicorp.com
Install Vault using helm:
helm install vault hashicorp/vault --namespace vault
This will install vault in standalone mode. This is ideal for simple use cases such as this demo, however, if you plan to run Vault in production, take a look at this doc on running Vault in high availability mode.
Confirm the installation to the cluster has been successful. You should see something like the following when you run kubectl get pods -n vault:
$ kubectl get pods -n vault
vault vault-agent-injector-5c5b87595-4fr7v 1/1 Running 0 2m23s
vault vault-0 0/1 Running 0 2m23s
At first glance it will appear that vault-0 is not running:
This is because the status check defined in a readinessProbe returns a non-zero exit code.
Unsealing Vault
By default, every Vault installation is sealed. This means Vault knows where and how to access the physical storage, but doesn't know how to decrypt any of it.
Vault uses Shamir’s secret sharing algorithm to distribute the keys required to unseal a vault. As we are running in standalone mode, we only have to unseal one Vault instance. In a production environment where Vault is running in HA mode you would be required to unseal all the Vault instances. For an easier approach to this operation, have a look into auto unsealing.
Initialise Vault:
kubectl exec -n vault vault-0 -- vault operator init \
-key-shares=1 \
-key-threshold=1 \
-format=json > keys.json
Unseal Vault:
kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault operator unseal #unseal_key_b64
# output
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.11.3
Build Date 2022-08-26T10:27:10Z
Storage Type file
Cluster Name vault-cluster-2695a1ff
Cluster ID 023d5e2a-ca48-c1ed-94c6-1268d6f20b23
HA Enabled false
NOTE: This would output your vault unseal keys and root token into a file named
keys.json, both of which are vital to Vault's operation. Make sure to store them somewhere secure.
Enabling Kubernetes Authentication
Vault supports several methods for authentication. This is useful for assigning identity and a set of policies to a user. In this tutorial we will be using Kubernetes as the authentication provider.
For a more comprehensive list of supported providers take a look at this section of the Vault documentation.
First let's provide the Vault CLI access to the Vault instance running in your cluster:
# export vault server address
# portfoward vault-0
export VAULT_ADDR=http://localhost:8200
kubectl -n vault port-forward vault-0 8200:8200
In another terminal window, log in using the vault CLI:
vault login # your_root token
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token #token
token_accessor #token accessor
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
You can also explore the Vault UI which should be available here during the port-forward.
Enable Kubernetes auth:
vault auth enable kubernetes
Kubernetes authentication is now enabled but we need to provide Vault some information about our cluster:
# exec into vault-0
kubectl exec -n vault -it vault-0 -- /bin/sh
# login to vault
vault login root_token
#pass auth info
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
In the command above we provided vault with a jwt token, Kubernetes CA certificate and the Kubernetes API server address. This will allow Vault to make authenticated API requests.
We’re not quite done yet, though. Next we need to create a service account with a cluster-role binding of system:auth-delegator.
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth
namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: vault
This will need to be applied to the cluster:
kubectl apply -f cluster-rolebinding.yaml
To ensure our application has access to secrets that would be passed to it , we will create a vault policy that has read and list permissions. Create a file called policy.hcl and add the following:
# policy.hcl
path "apps/data/guardian" {
capabilities = ["read", "list"]
}
Write the policy to vault:
vault policy write guardian-ro policy.hcl
The command above shows the policy definition for an application called guardian and subsequent secrets would be placed under the path apps/data/guardian.
You can view the applied policy in the Vault UI by clicking on the policies tab
Finally bind the policy and service account to the role for the guardian app by running:
vault write auth/kubernetes/role/guardian \
bound_service_account_names=guardian-sa \
bound_service_account_namespaces=default \
policies=guardian-ro \
ttl=24h
# Success! Data written to: auth/kubernetes/role/guardian
In the command above we bind a service account guardian-sa to a the default Kubernetes namespace and give specify a time to live of 24hrs.
Deploying an application
Now that we have vault all set up, let's deploy an application that we can inject secrets into. As mentioned earlier we would be deploying an app called guardian which spits out the secrets vault would pass to it. Save this app definition in a file called deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: guardian
spec:
selector:
matchLabels:
app: guardian
template:
metadata:
labels:
app: guardian
spec:
serviceAccountName: guardian-sa
containers:
- name: guardian
image: ghcr.io/s1ntaxe770r/guardian:v1.1
resources:
limits:
memory: "128Mi"
cpu: "500m"
imagePullPolicy: Always
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: guardian
spec:
selector:
app: guardian
ports:
- protocol: TCP
port: 8080
targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: guardian-sa
namespace: default
Apply the deployment:
kubectl apply -f deployment.yaml
Notice we are using guardian-sa as the service account in this deployment. This is required for Vault to grab secrets we would pass shortly.
Next, once applied to your cluster, expose the deployment through a local port-forward:
kubectl port-forward svc/guardian 8080:8080
Explore the secrets endpoint:
curl http://localhost:8080/secrets
# result
{
"msg": "I hold no secrets",
"secrets": {}
}
Guardian currently holds no secrets so lets give it some using Vault.
Injecting Vault secrets
Begin by enabling the key/value store:
vault secrets enable -path=apps kv-v2
Add the following secrets:
vault kv put apps/guardian guardian_message="a word from our sponsor civo cloud" guardian_encryption_key="zsdasdfaskfjhj4534"
We specifically enable version 2 of Vault’s key value store. There are some differences in how Vault handles data stored in v1 and v2 of the store. In this example, secrets stored in
apps/guardianwould be available atapps/data/guardian. The defined Guardian app looks for the secrets placed under/vault/secrets/guardian.txtin the pod
Update the deployment to tell it to grab the secrets:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: guardian
spec:
selector:
matchLabels:
app: guardian
template:
metadata:
labels:
app: guardian
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: 'guardian'
vault.hashicorp.com/agent-inject-secret-guardian.txt: 'apps/data/guardian'
vault.hashicorp.com/agent-inject-template-guardian.txt: |
{{ with secret "apps/data/guardian" }}
guardian_encryption_key:{{ .Data.data.guardian_encryption_key }}
guardian_message:{{ .Data.data.guardian_message }}
{{- end }}
spec:
serviceAccountName: guardian-sa
containers:
- name: guardian
image: ghcr.io/s1ntaxe770r/guardian:v1.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
imagePullPolicy: Always
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: guardian
spec:
selector:
app: guardian
ports:
- protocol: TCP
port: 8080
targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: guardian-sa
namespace: default
Apply the changed deployment:
kubectl apply -f deployment.yaml
The updated deployment has a few new annotations, let walk through each of them:
vault.hashicorp.com/agent-inject: 'true' enable tells Vault to inject a sidecar container into the application pod. If you don't want a sidecar running alongside your application consider adding vault.hashicorp.com/agent-pre-populate-only: 'true'. This would still inject secrets without a sidecar.
vault.hashicorp.com/agent-inject-secret-guardian.txt: 'apps/data/guardian' this tells Vault to inject the secret we added earlier into a file called guardian.txt.
vault.hashicorp.com/role: 'guardian' tells Vault what role to use, in this example the role named “guardian” which we created earlier.
vault.hashicorp.com/agent-inject-template-guardian.txt tells Vault how secrets should be rendered in the file. This would be available in /vault/secrets/guardian.txt inside the container.
Once applied, you can curl query the secrets endpoint:
curl http://localhost:8080/secrets
{
"msg": "Egads!! I have some secrets, I must inform you at once",
"secrets": {
"encryption_key": "zsdasdfaskfjhj4534",
"lets hear": "a word from your sponsor civo cloud "
}
}
You can explore the container to see how the secrets are placed:
# exec into guardian
kubectl exec --stdin=true --tty=true svc/guardian -- sh
# cat guardian.txt
cat /vault/secrets/guardian.txt
Summary
In this tutorial, we deployed Hashicorp Vault and injected secrets into a secure app that tells us exactly what those secrets are. No doubt Vault has a learning curve, however, when you take into account that rotating credentials and secrets can now be managed from a single place, you start to see where Vault shines. The application example used in this tutorial is available over here.
Next Steps
From here you might be wondering what to do next. Here are a few ideas:
Check out the helm chart reference for more values that can be used to tune Vault here.
Vault has a few more annotations, asides the ones covered in this guide. Take a look at the annotations reference for more information here.
Finally, take a look at the Go SDK for vault if you would like to programmatically interact with Vault here.
