This piece with co-written by Keith Hubner, Integration Engineer (EMEA), at Bitwarden.

Handling secrets directly in Kubernetes presents a myriad of challenges that can complicate the management of sensitive information. Hardcoding secrets can lead to significant difficulties, as they become harder to manage and may create gaps in access control and auditing features. Additionally, the processes of rotating and revoking secrets can be cumbersome, increasing the risk of exposure.

To add to these challenges, integrating Kubernetes with external secrets management tools often proves complicated, making secure and efficient secrets management within Kubernetes a daunting task. Learn more about some of these challenges.

Effective secrets management is crucial in DevOps, as it securely handles sensitive information such as passwords, API keys, and tokens. Proper secrets management ensures secure storage, access control, rotation, and auditing, preventing the risk of exposure within code or pipelines. Tools like Bitwarden Secrets Manager play a vital role in automating secret management, seamlessly integrating it into DevOps workflows to enhance both security and efficiency.

By addressing these challenges and implementing robust secrets management strategies, organizations can protect their sensitive data while benefiting from the collaborative and agile nature of DevOps practices.

Prerequisites

Before getting started with this tutorial, you will need to ensure you have the following:

  • Access to Bitwarden Secrets Manager account. You can read the help documentation here.
  • You will also need a Civo account. Instructions on how to do this can be found here.

Implementing Secrets Management for Kubernetes with Bitwarden

Bitwarden Secrets Manager is a unique solution to efficiently and securely store and retrieve secrets. In order to create a more secure digital world, we need to make security convenient and effective.

Adding secrets to Kubernetes workflows is one way Bitwarden is helping make security fast and effective. Using the Secrets Manager Kubernetes Operator, which is deployed using Helm package manager, secrets can be stored and retrieved from Secrets Manager. In this tutorial, we will show you how this solution can be enabled and used with your Civo workflow.

Setting up Secrets Manager

To start, we will need to open the Bitwarden Secrets Manager on the web app. Use the product switcher in the bottom left corner of the web app to navigate to Secrets Manager if you have not yet used Secrets Manager.

From the Secrets Manager web app, create a new project, which will be the primary way of grouping secrets and assigning access later.

Setting up Secrets Manager

Step 1: Select Projects from the navigation bar and create a New project using the dropdown menu.

Step 2: Next, create a secret by selecting Secrets on the navigation. Create a new secret to store your needed access information, such as a cluster access token.

Step 3: Create a Machine Account by navigating to Machine Accounts on the navigation menu. Create a machine account and generate a new access token. This token will be used by your machine to allow Bitwarden Secrets Manager to retrieve secrets from the project.

Step 4: Return to the project that was created and assign the following items to the project:

  • The created secrets
  • Assign users to the project
  • Machine account created

Setting up Civo

In this example we will create a new cluster with one medium node and then save and merge our kubeconfig. For more information, you can refer to the Civo documentation.

Create a cluster: 

civo k3s create -n 1 -s g4s.kube.medium --merge --save --wait

Save the Civo API key in Secrets Manager Once the Civo CLI has been installed, you will be required to retrieve a Civo API key. This can be securely stored in secrets manager. Retrieve the API key from your Civo Dashboard. Once the API key has been saved, copy the value and Run:

For additional information on Chart setup and adding the repository to Helm, see https://bitwarden.com/help/secrets-manager-kubernetes-operator/#add-the-repository-to-helm.

Create values:

helm show values bitwarden/sm-operator --devel > my-values.yaml

You may wish to edit fields:

# How often the secrets synchronize in seconds. Minimum value is 180.
bwSecretManagerResreshInterval: 300

# Cloud region for sync
cloudRegion: US

After changes have been made, upgrade the release to a new chart by running:

helm upgrade sm-operator bitwarden/sm-operator -i --debug -n sm-operator-system --create-namespace --values my-values.yaml --devel

Return to the Secrets Manager web vault and copy the secret IDs and Machine account access token.

We will deploy a test app and therefore create a test namespace for this and the secrets:

kubectl create namespace test

kubectl create secret generic bw-auth-token -n test --from-literal=token="<your_machine_token_here>"

Create a secret

If using the following example, be sure to replace the values for your_org_id_here and your_secret_id_here. These values can be retrieved from the Secrets Manager web vault. 
kubectl apply -f - <<EOF
apiVersion: k8s.bitwarden.com/v1
kind: BitwardenSecret
metadata:
  namespace: test
  labels:
    app.kubernetes.io/name: bitwardensecret
    app.kubernetes.io/instance: bitwardensecret-sample
    app.kubernetes.io/part-of: sm-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: sm-operator
  name: bitwardensecret-sample
spec:
  organizationId: "<your_org_id_here>"
  secretName: bw-sample-secret
  map:
    - bwSecretId: <your_secret_id_here>
      secretKeyName: test__secret__1
    - bwSecretId: <your_secret_id_here>
      secretKeyName: test__secret__2
  authToken:
    secretName: bw-auth-token
    secretKey: token

EOF

Check that the secrets and operator are working in K8s:

kubectl get bitwardensecrets -n test                                                                                    
NAME                     AGE
bitwardensecret-sample   27m

Check the health of the operator:

kubectl describe bitwardensecrets -n test

Check the sync status:

Message:                  Completed sync for test/bitwardensecret-sample
    Reason:                   ReconciliationComplete
    Status:                   True
    Type:                     SuccessfulSync
  Last Successful Sync Time:  2024-09-25T10:36:12Z

Check secrets:

kubectl get secrets -n test                                                                                                
NAME               TYPE     DATA   AGE
bw-auth-token      Opaque   1      7m47s
bw-sample-secret   Opaque   2      6m56s

More secret information:

kubectl describe secrets bw-sample-secret -n test                                                                            
Name:         bw-sample-secret
Namespace:    test
Labels:       k8s.bitwarden.com/bw-secret=a9468555-5e5e-4621-93a1-eeaf028862bd
Annotations:  k8s.bitwarden.com/custom-map:
                [
                  {
                    "bwSecretId": "<your_secret_id_here>",
                    "secretKeyName": "test__secret__1"
                  },
                  {
                    "bwSecretId": "<your_secret_id_here>",
                    "secretKeyName": "test__secret__2"
                  }
                ]
              k8s.bitwarden.com/sync-time: 2024-09-25T10:36:12.922941793Z

Type:  Opaque

Data
====
test__secret__1:  10 bytes
test__secret__2:  12 bytes

Check the secret contents:

kubectl get secret bw-sample-secret -n test -o jsonpath="{.data}" | jq 'to_entries[] | "\(.key): \(.value | @base64d)"'

"test__secret__1: <your_secret_shown_here>"
"test__secret__2: <your_secret_shown_here>"

Test app to deploy a Bitwarden Secret

We can create and use Bitwarden secrets with the operator. The Kubernetes secret belongs to a namespace and will be injected with the data that the Secrets Manager machine account has access to.

Create test app:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: test
  labels:
    app: my-deployment
spec:
  selector:
    matchLabels:
      app: my-deployment
  template:
    metadata:
      labels:
        app: my-deployment
    spec:
      containers:
      - name: my-deployment
        image: nginx
        envFrom:
        - secretRef:
            name: bw-sample-secret
      imagePullSecrets:
      - name: bw-sample-secret

EOF

Check that these secrets have been injected into the app:

kubectl get pods -n test

Find the test pod and copy the name:

my-deployment-7ff457d764-qsp5m   1/1     Running   0          15m

View the env vars of the pod:

kubectl exec -it my-deployment-7ff457d764-qsp5m -n test -- env

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=my-deployment-7ff457d764-g82dx
NGINX_VERSION=1.27.1
NJS_VERSION=0.8.5
NJS_RELEASE=1~bookworm
PKG_RELEASE=1~bookworm
DYNPKG_RELEASE=2~bookworm
test__secret__2=<your_secret_shown_here>"
test__secret__1=<your_secret_shown_here>"
KUBERNETES_SERVICE_HOST=10.43.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
TERM=xterm
HOME=/root

If we change the secret value in Secrets Manager, we can restart the deployment and then check the value has changed:

kubectl get secret bw-sample-secret -n test -o jsonpath="{.data}" | jq 'to_entries[] | "\(.key): \(.value | @base64d)"'     
"test__secret__1: <your_new_secret_vaule_shown_here>"
"test__secret__2: <your_new_secret_vaule_shown_here>"

You will see the value has changed in the secret, but not yet in the pod, until it is restarted and it grabs the new values:

kubectl exec -it my-deployment-7ff457d764-p2fcl -n test -- env                                                               
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=my-deployment-7ff457d764-p2fcl
NGINX_VERSION=1.27.1
NJS_VERSION=0.8.5
NJS_RELEASE=1~bookworm
PKG_RELEASE=1~bookworm
DYNPKG_RELEASE=2~bookworm
test__secret__2=<your_new_secret_vaule_shown_here>"
test__secret__1=<your_new_secret_vaule_shown_here>"
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
KUBERNETES_SERVICE_HOST=10.43.0.1
TERM=xterm
HOME=/root

Summary

In this tutorial we demonstrated using the Secrets Manager Helm Operator to deploy a secret stored in Secrets Manager. This setup has allowed us to store and retrieve the secrets for use in any Kubernetes project.

Overview of steps completed:

  • Setup Secrets Manager from the UI to include: Project, Secret, and Machine Account.
  • Setup Kubernetes cluster using Civo’s CLI.
  • Setup the Bitwarden Helm repository and created a custom configuration file.
  • Created a BitwardenSecret object to be deployed by the operator.