We often talk about Infrastructure as Code (IaC) which means every aspect of the infrastructure should be in a code and reproducible.

In the Kubernetes ecosystem, we use various projects to keep manifests in code, whether in plain YAML, Kustomize, or Helm. However, one component that is often manually created and not stored as code is the Secret.

Note: Secrets in Kubernetes are base64 encoded, not encrypted, offering no real security guarantee. Base64 encoding obfuscates data but offers no security, whereas encryption provides robust protection against unauthorized access.

Let’s take a quick example.

$ cat secret.yaml
apiVersion: v1
data:
  foo: YmFy # This is a base64 encoded bar
kind: Secret
metadata:
  name: mysecret
  namespace: default

Now anyone with access to this secret can just run base64 decode and obtain the secret value:

$ echo "YmFy" | base64 -d
bar

This leads to manual management of secrets, and we don’t achieve 100% Infrastructure as Code.

This tutorial will guide you on how to enhance secure Kubernetes management with Sealed Secrets. Additionally, we will dive into installation, usage, and best practices for a robust Infrastructure-as-Code approach.

Introducing Sealed Secrets

Sealed Secrets is an open-source project started at Bitnami and is used to encrypt Kubernetes secrets. Once encrypted, you can store this encrypted secret in your Git repository safely. It allows DevOps practices without exposing sensitive data.

Only the Sealed Secrets controller can decrypt these encrypted secrets.

How do Sealed Secrets work?

By following these steps, users can ensure the secure management and deployment of their Kubernetes secrets.

  • Encryption: The user encrypts Kubernetes secrets using a Sealed Secrets controller.
  • Storage: The Sealed Secrets are safely stored in Git.
  • Deployment: The user deploys the Sealed Secrets to the Kubernetes cluster.
  • Decryption and Creation: The Sealed Secrets controller decrypts the sealed secrets and creates a Kubernetes secrets resource on the cluster.

    The Sealed Secrets controller in the Sealed Secrets ecosystem is responsible for watching for Sealed Secret custom resources. When it detects one, it decrypts the enclosed secret using its private key and then creates a standard Kubernetes Secret. Its significance lies in its ability to manage and decrypt Sealed Secrets securely, ensuring that only the cluster with the corresponding private key can access the original secret content.

How do Sealed Secrets work?

Prerequisites

Before we begin, there are some prerequisites you’ll need to have in place:

Note: This tutorial assumes a basic understanding of Kubernetes concepts.

Installing Sealed Secrets through Civo Marketplace

There are two ways to install the Sealed Secrets through Civo Marketplace: the UI or the CLI. This section will explore how to install Sealed Secrets using both methods.

Civo Marketplace UI

After successfully creating the Kubernetes cluster, log in to the Civo dashboard and go to "Kubernetes" > "Kubernetes Cluster" > "Marketplace," as shown in the screenshot below:

Installing Sealed Secrets through Civo Marketplace

In the Marketplace section, select the “Architecture” tab, and click “Sealed Secrets”. Click “Install Apps” from here to install the Sealed Secrets application inside the cluster.

Note: At the time of writing this blog, the latest version is 0.24.0. Always check for the latest version.

For more information, check out the Git Repository here.

Civo CLI

If you want to install the controller separately, you can run the following:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.23.1/controller.yaml

The following resources will be created in the kube-system namespace:

  • serviceaccount sealed-secrets-controller
  • Deployment sealed-secrets-controller
  • CustomResourceDefinition sealedsecrets.bitnami.com
  • Service sealed-secrets-controller
  • RBAC resources

Installing Kubeseal

Once you’ve installed Sealed Secrets, we will also need something on the client side to encrypt the secrets and for other operations. For this purpose, the project has a kubeseal command-line utility which we will install. It's used to encrypt Kubernetes Secrets, turning them into "Sealed Secrets" which can be safely stored in version control systems such as Git.

Note: For other platforms, you can download kubeseal from the release page here

Installing Kubeseal on Mac

To install Kubeseal on Mac, you can run the following command:

brew install kubeseal

Installing Kubeseal on Linux

To install Kubeseal on Linux, you can run the following command:

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-v0.24.0-linux-amd64.tar.gz
tar -xvzf kubeseal-v0.24.0-linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
Note: At the time of writing this tutorial, the latest version is 0.24.0. Always check for the latest version.

Till now, we have installed a sealed secrets controller and kubeseal CLI. Before we start encrypting secrets, we need to understand the concept of scopes.

Understanding scopes

The scope is nothing but the context or visibility of a sealed secret within a Kubernetes cluster. The scope of a Sealed Secret relates to where and how the Sealed Secret can be decrypted and used within your cluster.

There are 3 types of scopes in Sealed Secrets:

Scope Type Description
strict The name and namespace of the secret are included in the encrypted data. Therefore, you must seal the secret using the same name and namespace.
namespace-wide You can freely rename the sealed secret within a given namespace.
cluster-wide You have the flexibility to unseal the secret using any name and in any namespace.


By default, strict scope is selected unless you pass the --scope flag to kubeseal CLI with a different value.

Sealed Secrets Use Case

In this section, we will walk through a practical example of converting a secret from a file to a Sealed Secret. This step-by-step guide will provide a clearer understanding of the use of Sealed Secrets in Kubernetes.

Overview

Before diving into the commands, let's understand the workflow. We will take a secret.yaml file, convert it into a Sealed Secret, and then apply this Sealed Secret to a Kubernetes cluster. The Sealed Secret will then be unsealed by the controller in the cluster, creating a usable secret.

Initial Setup

Ensure you have a secret.yaml file as used in earlier sections. This file contains the secret data in base64 encoded format.

Converting Secret to Sealed Secret

Step 1: Run the kubeseal command

Use the kubeseal command to convert the secret.yaml file to a Sealed Secret. The -f flag specifies the input file, and the -w flag writes the output to sealedsecret.yaml.

kubeseal -f secret.yaml -w sealedsecret.yaml

This command will generate a new file, sealedsecret.yaml, containing the encrypted secret.

Step 2: Review the Sealed Secret file

Open the sealedsecret.yaml file to review the contents. It should look similar to the example below:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: default
spec:
  encryptedData:
    foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

This file is safe to store in version control systems as it is encrypted.

Applying the Sealed Secret

Step 1: Apply the Sealed Secret

Use the kubectl apply command to apply the sealedsecret.yaml file to your Kubernetes cluster.

kubectl apply -f sealedsecret.yaml

This command will create a Sealed Secret in the cluster.

Step 2: Verify the Sealed Secret

Confirm the creation of the Sealed Secret and the corresponding secret by running the following commands:

kubectl get sealedsecrets
kubectl get secrets

These commands will list the Sealed Secrets and secrets in the cluster, respectively.

Unsealing the Secret

Step 1: Describe the Sealed Secret

Use the kubectl describe command to view the details of the Sealed Secret.

kubectl describe sealedsecrets mysecret
…

In the events section, you should see a message indicating that the Sealed Secret was unsealed successfully:

…
Events:
  Type    Reason    Age   From            Message
  ----    ------    ----  ----            -------
  Normal  Unsealed  67s   sealed-secrets  SealedSecret unsealed successfully

This message confirms that the Sealed Secret has been successfully converted back into a secret in the cluster.

Creating and Converting a Secret Simultaneously

In some scenarios, you may want to create a secret and convert it to a Sealed Secret in one go. Follow the steps below to achieve this:

Step 1: Create a Secret

Use the kubectl create secret command to create a secret and output it to a secret.yaml file. The --dry-run option ensures the command doesn’t execute but returns the output as YAML.

kubectl create secret generic mysecret –from-literal=foo=bar –dry-run -o yaml > secret.yaml

Step 2: Convert the Secret to a Sealed Secret

Use the kubeseal command to convert the secret.yaml file to a sealedsecret.yaml file.

kubeseal -f secret.yaml -o sealedsecret.yaml

Step 3: Apply the Sealed Secret

Apply the sealedsecret.yaml to your Kubernetes cluster.

kubectl apply -f sealedsecret.yaml

This sequence of commands creates a secret and converts it to a Sealed Secret efficiently.

Managing Existing Secrets

If you are installing Sealed Secrets on an existing cluster with pre-existing secrets, follow the steps below:

Step 1: Annotate the existing Secret

sealedsecrets.bitnami.com/managed: "true"

This annotation marks the secret for management as a Sealed Secret.

Updating Keys in Existing Sealed Secrets

In instances where you need to update or add new values to an existing Sealed Secret, follow the steps below:

Step 1: Append New Secret Value

Use the following command to append a new value to the existing Sealed Secret without updating the entire secret.

echo -n baz | kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o yaml | kubeseal --merge-into sealedsecret.yaml -o yaml

This command adds a new value to the sealedsecret.yaml file.

Validating Sealed Secrets Locally

Before applying Sealed Secrets to the cluster, it’s crucial to validate the Sealed Secrets file to ensure its correctness.

Step 1: Validate the Sealed Secrets File

Use the kubeseal --validate command to validate the sealedsecret.yaml file.

cat sealedsecret.yaml | kubeseal –validate

If the Sealed Secret is not valid, the command will return an error message:

cat sealedsecret.yaml | kubeseal --validate
error: unable to decrypt sealed secret

This validation step helps in avoiding the application of invalid Sealed Secrets to the cluster.

Key Considerations

If you run a sealed secrets controller in a namespace other than kube-system, you will have to pass the --controller-namespace flag every time you run the kubeseal commands. Deploying Sealed Secrets outside the kube-system namespace can enhance security by isolating it from critical cluster components and offer more granular control through namespace-specific configurations and RBAC policies.

Alternatively, you can set the environment variable SEALEDSECRETSCONTROLLER_NAMESPACE

Summary

Through this tutorial, we’ve learned how to leverage Sealed Secrets to securely store secrets in Git.

Sealed Secrets has become essential software to complete the story of Infrastructure as a code and GitOps.

If you have any issues or doubts, feel free to reach out to me on Twitter.

If you want to know more about using Sealed Secrets, take a look at these resources given here: