Securely managing secrets in a GitOps workflow using SealedSecrets

For security reasons, Kubernetes secrets are usually the only resource that cannot be managed with a GitOps workflow. Instead of managing secrets outside of GitOps and having to use a third-party tool like Vault, SealedSecrets provides a way to keep all the advantages of using a GitOps workflow while avoiding exposing secrets. SealedSecrets is composed of two main components:

  • A CLI (Kubeseal) to encrypt secrets.

  • A cluster-side controller to decrypt the sealed secrets into regular Kubernetes secrets. Only this controller can decrypt sealed secrets, not even the original author.

This tutorial describes how to install these two components, configure the controller, and add or remove sealed secrets.

Set up

These instructions are used as an example. For instructions on the latest release, see the release page. For full documentation on SealedSecrets, see the GitHub repo.

Install Kubeseal CLI to Encrypt Your Secrets

  • On MacOS

    brew install kubeseal
    CODE

  • On Linux

    wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.1/kubeseal-0.18.1-linux-amd64.tar.gz -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
    CODE

Install the SealedSecrets Controller on Your Cluster

This controller will be able to decrypt SealedSecrets and create Kubernetes secrets.

  1. Create the controller:

    kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.1/controller.yaml
    CODE

  2. Fetch the certificate that you will use to encrypt your secrets into sealed secrets:

    kubeseal --fetch-cert > mycert.pem
    CODE

  3. Commit mycert.pem to your git repo.

Add a Secret

Secrets can be securely added to Git using sealed secrets:

  1. Export the project namespace that you are using for your GitOps repository

    export PROJECT_NAMESPACE=<your-project-with-gitops>
    CODE

  2. Create a Kubernetes secret and pipe it into kubeseal using the certificate mycert.pem that you fetched from the controller in the setup:

    echo '---' >> secrets.yaml
kubectl create secret -n ${PROJECT_NAMESPACE} generic mysecret --dry-run=client -o yaml --from-literal=my-secret=value | \
    kubeseal --format yaml --cert mycert.pem >> secrets.yaml
    CODE

  3. Go to the end of secrets.yaml where you just added your new sealed secret. Remove any “creationTimestamp” fields from the YAML.

  4. Apply the secrets.yaml file to your namespace. If you do not have permission, commit your changes to the repo and let FluxCD apply the changes for you.

    kubectl apply -f secrets.yaml
    CODE

  5. The sealed secret controller will then decrypt the sealed secret and generate a Kubernetes secret from it. Your secret got successfully created by running:

    kubectl get secret mysecret -n ${PROJECT_NAMESPACE} -o yaml
    CODE

  6. If your sealed secret got created successfully but did not generate the matching secret, look at the logs of the controller:

    kubectl logs -l=name=sealed-secrets-controller -n kube-system
    CODE

  7. Commit secrets.yaml to your repo if you have not already done so in step 3.

Remove a Secret

  1. Following the same example from above in “Adding a secret”, now remove the manifest for mysecret in secrets.yaml and commit those changes to the repo.

  2. Delete the SealedSecret in the cluster:

    kubectl delete SealedSecret -n ${PROJECT_NAMESPACE} mysecret
    CODE

  3. Delete the secret itself:

    kubectl delete secret -n ${PROJECT_NAMESPACE} mysecret
    CODE

Rotate the Controller’s Sealing Key

For added security, it is a good practice to rotate the key the controller uses to decrypt sealed secrets. By default, the controller generates a new key every 30 days. When this happens, you need to update the certificate you use to create sealed secrets by fetching the latest one:

   kubeseal --fetch-cert > mycert.pem
CODE

Do not forget to commit it back to the repo!

In a disaster case, let’s say your cluster gets destroyed, you would lose all your sealing keys, so you would not be able to recreate all the secrets from the sealed secrets in your GitOps repo. For this reason, you might want to back up the sealing keys. To do this every time a new sealing key is generated, run:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-key
CODE

Then store sealing-key with the others in a safe location such as OneLogin Notes or Vault. To restore from a backup after a disaster, recreate all of the sealing keys with kubectl apply -f sealing-key1 sealing-key2 ... before starting the controller. If the controller was already started, restart it: kubectl delete pod -n kube-system -l name=sealed-secrets-controller

To disable sealing key rotation For example, configure the controller’s command in the pod template with --key-renew-period=0. See the following YAML file.

Pod Template:
  Labels:           name=sealed-secrets-controller
  Service Account:  sealed-secrets-controller
  Containers:
   sealed-secrets-controller:
    Image:      docker.io/bitnami/sealed-secrets-controller:v0.18.1
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:    controller
      --key-renew-period=0
CODE

If required, edit the controller’s manifest with:

kubectl edit deployment.apps/sealed-secrets-controller -n kube-system
CODE