Store secrets in GitOps repository using SealedSecrets

Securely managing secrets in a GitOps workflow using SealedSecrets

For security reasons, Kubernetes secrets are usually the only resource that cannot be managed with a GitOps workflow. Instead of managing secrets outside of GitOps and having to use a third-party tool like Vault, SealedSecrets provides a way to keep all the advantages of using a GitOps workflow while avoiding exposing secrets. SealedSecrets is composed of two main components:

• A CLI (Kubeseal) to encrypt secrets.

• A cluster-side controller to decrypt the sealed secrets into regular Kubernetes secrets. Only this controller can decrypt sealed secrets, not even the original author.

This tutorial describes how to install these two components, configure the controller, and add or remove sealed secrets.

Set up

These instructions are used as an example. For instructions on the latest release, see the release page. For full documentation on SealedSecrets, see the GitHub repo.

Install Kubeseal CLI to Encrypt Your Secrets

• On MacOS

  CODE

  brew install kubeseal

• On Linux

  CODE

  wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.1/kubeseal-0.18.1-linux-amd64.tar.gz -O kubeseal
  sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Install the SealedSecrets Controller on Your Cluster

This controller will be able to decrypt SealedSecrets and create Kubernetes secrets.

1. Create the controller:

   CODE

   kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.1/controller.yaml

2. Fetch the certificate that you will use to encrypt your secrets into sealed secrets:

   CODE

   kubeseal --fetch-cert > mycert.pem

3. Commit mycert.pem to your git repo.

Add a Secret

Secrets can be securely added to Git using sealed secrets:

1. Export the project namespace that you are using for your GitOps repository

   CODE

   export PROJECT_NAMESPACE=<your-project-with-gitops>

2. Create a Kubernetes secret and pipe it into kubeseal using the certificate mycert.pem that you fetched from the controller in the setup:

   CODE

   echo '---' >> secrets.yaml
   kubectl create secret -n ${PROJECT_NAMESPACE} generic mysecret --dry-run=client -o yaml --from-literal=my-secret=value | \
       kubeseal --format yaml --cert mycert.pem >> secrets.yaml

3. Go to the end of secrets.yaml where you just added your new sealed secret. Remove any “creationTimestamp” fields from the YAML.

4. Apply the secrets.yaml file to your namespace. If you do not have permission, commit your changes to the repo and let FluxCD apply the changes for you.

   CODE

   kubectl apply -f secrets.yaml

5. The sealed secret controller will then decrypt the sealed secret and generate a Kubernetes secret from it. Your secret got successfully created by running:

   CODE

   kubectl get secret mysecret -n ${PROJECT_NAMESPACE} -o yaml

6. If your sealed secret got created successfully but did not generate the matching secret, look at the logs of the controller:

   CODE

   kubectl logs -l=name=sealed-secrets-controller -n kube-system

7. Commit secrets.yaml to your repo if you have not already done so in step 3.

Remove a Secret

1. Following the same example from above in “Adding a secret”, now remove the manifest for mysecret in secrets.yaml and commit those changes to the repo.

2. Delete the SealedSecret in the cluster:

   CODE

   kubectl delete SealedSecret -n ${PROJECT_NAMESPACE} mysecret

3. Delete the secret itself:

   CODE

   kubectl delete secret -n ${PROJECT_NAMESPACE} mysecret

Rotate the Controller’s Sealing Key

For added security, it is a good practice to rotate the key the controller uses to decrypt sealed secrets. By default, the controller generates a new key every 30 days. When this happens, you need to update the certificate you use to create sealed secrets by fetching the latest one:

   kubeseal --fetch-cert > mycert.pem

Do not forget to commit it back to the repo!

In a disaster case, let’s say your cluster gets destroyed, you would lose all your sealing keys, so you would not be able to recreate all the secrets from the sealed secrets in your GitOps repo. For this reason, you might want to back up the sealing keys. To do this every time a new sealing key is generated, run:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealing-key

Then store sealing-key with the others in a safe location such as OneLogin Notes or Vault. To restore from a backup after a disaster, recreate all of the sealing keys with kubectl apply -f sealing-key1 sealing-key2 ... before starting the controller. If the controller was already started, restart it: kubectl delete pod -n kube-system -l name=sealed-secrets-controller

To disable sealing key rotation For example, configure the controller’s command in the pod template with --key-renew-period=0. See the following YAML file.

Pod Template:
  Labels:           name=sealed-secrets-controller
  Service Account:  sealed-secrets-controller
  Containers:
   sealed-secrets-controller:
    Image:      docker.io/bitnami/sealed-secrets-controller:v0.18.1
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:    controller
      --key-renew-period=0

If required, edit the controller’s manifest with:

kubectl edit deployment.apps/sealed-secrets-controller -n kube-system