Backup and Restore

For production clusters, regular maintenance should include routine backup operations to ensure data integrity and reduce the risk of data loss due to unexpected events. Backup operations should include the cluster state, application state, and the running configuration of both stateless and stateful applications in the cluster.

DKP stores all data as CRDs in the Kubernetes API and you can back up and restore it using the following procedure.

Velero

DKP provides Velero by default, to support backup and restore operations for your Kubernetes clusters and persistent volumes.

MinIO

For on-premises deployments, DKP deploys Velero integrated with MinIO, operating inside the same cluster.

For production use-cases, D2iQ advises to provide an external storage class to use with MinIO.

You can customize your Velero instance in two ways: You can add the values below to the configuration file when installing DKP, or you can apply the configuration after the cluster is configured by running kubectl apply -f.

To specify an external storageClass for the MinIO instances, create a file called velero-overrides.yaml with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: velero-overrides
  namespace: kommander
data:
  values.yaml: |
     minio:
      persistence:
         storageClass: <external storage class name>

CODE

Amazon S3

You can also store your backups in Amazon S3. To do so, create a file called velero-overrides.yaml with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: velero-overrides
  namespace: kommander
data:
  values.yaml: |
    minioBackend: false
    configuration:
      backupStorageLocation:
        # `name:` must be empty
        bucket: <BUCKET_NAME>
        config:
          region: <AWS_REGION> # such as us-west-2
          s3ForcePathStyle: "false"
          insecureSkipTLSVerify: "false"
          s3Url: ""
          # profile should be set to the AWS profile name mentioned in the secretContents below
          profile: default
    credentials:
      # With the proper IAM permissions with access to the S3 bucket,
      # you can attach the EC2 instances using the IAM Role, OR fill in `existingSecret` OR `secretContents` below.
      #
      # Name of a pre-existing secret (if any) in the Velero namespace
      # that should be used to get IAM account credentials.
      existingSecret:
      # The key must be named `cloud`, and the value corresponds to the entire content of your IAM credentials file.
      # For more information, consult the documentation for the velero plugin for AWS at:
      # [AWS] https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
      secretContents: 
        # cloud: |
        #   [default]
        #   aws_access_key_id=<REDACTED>
        #   aws_secret_access_key=<REDACTED>

CODE

Azure Blob

Prerequisites: Create Azure related assets such as storage account, blob containers, resource group, roles, service principals prior to continuing.

Prep your credentials-velero file with the values:

cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

CODE

Use the credentials-velero file to create the secret (in your case you named it as azure-bsl-credentials). Note that we used --from-env-file referencing the credentials-velero file.
```
kubectl create secret generic -n kommander azure-bsl-credentials --from-env-file="credentials-velero" --kubeconfig=${CLUSTER_NAME}.conf
```
CODE

Create the backup storage location via Velero CLI

velero backup-location create -n kommander azure \
--provider azure \
--bucket $BLOB_CONTAINER \
--config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID \
--credential=azure-bsl-credentials=azure --kubeconfig=${CLUSTER_NAME}.conf

BASH

Check to make sure the Azure backup location is created:

velero backup-location get -n kommander --kubeconfig=${CLUSTER_NAME}.conf

BASH

Configure DKP to load the plugins and to include the secret:
NOTE: This process has been tested to work with AWS v1.1.0 and Azure v1.1.2. Newer versions of these plugins can be used, but have not been tested by D2iQ.

:
velero:
    values: |
      minioBackend: false
      initContainers:
        - name: initialize-velero
          image: mesosphere/kubeaddons-addon-initializer:v0.5.5
          args: ["velero"]
          env:
            - name: "MINIO_INGRESS_NAMESPACE"
              value: kommander
            - name: "MINIO_INGRESS_SERVICE_NAME"
              value: kommander-traefik
            - name: "VELERO_NAMESPACE"
              value: kommander
            - name: "VELERO_MINIO_FALLBACK_SECRET_NAME"
              value: "velero-d2iq-credentials"
        - name: velero-plugin-for-aws
          image: velero/velero-plugin-for-aws:v1.1.0
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /target
              name: plugins
        - name: velero-plugin-for-microsoft-azure
          image: velero/velero-plugin-for-microsoft-azure:v1.1.2
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /target
              name: plugins
      credentials:
          extraSecretRef: azure-bsl-credentials
:

YAML

Check the Helm releases that the new Velero configuration applied/loaded:
```
kubectl get hr -n kommander --kubeconfig=${CLUSTER_NAME}.conf
```
CODE

Make sure that the Velero pod is running:

kubectl get pods -A --kubeconfig=${CLUSTER_NAME}.conf |grep velero

CODE

And connect to it via exec:

kubectl -n kommander exec -it velero-<hash> --kubeconfig=${CLUSTER_NAME}.conf bash

CODE

Check the plugin and the env variables. The env variable should contain the contents of the credentials-velero file.
```
cd plugins
ls -l
env |grep AZURE
```
BASH

Create a test backup:

velero backup create azure-velero-testbackup -n kommander --kubeconfig=${CLUSTER_NAME}.conf --storage-location azure

CODE