Configure Velero to use AWS S3

See the official docs for details on how to use IAM roles instead of static credentials

  1. Create a file named cloud containing your static AWS credentials

    cat << EOF > cloud
      [default]
      aws_access_key_id=<REDACTED>
      aws_secret_access_key=<REDACTED>
    EOF
    CODE
  2. Create a secret on the cluster you are configuring Velero on (alternatively you can set them directly in the ConfigMap)

    kubectl create secret generic velero-aws-credentials -n ${WORKSPACE_NAMESPACE} --from-file cloud
    CODE

Specific instructions must be followed depending on if you want to deploy a Management or Attached/Managed Cluster.

Configuring Velero on Attached or Managed Clusters

  1. Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace

    export WORKSPACE_NAMESPACE=<your_workspace_namespace>
    CODE
  2. Create a ConfigMap for the Velero configuration

    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: ${WORKSPACE_NAMESPACE}
      name: velero-overrides
    data:
      values.yaml: |
        configuration:
          backupStorageLocation:
            bucket: <BUCKET_NAME>
            config:
              region: <AWS_REGION> # such as us-west-2
              s3ForcePathStyle: "false"
              insecureSkipTLSVerify: "false"
              s3Url: ""
              # profile should be set to the AWS profile name mentioned in the secret
              profile: default
        credentials:
          # With the proper IAM permissions with access to the S3 bucket,
          # you can attach the EC2 instances using the IAM Role, OR fill in "existingSecret" OR "secretContents" below.
          #
          # Name of a pre-existing secret (if any) in the Velero namespace
          # that should be used to get IAM account credentials.
          existingSecret: velero-aws-credentials
          # The key must be named "cloud", and the value corresponds to the entire content of your IAM credentials file.
          # For more information, consult the documentation for the velero plugin for AWS at:
          # [AWS] https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
          secretContents: 
            # cloud: |
            #   [default]
            #   aws_access_key_id=<REDACTED>
            #   aws_secret_access_key=<REDACTED>
    EOF
    CODE
  3. Patch the Velero AppDeployment by adding the configOverrides value. This applies the ConfigMap to your instance after the cluster has been configured.

    cat << EOF | kubectl -n ${WORKSPACE_NAMESPACE} patch appdeployment velero --type="merge" --patch-file=/dev/stdin
    spec:
      configOverrides:
        name: velero-overrides
    EOF
    CODE
  4. Shortly after patching the AppDeployment, you will see the ConfigMap on the HelmRelease object

    kubectl wait --for=jsonpath='{.spec.valuesFrom[1].name}'=velero-overrides HelmRelease/velero -n ${WORKSPACE_NAMESPACE}
    CODE
  5. Create the backup storage location:

    velero backup-location create -n ${WORKSPACE_NAMESPACE} <bsl-name> \
      --provider aws \
      --bucket $BUCKET \
      --config region=$REGION \
      --credential=velero-aws-credentials=aws
    CODE
  6. Check that the backup storage location is Available and referencing the correct S3 bucket

    kubectl get backupstoragelocations -n ${WORKSPACE_NAMESPACE} -oyaml
    CODE

If the BackupStorageLocation is not Available, you can view any error events by using: kubectl describe backupstoragelocations -n ${WORKSPACE_NAMESPACE}

Configuring Velero on the Management Cluster

  1. Output the Kommander configuration to kommander.yaml

    dkp install kommander -o yaml --init > kommander.yaml
    CODE
    1. Add the Velero configuration to the apps.velero.values section:
      NOTE: This process has been tested to work with plugins for AWS v1.1.0. Newer versions of these plugins can be used, but have not been tested by D2iQ.

      ...
        velero:
          values: |
            configuration:
            backupStorageLocation:
              bucket: <BUCKET_NAME>
              config:
                region: <AWS_REGION> # such as us-west-2
                s3ForcePathStyle: "false"
                insecureSkipTLSVerify: "false"
                s3Url: ""
                # profile should be set to the AWS profile name mentioned in the secret
                profile: default
          credentials:
            # With the proper IAM permissions with access to the S3 bucket,
            # you can attach the EC2 instances using the IAM Role, OR fill in "existingSecret" OR "secretContents" below.
            #
            # Name of a pre-existing secret (if any) in the Velero namespace
            # that should be used to get IAM account credentials.
            existingSecret: velero-aws-credentials
            # The key must be named "cloud", and the value corresponds to the entire content of your IAM credentials file.
            # For more information, consult the documentation for the velero plugin for AWS at:
            # [AWS] https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
            secretContents: 
              # cloud: |
              #   [default]
              #   aws_access_key_id=<REDACTED>
              #   aws_secret_access_key=<REDACTED>
      ...
      YAML
  2. Run dkp install kommander using the kommander.yaml configuration file:

    dkp install kommander --installer-config kommander.yaml --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  3. After running dkp install kommander see the ConfigMap on the HelmRelease object

    kubectl wait --for=jsonpath='{.spec.valuesFrom[1].name}'=velero-overrides HelmRelease/velero -n kommander
    CODE
  4. Check that the backup storage location is Available and referencing the correct S3 bucket

    kubectl get backupstoragelocations -n kommander -oyaml
    CODE

Configure Velero to use Azure Blob Storage

Prerequisites: Create Azure related assets such as storage account, blob containers, resource group, roles, service principals prior to continuing.

  1. Confirm that you have created your storage account, and your blob container using these instructions.

  2. Prep your credentials-velero file with the values. You will need to use the same credentials that you created when creating the cluster. Please note that these credentials should not be Base64 encoded, as Velero will not read them properly. Export the AZURE_BACKUP_RESOURCE_GROUP that you created in the last step to be the AZURE_RESOURCE_GROUP in this step (in a later step, you will also use AZURE_BACKUP_RESOURCE_GROUP).

    cat << EOF > ./credentials-velero
    AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
    AZURE_TENANT_ID=${AZURE_TENANT_ID}
    AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
    AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
    AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
    AZURE_CLOUD_NAME=AzurePublicCloud
    EOF
    CODE
  3. Use the credentials-velero file to create the secret (in this case, it is named as azure-bsl-credentials). Note that we used --from-env-file referencing the credentials-velero file. If you are backing up the Management cluster, the namespace is kommander.

    kubectl create secret generic -n ${WORKSPACE_NAMESPACE} azure-bsl-credentials --from-file=azure="credentials-velero" --kubeconfig=${CLUSTER_NAME}.conf
    CODE

Configuring Velero on Attached or Managed Clusters

  1. Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace

    export WORKSPACE_NAMESPACE=<your_workspace_namespace>
    CODE
  2. Create a ConfigMap to apply Azure to the Velero configuration

    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: ${WORKSPACE_NAMESPACE}
      name: velero-overrides
    data:
      values: |
        initContainers:
          - name: velero-plugin-for-aws
            image: velero/velero-plugin-for-aws:v1.1.0
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - mountPath: /target
                name: plugins
          - name: velero-plugin-for-microsoft-azure
            image: velero/velero-plugin-for-microsoft-azure:v1.5.1
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - mountPath: /target
                name: plugins
        credentials:
          extraSecretRef: azure-bsl-credentials
    EOF
    CODE
  3. Patch the Velero AppDeployment by adding the configOverrides value. This applies the ConfigMap to your instance after the cluster has been configured.

    cat << EOF | kubectl -n ${WORKSPACE_NAMESPACE} patch appdeployment velero --type="merge" --patch-file=/dev/stdin
    spec:
      configOverrides:
        name: velero-overrides
    EOF
    CODE
  4. After patching the AppDeployment, you will see the ConfigMap on the HelmRelease object

    kubectl wait --for=jsonpath='{.spec.valuesFrom[1].name}'=velero-overrides HelmRelease/velero -n ${WORKSPACE_NAMESPACE}
    CODE
  5. Create the backup storage location via Velero CLI (note that this calls for the BLOB_CONTAINER and AZURE_STORAGE_ACCOUNT_ID variable that was used when creating the blob container in step 1, as well as the AZURE_BACKUP_SUBSCRIPTION_ID which will be the same as the AZURE_SUBSCRIPTION_ID set previously):

    velero backup-location create azure -n ${WORKSPACE_NAMESPACE} \
    --provider azure \
    --bucket ${BLOB_CONTAINER} \
    --config resourceGroup=${AZURE_BACKUP_RESOURCE_GROUP},storageAccount=${AZURE_STORAGE_ACCOUNT_ID},subscriptionId=${AZURE_BACKUP_SUBSCRIPTION_ID} \
    --credential=azure-bsl-credentials=azure --kubeconfig=${CLUSTER_NAME}.conf
    BASH
  6. Verify that the Azure backup location is created:

    velero backup-location get -n ${WORKSPACE_NAMESPACE} --kubeconfig=${CLUSTER_NAME}.conf
    BASH
  7. Check the Helm releases that the new Velero configuration has been applied:

    kubectl get helmrelease -n ${WORKSPACE_NAMESPACE} --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  8. Verify that the Velero pod is running:

    kubectl get pods -A --kubeconfig=${CLUSTER_NAME}.conf |grep velero
    CODE
  9. Create a test backup for Azure:

    velero backup create azure-velero-testbackup -n ${WORKSPACE_NAMESPACE} --kubeconfig=${CLUSTER_NAME}.conf --storage-location azure --snapshot-volumes=false
    CODE
  10. View your backup:

    velero backup describe azure-velero-testbackup
    CODE

Configuring Velero on the Management Cluster

  1. Create the backup storage location via Velero CLI (note that this calls for the BLOB_CONTAINER and AZURE_STORAGE_ACCOUNT_ID variable that was used when creating the blob container in step 1, as well as the AZURE_BACKUP_SUBSCRIPTION_ID which will be the same as the AZURE_SUBSCRIPTION_ID set earlier):

    velero backup-location create azure -n kommander \
    --provider azure \
    --bucket ${BLOB_CONTAINER} \
    --config resourceGroup=${AZURE_BACKUP_RESOURCE_GROUP},storageAccount=${AZURE_STORAGE_ACCOUNT_ID},subscriptionId=${AZURE_BACKUP_SUBSCRIPTION_ID} \
    --credential=azure-bsl-credentials=azure --kubeconfig=${CLUSTER_NAME}.conf
    BASH
  2. Verify that the Azure backup location is created:

    velero backup-location get -n kommander --kubeconfig=${CLUSTER_NAME}.conf
    BASH
  3. Output the Kommander configuration to kommander.yaml (or use your existing configuration file)

    dkp install kommander -o yaml --init > kommander.yaml
    CODE
    1. Configure DKP to load the plugins and to include the secret in the apps.velero section:
      NOTE: This process has been tested to work with plugins for AWS v1.1.0 and Azure v1.5.1. Newer versions of these plugins can be used, but have not been tested by D2iQ.

      ...
        velero:
          values: |
            initContainers:
              - name: velero-plugin-for-aws
                image: velero/velero-plugin-for-aws:v1.1.0
                imagePullPolicy: IfNotPresent
                volumeMounts:
                  - mountPath: /target
                    name: plugins
              - name: velero-plugin-for-microsoft-azure
                image: velero/velero-plugin-for-microsoft-azure:v1.5.1
                imagePullPolicy: IfNotPresent
                volumeMounts:
                  - mountPath: /target
                    name: plugins
            credentials:
              extraSecretRef: azure-bsl-credentials
      ...
      YAML
  4. Use the modified kommander.yaml configuration in install this Velero configuration:

    dkp install kommander --installer-config kommander.yaml --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  5. Check the Helm releases that the new Velero configuration applied/loaded:

    kubectl get helmrelease -n kommander --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  6. Ensure that the Velero pod is running:

    kubectl get pods -A --kubeconfig=${CLUSTER_NAME}.conf |grep velero
    CODE
  7. Create a test backup for Azure:

    velero backup create azure-velero-testbackup -n kommander --kubeconfig=${CLUSTER_NAME}.conf --storage-location azure --snapshot-volumes=false
    CODE
  8. View your backup:

    velero backup describe azure-velero-testbackup
    CODE

Configure Velero to use Google Cloud Buckets

You can also store your backups in Google Cloud/GCP.

See the official docs for details on how to use different types of authentication.

Prerequisites: Create GCP related assets such as GCS Bucket, GCP project, service accounts, and service account keys prior to continuing, and the velero, gcloud, and gsutil CLIs installed locally (gsutil is optional, you may buckets through the GCP web application).

  1. Confirm that you have created your storage account, and your bucket, using these instructions.

  2. Prep your credentials-velero file with the values, using the SERVICE_ACCOUNT_EMAIL you used to grant permissions to your bucket. This creates a credentials-velero file in your local directory.

    gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL
    CODE
  3. Use the credentials-velero file to create the secret (in this case, we named it bsl-credentials). Note that we used --from-env-file referencing the credentials-velero file. If you are backing up the Management cluster, the namespace is kommander.

    kubectl create secret generic -n ${WORKSPACE_NAMESPACE} bsl-credentials --from-file=gcp="credentials-velero" --kubeconfig=${CLUSTER_NAME}.conf
    CODE

Configuring Velero on Attached or Managed Clusters

  1. Set the WORKSPACE_NAMESPACE environment variable to the name of the workspace’s namespace

    export WORKSPACE_NAMESPACE=<your_workspace_namespace>
    CODE
  2. Create the backup storage location via Velero CLI (note that this calls for the BUCKET variable that was used when creating the bucket container in step 1:

    velero backup-location create gcp-backup -n ${WORKSPACE_NAMESPACE} \
      --provider gcp \
      --bucket $BUCKET \
      --credential=bsl-credentials=gcp
    CODE
  3. Verify that the GCP backup location is created:

    velero backup-location get -n ${WORKSPACE_NAMESPACE} --kubeconfig=${CLUSTER_NAME}.conf
    BASH
  4. Create a ConfigMap to apply GCP to the Velero configuration

    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: ${WORKSPACE_NAMESPACE}
      name: velero-overrides
    data:
      values: |
        initContainers:
          - name: velero-plugin-for-aws
            image: velero/velero-plugin-for-aws:v1.1.0
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - mountPath: /target
                name: plugins
          - name: velero-plugin-for-gcp
            image: velero/velero-plugin-for-gcp:v1.5.0
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - mountPath: /target
                name: plugins
        credentials:
          extraSecretRef: bsl-credentials
    EOF
    CODE
  5. Patch the Velero AppDeployment by adding the configOverrides value. This applies the ConfigMap in thisinstance after the cluster has been configured.

    cat << EOF | kubectl -n ${WORKSPACE_NAMESPACE} patch appdeployment velero --type="merge" --patch-file=/dev/stdin
    spec:
      configOverrides:
        name: velero-overrides
    EOF
    CODE
  6. After patching the AppDeployment, you will see the ConfigMap on the HelmRelease object:

    kubectl wait --for=jsonpath='{.spec.valuesFrom[1].name}'=velero-overrides HelmRelease/velero -n ${WORKSPACE_NAMESPACE}
    CODE
  7. Check the Helm releases that the new Velero configuration applied/loaded:

    kubectl get helmrelease -n ${WORKSPACE_NAMESPACE} --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  8. Ensure that the Velero pod is running:

    kubectl get pods -A --kubeconfig=${CLUSTER_NAME}.conf |grep velero
    CODE
  9. Create a test backup for GCP:

    velero backup create gcp-velero-testbackup -n ${WORKSPACE_NAMESPACE} --kubeconfig=${CLUSTER_NAME}.conf --storage-location gcp-backup
    CODE
  10. Please note: if your backup wasn’t created, Velero may have had an issue installing the plugin. If the plugin was not installed, run this command:

    velero plugin add velero/velero-plugin-for-gcp:v1.5.0 -n ${WORKSPACE_NAMESPACE}
    CODE

    And then confirm your backupstoragelocation was configured correctly

    kubectl get backupstoragelocations -n ${WORKSPACE_NAMESPACE}
    CODE

    If your backup storage location is “Available”, repeat step 9 and proceed to the rest of the setup

    NAME             PHASE       LAST VALIDATED   AGE   DEFAULT
    gcp-backup       Available   38s              60m   
    CODE
  11. View your backup:

    velero backup describe gcp-velero-testbackup
    CODE

Configuring Velero on the Management Cluster

  1. Create the backup storage location via Velero CLI (note that this calls for the BUCKET variable that was used when creating the bucket container in step 1):

    velero backup-location create gcp-backup -n kommander \
      --provider gcp \
      --bucket $BUCKET \
      --credential=bsl-credentials=gcp
    CODE
  2. Verify that the GCP backup location is created:

    velero backup-location get -n kommander --kubeconfig=${CLUSTER_NAME}.conf
    BASH
  3. Output the Kommander configuration to kommander.yaml (or use your existing configuration file)

    dkp install kommander -o yaml --init > kommander.yaml
    CODE
  4. Configure DKP to load the plugins and to include the secret under the apps.velero section:
    NOTE: This process has been tested to work with plugins for AWS v1.1.0 and GCP v1.5.0. Newer versions of these plugins can be used, but have not been tested by D2iQ.

    ...
      velero:
        values: |
          initContainers:
            - name: velero-plugin-for-aws
              image: velero/velero-plugin-for-aws:v1.1.0
              imagePullPolicy: IfNotPresent
              volumeMounts:
                - mountPath: /target
                  name: plugins
            - name: velero-plugin-for-gcp
              image: velero/velero-plugin-for-gcp:v1.5.0
              imagePullPolicy: IfNotPresent
              volumeMounts:
                - mountPath: /target
                  name: plugins
          credentials:
            extraSecretRef: bsl-credentials
    ...
    YAML
  5. Use the modified kommander.yaml configuration in install this Velero configuration:

    dkp install kommander --installer-config kommander.yaml --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  6. Check the Helm releases that the new Velero configuration applied/loaded:

    kubectl get helmrelease -n kommander --kubeconfig=${CLUSTER_NAME}.conf
    CODE
  7. Ensure that the Velero pod is running:

    kubectl get pods -A --kubeconfig=${CLUSTER_NAME}.conf |grep velero
    CODE
  8. Create a test backup for GCP:

    velero backup create gcp-velero-testbackup -n kommander --kubeconfig=${CLUSTER_NAME}.conf --storage-location gcp-backup
    CODE
  9. Please note: if your backup wasn’t created, Velero may have had an issue installing the plugin. If the plugin was not installed, run this command:

    velero plugin add velero/velero-plugin-for-gcp:v1.5.0 -n kommander
    CODE

    And then confirm your backupstoragelocation was configured correctly

    kubectl get backupstoragelocations -n kommander
    CODE

    If your backup storage location is “Available”, repeat step 8 and proceed to the rest of the setup

    NAME             PHASE       LAST VALIDATED   AGE   DEFAULT
    gcp-backup       Available   38s              60m   
    CODE
  10. View your backup:

    velero backup describe gcp-velero-testbackup
    CODE