DKP Enterprise Upgrade

Upgrade your Konvoy environment within the DKP Enterprise license.

Prerequisites

Create an on-demand backup of your current configuration with Velero.
Follow the steps listed in the DKP upgrade overview.
Check what version of DKP you have downloaded currently using cli command dkp version.
Ensure that all platform applications in the management cluster have been upgraded to avoid compatibility issues with the Kubernetes version included in this release. This is done automatically when upgrading Kommander, so ensure that you upgrade Kommander prior to upgrading Konvoy.
Set the appropriate environment variables:
- For Air-gapped, download the required bundles at our support site.
- For Azure, set the required environment variables.
- For AWS and EKS, set the required environment variables.
- For vSphere, set the required environment variables.
- For GCP, set the required environment variables.
vSphere only: If you want to resize your disk, ensure you have reviewed Create a vSphere Base OS Image.

Overview

To upgrade Konvoy for DKP Enterprise:

Upgrade the Cluster API (CAPI) components.
Upgrade the core addons.
Upgrade the Kubernetes version.
Upgrade the Managed clusters.
Upgrade the Kubernetes version of Managed clusters.

Perform all three steps on the management cluster first. Then, execute the second and third steps on additional managed clusters one cluster at a time. For the managed clusters, you use the KUBECONFIG for the management cluster and specify the name of the managed cluster(s) to upgrade. You must maintain your attached clusters manually. Review the documentation from your cloud provider for more information.

For a full list of DKP Enterprise features, see DKP Enterprise.

For air-gapped environments, seed the docker registry as explained here: Air-gapped Seed the Registry

For Pre-provisioned air-gapped environments only, you must run konvoy-image upload artifacts to copy the artifacts onto the cluster hosts before you begin the Upgrade the CAPI Components section below.

CODE

konvoy-image upload artifacts \
    --container-images-dir=./artifacts/images/ \
    --os-packages-bundle=./artifacts/$OS_PACKAGES_BUNDLE \
    --containerd-bundle=artifacts/$CONTAINERD_BUNDLE \
    --pip-packages-bundle=./artifacts/pip-packages.tar.gz

Upgrade the CAPI Components

New versions of DKP come pre-bundled with newer versions of CAPI, newer versions of infrastructure providers, or new infrastructure providers. When using a new version of the DKP CLI, upgrade all of these components first.

If you are running on more than one management cluster, you must upgrade the CAPI components on each of these clusters.

Ensure your dkp configuration references the management cluster where you want to run the upgrade by setting the KUBECONFIG environment variable, or using the --kubeconfig flag, in accordance with Kubernetes conventions.

Execute the upgrade command for the CAPI components.

If you created CAPI components using flags to specify values, use those same flags during Upgrade to preserve existing values while setting additional values.

Refer to dkp create cluster for flag descriptions for --with-aws-bootstrap-credentials and --aws-service-endpoints
Refer to the HTTP section for details: Universal Configurations for all Infrastructures

CODE

dkp upgrade capi-components

The output resembles the following:

CODE

✓ Upgrading CAPI components
✓ Waiting for CAPI components to be upgraded
✓ Initializing new CAPI components
✓ Deleting Outdated Global ClusterResourceSets

If the upgrade fails, review the prerequisites section and ensure that you’ve followed the steps in the DKP Upgrade overview. Furthermore, ensure you have adhered to the Prerequisites at the top of this page.

Upgrade the Core Addons

To install the core addons, DKP relies on the ClusterResourceSet Cluster API feature. In the CAPI component upgrade, we deleted the previous set of outdated global ClusterResourceSets because in past releases, some addons were installed using a global configuration. In order to support individual cluster upgrades, DKP now installs all addons with a unique set of ClusterResourceSets and corresponding referenced resources, all named using the cluster’s name as a suffix. For example: calico-cni-installation-my-aws-cluster.

If you have modified any of the ClusterResourceSet definitions, these changes will not be preserved when running the command dkp upgrade addons <provider>. You must define the cloud provider before you use the --dry-run -o yaml options to save the new configuration to a file and remake the same changes upon each upgrade.

Your cluster comes preconfigured with a few different core addons that provide functionality to your cluster upon creation. These include: CSI, CNI, Cluster Autoscaler, and Node Feature Discovery. New versions of DKP may come pre-bundled with newer versions of these addons.

Perform the following steps to update these addons:

If you have any additional managed clusters, you will need to upgrade the core addons and Kubernetes version for each one.
Ensure your dkp configuration references the management cluster where you want to run the upgrade by setting the KUBECONFIG environment variable, or using the --kubeconfig flag, in accordance with Kubernetes conventions.
Upgrade the core addons in a cluster using the dkp upgrade addons command specifying the cluster infrastructure (choose aws, azure, vsphere, eks,gcp, preprovisioned) and the name of the cluster.

If you need to verify or discover your cluster name to use with this example, first run the kubectl get clusters command.

Examples for upgrade core addons commands:

CODE

export CLUSTER_NAME=my-azure-cluster
dkp upgrade addons azure --cluster-name=${CLUSTER_NAME}

OR

CODE

export CLUSTER_NAME=my-aws-cluster
dkp upgrade addons aws --cluster-name=${CLUSTER_NAME}

The output for the AWS example should be similar to:

CODE

Generating addon resources
clusterresourceset.addons.cluster.x-k8s.io/calico-cni-installation-my-aws-cluster upgraded
configmap/calico-cni-installation-my-aws-cluster upgraded
clusterresourceset.addons.cluster.x-k8s.io/tigera-operator-my-aws-cluster upgraded
configmap/tigera-operator-my-aws-cluster upgraded
clusterresourceset.addons.cluster.x-k8s.io/aws-ebs-csi-my-aws-cluster upgraded
configmap/aws-ebs-csi-my-aws-cluster upgraded
clusterresourceset.addons.cluster.x-k8s.io/cluster-autoscaler-my-aws-cluster upgraded
configmap/cluster-autoscaler-my-aws-cluster upgraded
clusterresourceset.addons.cluster.x-k8s.io/node-feature-discovery-my-aws-cluster upgraded
configmap/node-feature-discovery-my-aws-cluster upgraded
clusterresourceset.addons.cluster.x-k8s.io/nvidia-feature-discovery-my-aws-cluster upgraded
configmap/nvidia-feature-discovery-my-aws-cluster upgraded

Upgrade the Kubernetes Version

When upgrading the Kubernetes version of a cluster, first upgrade the control plane and then the node pools. If you have any additional managed or attached clusters, you need to upgrade the core addons and Kubernetes version for each one.

Build a new image if applicable.
- If an AMI was specified when initially creating a cluster for AWS, you must build a new one with Konvoy Image Builder.
- If an Azure Machine Image was specified for Azure, you must build a new one with Konvoy Image Builder.
- If a vSphere template Image was specified for vSphere, you must build a new one with Konvoy Image Builder.
- You must build a new GCP image with Konvoy Image Builder.
Upgrade the Kubernetes version of the control plane. Each cloud provider has distinctive commands. Below is the AWS command example. Select the drop-down menu next to your provider for compliant CLI.
NOTE: The first example below is for AWS. If you created your initial cluster with a custom AMI using the --ami flag, it is required to set the --ami flag during the Kubernetes upgrade.
CODE
```
dkp update controlplane aws --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4
```

Azure

CODE

dkp update controlplane azure --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 --compute-gallery-id <Azure Compute Gallery built by KIB for Kubernetes v1.25.4>

If these fields were specified in the override file during image creation, the flags must be used in upgrade:
- --plan-offer, --plan-publisher and --plan-sku
- CODE
```
--plan-offer rockylinux-9
--plan-publisher erockyenterprisesoftwarefoundationinc1653071250513
--plan-sku rockylinux-9
```

vSphere

CODE

dkp update controlplane vsphere --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 --vm-template <vSphere template built by KIB for Kubernetes v1.25.4>

GCP

CODE

dkp update controlplane gcp --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 --image=projects/${GCP_PROJECT}/global/images/<GCP image built by KIB for Kubernetes v1.25.4>

Pre-provisioned

CODE

dkp update controlplane preprovisioned --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4

EKS

CODE

dkp update controlplane eks --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.24.7

The nodes.cluster-api-provider-aws.sigs.k8s.io role is required to be present in your AWS account when using EKS clusters with DKP 2.5.1. Refer to the DKP documentation regarding IAM Permissions for more information.

The output should be similar to the below example, with the provider name corresponding to the CLI you executed from the choices above:

CODE

Updating control plane resource controlplane.cluster.x-k8s.io/v1beta1, Kind=KubeadmControlPlane default/my-aws-cluster-control-plane
Waiting for control plane update to finish.
 ✓ Updating the control plane

Some advanced options are available for various providers. To see all the options for your particular provider, run this command dkp update controlplane aws|vsphere|preprovisioned|azure|gcp|eks --help for more advance options like the example below:

This example for AWS AMI instance type: aws: --ami, --instance-type would be some of the options mentioned in the note above.

NOTE: The commanddkp update controlplane {provider} has a 30 minute default timeout for the update process to finish. If you see the error "timed out waiting for the condition“, you can check the control plane nodes version using the command kubectl get machines -o wide $KUBECONFIG before trying again.

Additional Considerations for upgrading a FIPS cluster:

If upgrading a FIPS cluster, to correctly upgrade the Kubernetes version, instead run the command shown below:

CODE

dkp update controlplane aws --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4+fips.0 --ami=<ami-with-fips-id>

3. Upgrade the Kubernetes version of your node pools. Upgrading a nodepool involves draining the existing nodes in the nodepool and replacing them with new nodes. In order to ensure minimum downtime and maintain high availability of the critical application workloads during the upgrade process, we recommend deploying Pod Disruption Budget (Disruptions) for your critical applications. For more information, refer to Update Cluster Nodepools documentation.

a. First, get a list of all node pools available in your cluster by running the following command:

CODE

dkp get nodepool --cluster-name ${CLUSTER_NAME}

b. Select the nodepool you want to upgrade with the command below:

CODE

export NODEPOOL_NAME=my-nodepool

c. Then update the selected nodepool using the command below. The first example command shows AWS language, so select the drop-down menu for your provider for the correct command. Execute the update command for each of the node pools listed in the previous command:
NOTE: The first example below is for AWS. If you created your initial cluster with a custom AMI using the --ami flag, it is required to set the --ami flag during the Kubernetes upgrade.

CODE

dkp update nodepool aws ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4

Azure

CODE

dkp update nodepool azure ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 --compute-gallery-id <Azure Compute Gallery built by KIB for Kubernetes v1.25.4>

If these fields were specified in the override file during image creation, the flags must be used in upgrade:
- --plan-offer, --plan-publisher and --plan-sku
- CODE
```
--plan-offer rockylinux-9
--plan-publisher erockyenterprisesoftwarefoundationinc1653071250513
--plan-sku rockylinux-9
```

vSphere

CODE

dkp update nodepool vsphere ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 --vm-template <vSphere template built by KIB for Kubernetes v1.25.4>

GCP

CODE

dkp update nodepool gcp ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 --image=projects/${GCP_PROJECT}/global/images/<GCP image built by KIB for Kubernetes v1.25.4>

Pre-provisioned

CODE

dkp update nodepool preprovisioned ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4

EKS

CODE

dkp update nodepool eks ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.24.7

The nodes.cluster-api-provider-aws.sigs.k8s.io role is required to be present in your AWS account when using EKS clusters with DKP 2.5.1. Refer to the DKP documentation regarding IAM Permissions for more information.

The output should be similar to the following, with the name of the infrastructure provider shown accordingly:

CODE

Updating node pool resource cluster.x-k8s.io/v1beta1, Kind=MachineDeployment default/my-aws-cluster-my-nodepool
Waiting for node pool update to finish.
 ✓ Updating the my-aws-cluster-my-nodepool node pool

d. Repeat this step for each additional node pool.

Additional Considerations for upgrading a FIPS cluster:

If upgrading a FIPS cluster, to correctly upgrade the Kubernetes version, instead run the command shown below:

CODE

dkp update nodepool aws ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4+fips.0 --ami=<ami-with-fips-id>

When all nodepools have been updated, your upgrade is complete. For the overall process for upgrading to the latest version of DKP, refer back to DKP Upgrade for more details.

Upgrade Managed Clusters

If you have managed clusters, follow these steps to upgrade each cluster:

Using the kubeconfig of your management cluster, find your cluster name and be sure to copy the information for all of your clusters:
CODE
```
kubectl get clusters -A
```

Set your cluster variable:

CODE

export CLUSTER_NAME=<your-managed-cluster-name>

Set your cluster's workspace variable:

CODE

export CLUSTER_WORKSPACE=<your-workspace-namespace>

Then, upgrade the core addons (replacing aws with whatever infrastructure provider you would be using):
CODE
```
dkp upgrade addons aws --cluster-name=${CLUSTER_NAME} -n ${CLUSTER_WORKSPACE} 
```
Check to see if you have any cluster resource sets that need to be cleaned up:
CODE
```
kubectl get clusterresourcesets -n ${CLUSTER_WORKSPACE}
```

Delete the ClusterResourceSet for nvidia-feature-discovery by running:

CODE

kubectl delete clusterresourceset nvidia-feature-discovery-${CLUSTER_NAME} -n ${CLUSTER_WORKSPACE}

Delete ConfigMap ClusterResourceSet referred to by running the following command, ensure you use using nvidia-feature-discover-${CLUSTER_NAME}. If there is no related ConfigMap, then you can move on to the next step.
CODE
```
kubectl delete configmap nvidia-feature-discovery-${CLUSTER_NAME} -n ${CLUSTER_WORKSPACE}
```

Get the kubeconfig for the managed cluster by running:

CODE

dkp get kubeconfig -c ${CLUSTER_NAME} -n ${CLUSTER_WORKSPACE} >> ${CLUSTER_NAME}.conf

Delete the corresponding daemonset on the remote cluster by running the following command. If there is no related daemonset, then you can move on to the next step.
CODE
```
kubectl --kubeconfig=${CLUSTER_NAME}.conf delete daemonset nvidia-feature-discovery-gpu-feature-discovery -n node-feature-discovery
```

Upgrade Kubernetes Version on a Managed Cluster

After you complete the previous steps for all managed clusters and you update your core addons, begin upgrading the Kubernetes version.

You should first complete the upgrade of your Kommander Management Cluster before upgrading any managed clusters.

Use this command to start upgrading the Kubernetes version:

CODE

dkp update controlplane aws --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 -n ${CLUSTER_WORKSPACE}

Get a list of all node pools available in your cluster by running the following command:

CODE

dkp get nodepools -c ${CLUSTER_NAME} -n ${CLUSTER_WORKSPACE}

export NODEPOOL_NAME=<my-nodepool>

Use this command to upgrade the node pools:

CODE

dkp update nodepool aws ${NODEPOOL_NAME} --cluster-name=${CLUSTER_NAME} --kubernetes-version=v1.25.4 -n ${CLUSTER_WORKSPACE}

Prerequisites

Overview

Upgrade the CAPI Components

Upgrade the Core Addons

See Also

Upgrade the Kubernetes Version

Upgrade Managed Clusters

Upgrade Kubernetes Version on a Managed Cluster