Pre-provisioned GPU: Create Managed Clusters Using the DKP CLI
After initial cluster creation, you have the ability to create additional clusters from the CLI. In a previous step, the new cluster was created as Self-managed which allows it to be a Management cluster or a stand alone cluster. Subsequent new clusters are not self-managed as they will likely be Managed or Attached clusters to this Management Cluster.
When creating Managed clusters, you do not need to create and move CAPI objects, or install the Kommander component. Those tasks are only done on Management clusters!
Make New Cluster Part of a Workspace
If you have an existing Workspace name, run this command to find the name:
⚠️ NOTE: If you need to create a new Workspace, follow the instructions to Create a New Workspace.CODEkubectl get workspace -A
When you have the Workspace name, set the
WORKSPACE_NAMESPACE
environment variable:CODEexport WORKSPACE_NAMESPACE=<workspace_namespace>
Name Your Cluster
The cluster name may only contain the following characters: a-z
, 0-9
, .
, and -
. Cluster creation will fail if the name has capital letters. See Kubernetes for more naming information.
When specifying the cluster-name
, you must use the same cluster-name
as used when defining your inventory objects.
By default, the control-plane Nodes will be created in 3 different zones. However, the default worker Nodes will reside in a single Availability Zone. You may create additional node pools in other Availability Zones with the dkp create nodepool
command.
Follow these steps:
Give your cluster a unique name suitable for your environment.
Set the environment variable:
export CLUSTER_NAME=<preprovisioned-additional>
If a custom AMI was created using Konvoy Image Builder, the custom ami
id is printed and written to ./manifest.json
. To use the built ami
with Konvoy, specify it with the --ami
flag when calling cluster create command below.
Create a Kubernetes Cluster
Once you’ve defined the infrastructure and control plane endpoints, you can proceed to creating the cluster by following these steps to create a new pre-provisioned cluster. This process will create a self-managed cluster to be used as the Management cluster.
Before you create a new DKP cluster below, choose an external load balancer (LB) or virtual IP and use the corresponding dkp create cluster
command.
In a pre-provisioned environment, use the Kubernetes CSI and third party drivers for local volumes and other storage devices in your data center.
DKP uses local static provisioner as the default storage provider for a pre-provisioned environment. However, localvolumeprovisioner
is not suitable for production use. You should use a Kubernetes CSI compatible storage that is suitable for production.
After disabling localvolumeprovisioner
, you can choose from any of the storage options available for Kubernetes. To make that storage the default storage, use the commands shown in this section of the Kubernetes documentation: https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/
The following command relies on the pre-provisioned cluster API infrastructure provider to initialize the Kubernetes control plane and worker nodes on the hosts defined in the inventory YAML previously created.
This command uses the default external load balancer (LB) option:
CODEdkp create cluster preprovisioned --cluster-name ${CLUSTER_NAME} --control-plane-endpoint-host <control plane endpoint host> --control-plane-endpoint-port <control plane endpoint port, if different than 6443> --pre-provisioned-inventory-file preprovisioned_inventory.yaml --ssh-private-key-file <path-to-ssh-private-key> --ami <ami>
If your environment uses HTTP/HTTPS proxies, you must include the flags
--http-proxy
,--https-proxy
, and--no-proxy
and their related values in this command for it to be successful. More information is available in Configuring an HTTP/HTTPS Proxy.Use the wait command to monitor the cluster control-plane readiness:
CODEkubectl wait --for=condition=ControlPlaneReady "clusters/${CLUSTER_NAME}" --timeout=30m
Output:
CODEcluster.cluster.x-k8s.io/preprovisioned-additional condition met
NOTE: Depending on the cluster size, it will take a few minutes to create.
Manually Attach a DKP CLI Cluster to the Management Cluster
These steps are only applicable if you do not set a WORKSPACE_NAMESPACE when creating a cluster. If you already set a WORKSPACE_NAMESPACE, then you do not need to perform these steps since the cluster is already attached to the workspace.
Starting with DKP 2.6, when you create a Managed Cluster with the DKP CLI, it attaches automatically to the Management Cluster after a few moments.
However, if you do not set a workspace, the attached cluster will be created in the default
workspace. To ensure that the attached cluster is created in your desired workspace namespace, follow these instructions:
Confirm you have your MANAGED_CLUSTER_NAME variable set with the following command:
CODEecho ${MANAGED_CLUSTER_NAME}
Retrieve your kubeconfig from the cluster you have created without setting a workspace:
CODEdkp get kubeconfig --cluster-name ${MANAGED_CLUSTER_NAME} > ${MANAGED_CLUSTER_NAME}.conf
You can now either [attach it in the UI](link to attaching it to workspace via UI that was earlier), or attach your cluster to the workspace you want in the CLI.
NOTE: This is only necessary if you never set the workspace of your cluster upon creation.Retrieve the workspace where you want to attach the cluster:
CODEkubectl get workspaces -A
Set the WORKSPACE_NAMESPACE environment variable:
CODEexport WORKSPACE_NAMESPACE=<workspace-namespace>
You need to create a secret in the desired workspace before attaching the cluster to that workspace. Retrieve the kubeconfig secret value of your cluster:
CODEkubectl -n default get secret ${MANAGED_CLUSTER_NAME}-kubeconfig -o go-template='{{.data.value}}{{ "\n"}}'
This will return a lengthy value. Copy this entire string for a secret using the template below as a reference. Create a new attached-cluster-kubeconfig.yaml file:
CODEapiVersion: v1 kind: Secret metadata: name: <your-managed-cluster-name>-kubeconfig labels: cluster.x-k8s.io/cluster-name: <your-managed-cluster-name> type: cluster.x-k8s.io/secret data: value: <value-you-copied-from-secret-above>
Create this secret in the desired workspace:
CODEkubectl apply -f attached-cluster-kubeconfig.yaml --namespace ${WORKSPACE_NAMESPACE}
Create this
kommandercluster
object to attach the cluster to the workspace:CODEcat << EOF | kubectl apply -f - apiVersion: kommander.mesosphere.io/v1beta1 kind: KommanderCluster metadata: name: ${MANAGED_CLUSTER_NAME} namespace: ${WORKSPACE_NAMESPACE} spec: kubeconfigRef: name: ${MANAGED_CLUSTER_NAME}-kubeconfig clusterRef: capiCluster: name: ${MANAGED_CLUSTER_NAME} EOF
You can now view this cluster in your Workspace in the UI and you can confirm its status by running the below command. It may take a few minutes to reach "Joined" status:
CODEkubectl get kommanderclusters -A
If you have several Essential Clusters and want to turn one of them to a Managed Cluster to be centrally administrated by a Management Cluster, refer to Platform Expansion: Convert a DKP Essential Cluster to a DKP Enterprise Managed Cluster.