Kommander GPU configuration

Configure GPU for Kommander clusters

Prerequisites

Before you begin, you must:

• Ensure nodes provide an NVIDIA GPU.

• If you are using a public cloud service such as AWS, create an AMI with KIB using the instructions on the KIB for GPU page.

• If you are deploying in a pre-provisioned environment, ensure that you have created the appropriate secret for your GPU nodepool and have uploaded the appropriate artifacts to each node. See the GPU only steps section on the Pre-provisioned Prerequisites Air-gapped page for additional information.

Specific instructions must be followed for enabling nvidia-gpu-operator depending on if you want to deploy the app on a Management cluster or a Attached or a Managed cluster.

• For instructions on enabling the NVIDIA platform application on a Management cluster, follow the instructions in the Enable NVIDIA Platform Application on Kommander (Management Cluster) section.

• For instructions on enabling the NVIDIA platform application on attached or managed clusters, follow the instructions in the Enable NVIDIA Platform Application on Attached or Managed Clusters section.

Once nvidia-gpu-operator has been enabled depending on the cluster type, proceed to the Select the correct Toolkit version for your NVIDIA GPU Operator section.

Enable NVIDIA Platform Application on Kommander (Management Cluster)

If you intend to run applications that make use of GPU’s on your cluster, you should install the NVIDIA GPU operator. To enable NVIDIA GPU support when installing Kommander on a management cluster, perform the following steps:

1. Create an installation configuration file:

   dkp install kommander --init > install.yaml

   CODE

2. Append the following to the apps section in the install.yaml file to enable Nvidia platform services.

   apps:
     nvidia-gpu-operator:
      enabled: true

   CODE

3. Install Kommander using the configuration file you created:

   dkp install kommander --installer-config ./install.yaml

   CODE

4. Proceed to the Select the correct Toolkit version for your NVIDIA GPU Operator section.

Enable NVIDIA Platform Application on Attached or Managed Clusters

If you intend to run applications that utilize GPU’s on Attached or Managed clusters, you must enable the nvidia-gpu-operator platform application in the workspace.

To use the UI to enable the application, refer to the Platform Applications | Customize-a-workspace’s-applications page.

To use the CLI, refer to the Deploy Platform Applications via CLI page.

If only a subset of attached or managed clusters in the workspace are utilizing GPU’s, refer to Enable an Application per Cluster on how to only enable the nvidia-gpu-operator on specific clusters.

After you have enabled the nvidia-gpu-operator app in the workspace on the necessary clusters, proceed to the next section.

Select the Correct Toolkit Version for your NVIDIA GPU Operator

The NVIDIA Container Toolkit allows users to run GPU accelerated containers. The toolkit includes a container runtime library and utilities to automatically configure containers to leverage NVIDIA GPU and must be configured correctly according to your base operating system.

Kommander (Management Cluster) Customization

1. Select the correct Toolkit version based on your OS:
   
   Centos 7.9/RHEL 7.9:
   If you’re using Centos 7.9 or RHEL 7.9 as the base operating system for your GPU enabled nodes, set the toolkit.version parameter in your install.yaml to the following:
   

   kind: Installation
   apps:
     nvidia-gpu-operator:
       values: |
         toolkit:
           version: v1.10.0-centos7

   CODE

   
   RHEL 8.4/8.6 and SLES 15 SP3
   If you’re using RHEL 8.4/8.6 or SLES 15 SP3 as the base operating system for your GPU enabled nodes, set the toolkit.version parameter in your install.yaml to the following:
   

   kind: Installation
   apps:
     nvidia-gpu-operator:
       values: |
         toolkit:
           version: v1.10.0-ubi8

   CODE

   
   Ubuntu 18.04 and 20.04
   If you’re using Ubuntu 18.04 or 20.04 as the base operating system for your GPU enabled nodes, set the toolkit.version parameter in your install.yaml to the following:
   

   kind: Installation
   apps:
     nvidia-gpu-operator:
       values: |
         toolkit:
           version: v1.11.0-ubuntu20.04

   CODE

2. Install Kommander, using the configuration file you created:

   dkp install kommander --installer-config ./install.yaml

   CODE

Workspace (Attached and Managed clusters) Customization

Refer to AppDeployment resources for how to use the CLI to customize the platform application on a workspace.

If specific attached/managed clusters in the workspace require different configurations, refer to Customize an Application per Cluster for how to do this.

1. Select the correct Toolkit version based on your OS and create a ConfigMap with these configuration override values:

   Centos 7.9/RHEL 7.9:
   If you’re using Centos 7.9 or RHEL 7.9 as the base operating system for your GPU enabled nodes, set the toolkit.version parameter in your install.yaml to the following:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: ${WORKSPACE_NAMESPACE}
     name: nvidia-gpu-operator-overrides-attached
   data:
     values.yaml: |
       toolkit:
         version: v1.10.0-centos7
   EOF

   CODE

   
   RHEL 8.4/8.6 and SLES 15 SP3
   If you’re using RHEL 8.4/8.6 or SLES 15 SP3 as the base operating system for your GPU enabled nodes, set the toolkit.version parameter in your install.yaml to the following:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: ${WORKSPACE_NAMESPACE}
     name: nvidia-gpu-operator-overrides-attached
   data:
     values.yaml: |
       toolkit:
         version: v1.10.0-ubi8
   EOF

   CODE

   Ubuntu 18.04 and 20.04
   If you’re using Ubuntu 18.04 or 20.04 as the base operating system for your GPU enabled nodes, set the toolkit.version parameter in your install.yaml to the following:

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: ${WORKSPACE_NAMESPACE}
     name: nvidia-gpu-operator-overrides-attached
   data:
     values.yaml: |
       toolkit:
         version: v1.11.0-ubuntu20.04
   EOF

   CODE

2. Note the name of this ConfigMap (nvidia-gpu-operator-overrides-attached) and use it to set the necessary nvidia-gpu-operator AppDeployment spec fields depending on the scope of the override. Alternatively, you can also use the UI to pass in the configuration overrides for the app per workspace or per cluster.

Validate that the Application has Started Correctly

Run the following command to validate that your application has started correctly:

kubectl get pods -A | grep nvidia

The output should be similar to the following:

nvidia-container-toolkit-daemonset-7h2l5   1/1 	Running 	0          	   150m
nvidia-container-toolkit-daemonset-mm65g   1/1 	Running 	0          	   150m
nvidia-container-toolkit-daemonset-mv7xj   1/1 	Running 	0          	   150m
nvidia-cuda-validator-pdlz8            	0/1 	Completed   0          	   150m
nvidia-cuda-validator-r7qc4            	0/1 	Completed   0          	   150m
nvidia-cuda-validator-xvtqm            	0/1 	Completed   0          	   150m
nvidia-dcgm-exporter-9r6rl             	1/1 	Running 	1 (149m ago)   150m
nvidia-dcgm-exporter-hn6hn             	1/1 	Running 	1 (149m ago)   150m
nvidia-dcgm-exporter-j7g7g             	1/1 	Running 	0          	   150m
nvidia-dcgm-jpr57                      	1/1 	Running 	0          	   150m
nvidia-dcgm-jwldh                      	1/1 	Running 	0          	   150m
nvidia-dcgm-qg2vc                      	1/1 	Running 	0          	   150m
nvidia-device-plugin-daemonset-2gv8h   	1/1 	Running 	0          	   150m
nvidia-device-plugin-daemonset-tcmgk   	1/1 	Running 	0          	   150m
nvidia-device-plugin-daemonset-vqj88   	1/1 	Running 	0          	   150m
nvidia-device-plugin-validator-9xdqr   	0/1 	Completed   0          	   149m
nvidia-device-plugin-validator-jjhdr   	0/1 	Completed   0          	   149m
nvidia-device-plugin-validator-llxjk   	0/1 	Completed   0          	   149m
nvidia-operator-validator-9kzv4        	1/1 	Running 	0          	   150m
nvidia-operator-validator-fvsr7        	1/1 	Running 	0          	   150m
nvidia-operator-validator-qr9cj        	1/1 	Running 	0          	   150m

If you are seeing errors, ensure that you set the container toolkit version appropriately based on your OS, as described in the previous section.

NVIDIA GPU Monitoring

Kommander uses the NVIDIA Data Center GPU Manager to export GPU metrics towards Prometheus. By default, Kommander has a Grafana dashboard called NVIDIA DCGM Exporter Dashboard to monitor GPU metrics. This GPU dashboard is shown in Kommander’s Grafana UI.

NVIDIA MIG Settings

MIG stands for Multi-Instance-GPU. It is a mode of operation for future NVIDIA GPUs that allows the user to partition a GPU into a set of MIG devices. Each set appears to the software that is consuming them as a mini-GPU with a fixed partition of memory and a fixed partition of compute resources.

NOTE: MIG is only available for the following NVIDIA devices: H100, A100, and A30.

To Configure MIG

1. Set the MIG strategy according to your GPU topology.
   • mig.strategy should be set to mixed when MIG mode is not enabled on all GPUs on a node.
   • mig.strategy should be set to single when MIG mode is enabled on all GPUs on a node and they have the same MIG device types across all of them.
   
   For the Management Cluster, this can be set at install time by modifying the Kommander configuration file to add configuration for the nvidia-gpu-operator application:
   

   apiVersion: config.kommander.mesosphere.io/v1alpha1
   kind: Installation
   apps:
     nvidia-gpu-operator:
       values: |
         mig:
           strategy: single
   ...

   CODE

   Or by modifying the clusterPolicy object for the GPU operator once it has already been installed.

2. Set the MIG profile for the GPU you are using. In our example, we are using the A30 GPU that supports the following MIG profiles:
   

   4 GPU instances @ 6GB each
   2 GPU instances @ 12GB each
   1 GPU instance @ 24GB

   CODE

   Set the mig profile by labeling the node ${NODE} with the profile as in the example below:

   kubectl label nodes ${NODE} nvidia.com/mig.config=all-1g.6gb --overwrite

   CODE

3. Check the node labels to see if the changes were applied to your MIG enabled GPU node

   kubectl get no -o json | jq .items[0].metadata.labels

   CODE

   
   

   "nvidia.com/mig.config": "all-1g.6gb",
     "nvidia.com/mig.config.state": "success",
     "nvidia.com/mig.strategy": "single"

   CODE

4. Deploy a sample workload:
   

   apiVersion: v1
   kind: Pod
   metadata:
     name: cuda-vector-add
   spec:
     restartPolicy: OnFailure
     containers:
         - name: cuda-vectoradd
         image: "nvidia/samples:vectoradd-cuda11.2.1"
         resources:
           limits:
             nvidia.com/gpu: 1
   
     nodeSelector:
       "nvidia.com/gpu.product": NVIDIA-A30-MIG-1g.6gb

   CODE

   
   If the workload successfully finishes, then your GPU has been properly MIG partitioned.

Troubleshooting NVIDIA GPU Operator on Kommander

In case you run into any errors with NVIDIA GPU Operator, here are a couple commands you can run to troubleshoot:

1. Connect (using SSH or similar) to your GPU enabled nodes and run the nvidia-smi command. Your output should be similar to the following example:
   

     [ec2-user@ip-10-0-0-241 ~]$ nvidia-smi
     Thu Nov  3 22:52:59 2022       
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
     | N/A   54C    P8    11W /  70W |      0MiB / 15109MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+                                                                        
     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |  No running processes found                                                 |
     +-----------------------------------------------------------------------------+

   CODE

2. Another common issue is having a misconfigured toolkit version, resulting in NVIDIA pods in a bad state
   For example:
   

   nvidia-container-toolkit-daemonset-jrqt2                          1/1     Running                 0             29s
   nvidia-dcgm-exporter-b4mww                                        0/1     Error                   1 (9s ago)    16s
   nvidia-dcgm-pqsz8                                                 0/1     CrashLoopBackOff        1 (13s ago)   27s
   nvidia-device-plugin-daemonset-7fkzr                              0/1     Init:0/1                0             14s
   nvidia-operator-validator-zxn4w                                   0/1     Init:CrashLoopBackOff   1 (7s ago)    11s

   CODE

   To modify the toolkit version, run the following commands to modify the AppDeployment for the nvidia gpu operator application:
   
   • Provide the name of a ConfigMap with the custom configuration in the AppDeployment:
   

   cat <<EOF | kubectl apply -f -
   apiVersion: apps.kommander.d2iq.io/v1alpha3
   kind: AppDeployment
   metadata:
     name: nvidia-gpu-operator
     namespace: kommander
   spec:
     appRef:
       kind: ClusterApp
       name: nvidia-gpu-operator-1.11.1
     configOverrides:
       name: nvidia-gpu-operator-overrides
   EOF
   

   CODE

   
   • Create the ConfigMap with the name provided in the previous step, which provides the custom configuration on top of the default configuration in the config map, set the version appropriately:
   

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     namespace: kommander
     name: nvidia-gpu-operator-overrides
   data:
     values.yaml: |
       toolkit:
         version: v1.10.0-centos7
   EOF

   CODE

3. If a node has an NVIDIA GPU installed and the nvidia-gpu-operator application is enabled on the cluster, but the node is still not accepting GPU workloads, it's possible that the nodes do not have a label that indicates there is an NVIDIA GPU present.
   By default the GPU operator will attempt to configure nodes with the following labels present, which are usually applied by the node feature discovery component:
   

   	"feature.node.kubernetes.io/pci-10de.present":      "true",
   	"feature.node.kubernetes.io/pci-0302_10de.present": "true",
   	"feature.node.kubernetes.io/pci-0300_10de.present": "true",

   CODE

   If these labels are not present on a node that you know contains an NVIDIA GPU, you can manually label the node using the following command:
   

   kubectl label node ${NODE} feature.node.kubernetes.io/pci-0302_10de.present=true

   CODE

Disable NVIDIA GPU Operator Platform Application on Kommander

1. Delete all GPU workloads on the GPU nodes where the NVIDIA GPU Operator platform application is present.

2. Delete the existing NVIDIA GPU Operator AppDeployment using the following command:

   kubectl delete appdeployment -n kommander nvidia-gpu-operator

   CODE

3. Wait for all NVIDIA related resources in the Terminating state to be cleaned up. You can check pod status with the following command:

   kubectl get pods -A | grep nvidia

   CODE

For information on how to delete nodepools, refer to Pre-provisioned Create and Delete Node Pools