Configure Konvoy Automatic GPU Node Labels

When using GPU nodes, it is important they have the proper label identifying them as Nvidia GPU nodes. Node feature discovery (NFD), by default labels PCI hardware as:

CODE

"feature.node.kubernetes.io/pci-<device label>.present": "true"

where <device label> is by default as defined in this topic:

CODE

< class > _ < vendor >

However, because there is a wide variety in devices and their assigned PCI classes, you may find that the labels assigned to your GPU nodes do not always properly identify them as containing an Nvidia GPU.

If the default detection does not work, you can manually change the daemonset that the GPU operator creates by running the following command:

CODE

          nodeSelector:
            feature.node.kubernetes.io/pci-< class > _ < vendor>.present: "true"

where class is any 4 digit number starting with 03xy and the vendor for Nvidia is 10de. If this is already deployed, you can always change the daemonset and change the nodeSelector field so that it deploys to the right nodes.