When using GPU nodes, it is important they have the proper label identifying them as Nvidia GPU nodes. Node feature discovery (NFD), by default labels PCI hardware as:
"feature.node.kubernetes.io/pci-<device label>.present": "true"
<device label> is by default as defined in this topic:
< class > _ < vendor >
However, due to the wide variety in devices and their assigned PCI classes, you may find that the labels assigned to your GPU nodes do not always properly identify them as containing an Nvidia GPU.
If the default detection does not work, you can manually change the configmap “nvidia-feature-discovery.yaml” before creating the cluster and change the lines:
nodeSelector: feature.node.kubernetes.io/pci-< class > _ < vendor>.present: "true"
class is any 4 digit number starting with
03xy and the vendor for Nvidia is
10de. If this is already deployed, you can always change the
daemonset and change the
nodeSelector field so that it will deploy to the right nodes.