For Konvoy users there are certain issues and caveats you should know when preparing to use GPU nodes. These are:
Kommander can also access GPU resources.
How to use Konvoy automatic GPU node labels
Node feature discovery, (NFD), by default labels PCI hardware as:
"feature.node.kubernetes.io/pci-<device label>.present": "true"
<device label> is by default:
< class > _ < vendor >
However, due to the wide variety in devices and their assigned PCI classes, you may find that the labels assigned to your GPU nodes do not always properly identify them as containing an nVidia GPU.
For now, if the default detection does not work, you can manually change the configmap “nvidia-feature-discovery.yaml” before creating the cluster and change the lines:
nodeSelector: feature.node.kubernetes.io/pci-< class > _ < vendor>.present: "true"
class is any 4 digit number starting with
03xy and the vendor for Nvidia is
10de. If this is already deployed, you can always change the
daemonset and change the
nodeSelector field so that it will deploy to the right nodes.
Install GPU support on supported distributions on-premise
Using the Konvoy Image Builder, you can build an image that has support to use Nvidia GPU hardware to support GPU workloads.
overrides/nvidia.yamlfile add the following to enable GPU builds. You can also access and use the overrides repo.
gpu: type: nvidia
Use the following Konvoy image builder command to build your image:
konvoy-image build --region us-west-2 --source-ami=ami-12345abcdef images/ami/centos-7.yaml --overrides overrides/nvidia.yaml
By default it builds in the
us-west-2region. To specify another region set the
konvoy-image build --region us-east-1 --overrides override-source-ami.yaml --overrides override-images.yaml images/ami/centos-7.yaml
When the command is complete the
amiid is printed and written to
To use the built
amiwith Konvoy, specify it with the
--amiflag when calling cluster create.
dkp create cluster aws --cluster-name=$(whoami)-aws-cluster --region us-west-2 --ami ami-0123456789