Pre-provisioned Air-gapped Define Environment

Fulfill the prerequisites for using a pre-provisioned infrastructure when Air-Gapped

The instructions below outline how to fulfill the prerequisites for using pre-provisioned infrastructure when using an air-gapped environement. In the current release of DKP, there is a complete DKP air-gapped bundle available to download which contains all the DKP components needed for an air-gapped Install Overview environment installation. (i.e. dkp-air-gapped-bundle_v2.6.2_linux_amd64.tar.gz)

Air-Gapped Registry Prerequisites

DKP in an air-gapped environment requires a local container registry of trusted images to enable production level Kubernetes cluster management. In an environment with access to the internet, you retrieve artifacts from specialized repositories dedicated to them such as Docker images contained in DockerHub and Helm Charts that come from a dedicated Helm Chart repository. However, in an air-gapped environment, you need local repositories to store Helm charts, Docker images and other artifacts. Tools such as ECR, jFrog, Harbor and Nexus handle multiple types of artifacts in one local repository.

AWS ECR - AWS ECR (Elastic Container Registry) is supported as your air-gapped image registry. DKP added support for using AWS ECR as a default registry when uploading image bundles.
JFrog Artifactory - If you use Jfrog Artifactory or Jfrog Container Registry, you must update to a new version of the software. Any build newer than version 7.11 will work, as we have confirmed that older versions are not compatible.
Nexus Registry - If you use Nexus Registry, there was an issue that prevented usage with DKP 2.X and OCI Images, but support for OCI Images was added here in this publicly available Jira ticket:[NEXUS-21087] Support OCI registry format - Sonatype JIRA
Harbor Registry - Any newer version than Harbor Registry v2.1.1-5f52168e will support OCI images.

Bastion Host

If you have not set up a Bastion Host yet, refer to that section of the Documentation.

Load the bootstrap image

Assuming you have downloaded dkp-air-gapped-bundle_v2.6.2_linux_amd64.tar.gz from the download site mentioned above, extract the tarball to a local directory:
CODE
```
tar -xzvf dkp-air-gapped-bundle_v2.6.2_linux_amd64.tar.gz
```
The directory structure after extraction can be accessed in subsequent steps using commands to access files from different directories. For the bootstrap, change your directory to the dkp-<version> directory similar to example below depending on your current location:
CODE
```
cd dkp-v2.6.2
```
Load the bootstrap Docker image on your bastion machine:
CODE
```
docker load -i konvoy-bootstrap-image-v2.6.2.tar
```

Copy air-gapped artifacts onto cluster hosts

Using the Konvoy Image Builder, you can copy the required artifacts (such as charts, java or OS packages like RPM or Deb) onto your cluster hosts.

Assuming you have downloaded dkp-air-gapped-bundle_v2.6.2_linux_amd64.tar.gz and extracted the tarball from the section above, the Kubernetes image bundle will be located in kib/artifacts/images. Verify image and artifacts with the commands below:

Verify the image bundles exist in kib/artifacts/images:

CODE

$ ls kib/artifacts/images/
kubernetes-images-1.26.14-d2iq.1.tar kubernetes-images-1.26.14-d2iq.1-fips.tar

Verify the artifacts for your OS exist in the artifacts/ directory and export the appropriate variables:

CODE

$ ls kib/artifacts/
1.26.14_centos_7_x86_64.tar.gz	    1.26.14_redhat_8_x86_64_fips.tar.gz			    containerd-1.6.17-d2iq.1-rhel-7.9-x86_64.tar.gz	  containerd-1.6.17-d2iq.1-rhel-8.6-x86_64_fips.tar.gz	pip-packages.tar.gz
1.26.14_centos_7_x86_64_fips.tar.gz  1.26.14_rocky_9_x86_64.tar.gz			    containerd-1.6.17-d2iq.1-rhel-7.9-x86_64_fips.tar.gz  containerd-1.6.17-d2iq.1-rocky-9.0-x86_64.tar.gz
1.26.14_redhat_7_x86_64.tar.gz	    1.26.14_ubuntu_20_x86_64.tar.gz			    containerd-1.6.17-d2iq.1-rhel-8.4-x86_64.tar.gz	  containerd-1.6.17-d2iq.1-rocky-9.1-x86_64.tar.gz
1.26.14_redhat_7_x86_64_fips.tar.gz  containerd-1.6.17-d2iq.1-centos-7.9-x86_64.tar.gz	    containerd-1.6.17-d2iq.1-rhel-8.4-x86_64_fips.tar.gz  containerd-1.6.17-d2iq.1-ubuntu-20.04-x86_64.tar.gz
1.26.14_redhat_8_x86_64.tar.gz	    containerd-1.6.17-d2iq.1-centos-7.9-x86_64_fips.tar.gz  containerd-1.6.17-d2iq.1-rhel-8.6-x86_64.tar.gz	  images

For example, for RHEL 8.4 you would set:

CODE

export OS_PACKAGES_BUNDLE=1.26.14_redhat_8_x86_64.tar.gz
export CONTAINERD_BUNDLE=containerd-1.6.17-d2iq.1-rhel-8.4-x86_64.tar.gz

Export the following environment variables, ensuring that all control plane and worker nodes are included:

CODE

export CONTROL_PLANE_1_ADDRESS="<control-plane-address-1>"
export CONTROL_PLANE_2_ADDRESS="<control-plane-address-2>"
export CONTROL_PLANE_3_ADDRESS="<control-plane-address-3>"
export WORKER_1_ADDRESS="<worker-address-1>"
export WORKER_2_ADDRESS="<worker-address-2>"
export WORKER_3_ADDRESS="<worker-address-3>"
export WORKER_4_ADDRESS="<worker-address-4>"
export SSH_USER="<ssh-user>"
export SSH_PRIVATE_KEY_FILE="<private key file>"

SSH_PRIVATE_KEY_FILE must be either the name of the SSH private key file in your working directory or an absolute path to the file in your user’s home directory.

Generate an inventory.yaml which is automatically picked up by the konvoy-image upload in the next step. This inventory.yaml should exclude any GPU workers, which will be handled in steps #6-7.

CODE

cat <<EOF > inventory.yaml
all:
  vars:
    ansible_user: $SSH_USER
    ansible_port: 22
    ansible_ssh_private_key_file: $SSH_PRIVATE_KEY_FILE
  hosts:
    $CONTROL_PLANE_1_ADDRESS:
      ansible_host: $CONTROL_PLANE_1_ADDRESS
    $CONTROL_PLANE_2_ADDRESS:
      ansible_host: $CONTROL_PLANE_2_ADDRESS
    $CONTROL_PLANE_3_ADDRESS:
      ansible_host: $CONTROL_PLANE_3_ADDRESS
    $WORKER_1_ADDRESS:
      ansible_host: $WORKER_1_ADDRESS
    $WORKER_2_ADDRESS:
      ansible_host: $WORKER_2_ADDRESS
    $WORKER_3_ADDRESS:
      ansible_host: $WORKER_3_ADDRESS
    $WORKER_4_ADDRESS:
      ansible_host: $WORKER_4_ADDRESS
EOF

Upload the artifacts onto cluster hosts with the following command:
BASH
```
konvoy-image upload artifacts \
              --container-images-dir=./kib/artifacts/images/ \
              --os-packages-bundle=./kib/artifacts/$OS_PACKAGES_BUNDLE \
              --containerd-bundle=./kib/artifacts/$CONTAINERD_BUNDLE \
              --pip-packages-bundle=./kib/artifacts/pip-packages.tar.gz
```
The konvoy-image upload artifacts command copies all OS packages and other artifacts onto each of the machines in your inventory. When you create the cluster, the provisioning process connects to each node and runs commands to install those artifacts and consequently Kubernetes running. KIB uses variable overrides to specify base image and container images to use in your new machine image. The variable overrides files for NVIDIA and FIPS can be ignored unless adding an overlay feature.

Use the overrides flag (EX:--overrides overrides/fips.yaml) and reference either fips.yaml or offline-fips.yaml manifests located in the overrides directory or see these pages in the documentation:
- FIPS Overrides
- Create FIPS 140 Images

GPU Only Steps

If the NVIDIA runfile installer has not been downloaded, then retrieve and install the download first by running the following command. The first line in the command below downloads and installs the runfile and the second line places it in the artifacts directory.

CODE

curl -O https://download.nvidia.com/XFree86/Linux-x86_64/470.82.01/NVIDIA-Linux-x86_64-470.82.01.run
mv NVIDIA-Linux-x86_64-470.82.01.run artifacts

6. Create an inventory for GPU Nodes.

CODE

cat <<EOF >  gpu_inventory.yaml 
all:
  vars:
    ansible_port: 22
    ansible_ssh_private_key_file: $SSH_PRIVATE_KEY_FILE
    ansible_user: $SSH_USER

  hosts:
    $GPU_WORKER_1_ADDRESS:
      ansible_host: $GPU_WORKER_1_ADDRESS
EOF

7. Upload the artifacts to the gpu nodepool with the nvidia-runfile flag

CODE

konvoy-image upload artifacts \
              --inventory-file=gpu_inventory.yaml \
              --container-images-dir=./kib/artifacts/images/ \
              --os-packages-bundle=./kib/artifacts/$OS_PACKAGES_BUNDLE \
              --containerd-bundle=./kib/artifacts/$CONTAINERD_BUNDLE \
              --pip-packages-bundle=./kib/artifacts/pip-packages.tar.gz \
              --nvidia-runfile=./kib/artifacts/NVIDIA-Linux-x86_64-470.82.01.run

KIB uses variable overrides to specify base image and container images to use in your new machine image. The variable overrides files for NVIDIA and FIPS can be ignored unless adding an overlay feature.

Seed your Registry

Before creating an air-gapped Kubernetes cluster, you need to load the required images in a local registry. This registry must be accessible from both the bastion machine and either the AWS EC2 instances (if deploying to AWS) or other machines that will be created for the Kubernetes cluster.

If you do not already have a local registry set up, please refer to Local Registry Tools page for more information.

Download dkp-air-gapped-bundle_v2.6.2_linux_amd64.tar.gz , and extract the tarball to a local directory:
CODE
```
tar -xzvf dkp-air-gapped-bundle_v2.6.2_linux_amd64.tar.gz && cd dkp-v2.6.2
```

Set an environment variable with your registry address using this command:

CODE

export REGISTRY_ADDRESS=<registry-address>:<registry-port>
export REGISTRY_USERNAME=<username>
export REGISTRY_PASSWORD=<password>

Execute the following command to load the air-gapped image bundle into your private registry:
CODE
```
dkp push bundle --bundle ./container-images/konvoy-image-bundle-v2.6.2.tar --to-registry $REGISTRY_ADDRESS --to-registry-username $REGISTRY_USERNAME --to-registry-password $REGISTRY_PASSWORD
```
NOTE: To use ECR, set an environment variable with your registry address for ECR:
CODE
```
export REGISTRY_URL=<ecr-registry-URI>
```
- REGISTRY_URL: the address of an existing local registry accessible in the VPC that the new cluster nodes will be configured to use a mirror registry when pulling images.
- The environment where you are running the dkp push command must be authenticated with AWS in order to load your images into ECR.

It may take some time to push all the images to your image registry, depending on the performance of the network between the machine you are running the script on and the registry.

Next Step:

Pre-provisioned Set Infrastructure