Skip to main content
Skip table of contents

AWS Create a New Air-gapped Cluster

Prerequisites

Before you begin, make sure you have created a Bootstrap cluster.

By default, the control-plane Nodes will be created in 3 different zones. However, the default worker Nodes will reside in a single Availability Zone. You may create additional node pools in other Availability Zones with the dkp create nodepool command.

Create a New AWS Air-gapped Kubernetes Cluster

Create a new AWS Kubernetes cluster

  1. Name your cluster using the following characters: a-z, 0-9, ., and -. Cluster creation will fail if the name has capital letters. In AWS it is critical that the name is unique, as no two clusters in the same AWS account can have the same name. See Kubernetes for more naming information.
    Set the environment variable to the name you assigned this cluster:

    CODE
    export CLUSTER_NAME=<aws-example>
  2. Export the variables such as custom AMI and existing infrastructure details for later use with the dkp create cluster command:
    See the Custom AMI in Cluster Creation topic for more information.

    CODE
    export AWS_AMI_ID=<ami-...>
    export AWS_VPC_ID=<vpc-...>
    export AWS_SUBNET_IDS=<subnet-...,subnet-...,subnet-...>
    export AWS_ADDITIONAL_SECURITY_GROUPS=<sg-...>
Flags to use during cluster creation to apply these variables:
CODE
--ami=${AWS_AMI_ID} \
--vpc-id=${AWS_VPC_ID} \
--subnet-ids=${AWS_SUBNET_IDS} \
--additional-security-group-ids=${AWS_ADDITIONAL_SECURITY_GROUPS} \

In previous DKP releases, AMI images provided by the upstream CAPA project would be used if you did not specify an AMI. However, the upstream images are not recommended for production and may not always be available.   Therefore, DKP now requires you to specify an AMI when creating a cluster. To create an AMI, use Konvoy Image Builder.

There are two approaches to supplying the ID of your AMI. Either provide the ID of the AMI or provide a way for DKP to discover the AMI using location, format and OS information:

  • Option One - Provide the ID of your AMI:

    • Use the example command below leaving the existing flag that provides the AMI ID: --ami AMI_ID

  • Option Two - Provide a path for your AMI with the information required for image discover:

    1. Where the AMI is published using your AWS Account ID: --ami-owner AWS_ACCOUNT_ID

    2. The format or string used to search for matching AMIs and ensure it references the Kubernetes version plus the base OS name: --ami-base-os ubuntu-20.04

    3. The base OS information: --ami-format 'example-{{.BaseOS}}-?{{.K8sVersion}}-*'

When using an existing air-gapped infrastructure, DKP does not create, modify, or delete the AWS resources such as Internet gateways, VPSs, subnets, etc. For more information, refer to AWS Air-gapped Environment Variables

  1. Ensure your subnets do not overlap with your host subnet because they cannot be changed after cluster creation. If you need to change the kubernetes subnets, you must do this at cluster creation. The default subnets used in DKP are:

    CODE
    spec:
      clusterNetwork:
        pods:
          cidrBlocks:
          - 192.168.0.0/16
        services:
          cidrBlocks:
          - 10.96.0.0/12
  2. Your cluster uses the existing local registry when attempting to pull images previously pushed to your registry. The registry variables were set during the BOOTSTRAP process on previous page and will be used with the flags during the dkp create cluster command next.

Registry flags to use during cluster creation:
CODE
  --registry-url=${REGISTRY_URL} \
  --registry-mirror-cacert=${REGISTRY_CA} \
  --registry-mirror-username=${REGISTRY_USERNAME} \
  --registry-mirror-password=${REGISTRY_PASSWORD}

See also: Use a Registry Mirror

Definitions of the variables is still needed:

  • REGISTRY_URL: the address of an existing local registry accessible in the VPC that the new cluster nodes will be configured to use a mirror registry when pulling images.

Other local registries may use the options below:

  • JFrog - REGISTRY_CA: (optional) the path on the bastion machine to the registry CA. This value is only needed if the registry is using a self-signed certificate and the AMIs are not already configured to trust this CA.

  • REGISTRY_USERNAME: optional-set to a user that has pull access to this registry.

  • REGISTRY_PASSWORD: optional if username is not set.

To increase Docker Hub's rate limit use your Docker Hub credentials when creating the cluster, by setting flags --registry-mirror-url=https://registry-1.docker.io --registry-mirror-username=<your-username> --registry-mirror-password=<your-password> when running dkp create cluster.

  1. Create the Kubernetes cluster objects with a dry run. The following example shows a common configuration. See dkp create cluster aws reference for the full list of cluster creation options:

    CODE
    dkp create cluster aws 
      --cluster-name=${CLUSTER_NAME} \
      --vpc-id=${AWS_VPC_ID} \
      --ami=${AWS_AMI_ID} \
      --subnet-ids=${AWS_SUBNET_IDS} \
      --internal-load-balancer=true \
      --additional-security-group-ids=${AWS_ADDITIONAL_SECURITY_GROUPS} \
      --registry-mirror-url=${REGISTRY_URL} \
      --dry-run \ 
      --output=yaml \ 
      > ${CLUSTER_NAME}.yaml

Expand the drop-downs for more flags for use in cluster creation such as registry, HTTP, FIPS and other flags to apply in the step above.

Flatcar

Flatcar OS use this flag used to instruct the bootstrap cluster to make some changes related to the installation paths:

CODE
--os-hint flatcar
HTTP ONLY

If your environment uses HTTP/HTTPS proxies, you must include the flags --http-proxy, --https-proxy, and --no-proxy and their related values in this command for it to be successful. More information is available in Configuring an HTTP/HTTPS Proxy.

CODE
--http-proxy <<http proxy list>>
--https-proxy <<https proxy list>>
--no-proxy <<no proxy list>>
  • To configure the Control Plane and Worker nodes to use an HTTP proxy:

    CODE
    export CONTROL_PLANE_HTTP_PROXY=http://example.org:8080
    export CONTROL_PLANE_HTTPS_PROXY=http://example.org:8080
    export CONTROL_PLANE_NO_PROXY="example.org,example.com,example.net,localhost,127.0.0.1,10.96.0.0/12,192.168.0.0/16,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,.svc,.svc.cluster,.svc.cluster.local,169.254.169.254,.elb.amazonaws.com"
    
    export WORKER_HTTP_PROXY=http://example.org:8080
    export WORKER_HTTPS_PROXY=http://example.org:8080
    export WORKER_NO_PROXY="example.org,example.com,example.net,localhost,127.0.0.1,10.96.0.0/12,192.168.0.0/16,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,.svc,.svc.cluster,.svc.cluster.local,169.254.169.254,.elb.amazonaws.com"
  • Replace:

    • example.org,example.com,example.net with you internal addresses

    • localhost and 127.0.0.1 addresses should not use the proxy

    • 10.96.0.0/12 is the default Kubernetes service subnet

    • 192.168.0.0/16 is the default Kubernetes pod subnet

    • kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local is the internal Kubernetes kube-apiserver service

    • .svc,.svc.cluster,.svc.cluster.local is the internal Kubernetes services

    • 169.254.169.254 is the AWS metadata server

    • .elb.amazonaws.com is for the worker nodes to allow them to communicate directly to the kube-apiserver ELB

  • Apply the HTTP flags during the dkp create cluster command:

    CODE
    --control-plane-http-proxy=${CONTROL_PLANE_HTTP_PROXY} \
    --control-plane-https-proxy=${CONTROL_PLANE_HTTPS_PROXY} \
    --control-plane-no-proxy=${CONTROL_PLANE_NO_PROXY} \
    --worker-http-proxy=${WORKER_HTTP_PROXY} \
    --worker-https-proxy=${WORKER_HTTPS_PROXY} \
    --worker-no-proxy=${WORKER_NO_PROXY}
FIPS considerations

To create a cluster in FIPS mode, inform the controllers of the appropriate image repository and version tags of the official D2iQ FIPS builds of Kubernetes by adding those flags to dkp create cluster command:

CODE
--kubernetes-version=v1.28.7+fips.0 \
--etcd-version=3.5.10+fips.0 \
--kubernetes-image-repository=docker.io/mesosphere \
Individual manifests using the Output Directory flag:

You can create individual files with different smaller manifests for ease in editing using the --output-directory flag. This will create multiple files in the specified directory which must already exist:

CODE
--output-directory=<existing-directory>

Refer to the Cluster Creation Customization Choices section for more information on how to use optional flags such as the --output-directory flag.

  1. Inspect or edit the generated cluster objects. Familiarize yourself with Cluster API before editing the cluster objects as edits can prevent the cluster from deploying successfully. See AWS Customizing CAPI Clusters.

    CODE
    kubectl get clusters,kubeadmcontrolplanes,machinedeployments
  2. (Optional) Modify Control Plane Audit logs - Users can make modifications to the KubeadmControlplane cluster-api object to configure different kubelet options. See the following guide if you wish to configure your control plane beyond the existing options that are available from flags.

  3. Create the cluster from the objects generated in the dry run. A warning will appear in the console if the resource already exists and will require you to remove the resource or update your YAML.

    CODE
    kubectl create -f ${CLUSTER_NAME}.yaml

    NOTE: If you used the --output-directory flag in your dkp create .. --dry-run step above, create the cluster from the objects you created by specifying the directory:

    CODE
    kubectl create -f <existing-directory>/
OUTPUT will be similar to output shown in this drop-down:
CODE
cluster.cluster.x-k8s.io/aws-example created
awscluster.infrastructure.cluster.x-k8s.io/aws-example created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/aws-example-control-plane created
awsmachinetemplate.infrastructure.cluster.x-k8s.io/aws-example-control-plane created
secret/aws-example-etcd-encryption-config created
machinedeployment.cluster.x-k8s.io/aws-example-md-0 created
awsmachinetemplate.infrastructure.cluster.x-k8s.io/aws-example-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/aws-example-md-0 created
clusterresourceset.addons.cluster.x-k8s.io/calico-cni-installation-aws-example created
configmap/calico-cni-installation-aws-example created
configmap/tigera-operator-aws-example created
clusterresourceset.addons.cluster.x-k8s.io/aws-ebs-csi-aws-example created
configmap/aws-ebs-csi-aws-example created
clusterresourceset.addons.cluster.x-k8s.io/cluster-autoscaler-aws-example created
configmap/cluster-autoscaler-aws-example created
clusterresourceset.addons.cluster.x-k8s.io/node-feature-discovery-aws-example created
configmap/node-feature-discovery-aws-example created
clusterresourceset.addons.cluster.x-k8s.io/nvidia-feature-discovery-aws-example created
configmap/nvidia-feature-discovery-aws-example created
  1. Wait for the cluster control-plane to be ready:

    CODE
    kubectl wait --for=condition=ControlPlaneReady "clusters/${CLUSTER_NAME}" --timeout=60m
  2. After the objects are created on the API server, the Cluster API controllers reconcile them. They create infrastructure and machines. As they progress, they update the Status of each object. Konvoy provides a command to describe the current status of the cluster:

    CODE
    dkp describe cluster -c ${CLUSTER_NAME}
OUTPUT:
CODE
NAME                                                              READY  SEVERITY  REASON  SINCE  MESSAGE
Cluster/aws-example                                             True                     60s
├─ClusterInfrastructure - AWSCluster/aws-example                True                     5m23s
├─ControlPlane - KubeadmControlPlane/aws-example-control-plane  True                     60s
│ ├─Machine/aws-example-control-plane-55jh4                     True                     4m59s
│ ├─Machine/aws-example-control-plane-6sn97                     True                     2m49s
│ └─Machine/aws-example-control-plane-nx9v5                     True                     66s
└─Workers
  └─MachineDeployment/aws-example-md-0                          True                     117s
	├─Machine/aws-example-md-0-cb9c9bbf7-hcl8z                  True                     3m1s
	├─Machine/aws-example-md-0-cb9c9bbf7-rtdqw                  True                     3m2s
	├─Machine/aws-example-md-0-cb9c9bbf7-t894m                  True                     3m1s
	└─Machine/aws-example-md-0-cb9c9bbf7-td29r                  True
  1. As they progress, the controllers also create Events. List the Events using this command:

    CODE
    kubectl get events | grep ${CLUSTER_NAME}

DKP uses AWS CSI as the default storage provider. You can use a Kubernetes CSI compatible storage solution that is suitable for production. See the Kubernetes documentation called Changing the Default Storage Class for more information.

If you’re not using the default, you cannot deploy an alternate provider until after the dkp create cluster is finished. However, it must be determined before Kommander installation.

Next Step

Make the AWS Air-gapped Cluster Self-Managed

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.