Networking

Configure networking for Konvoy cluster

This section describes different networking components that come together to form a Konvoy networking stack. It assumes familiarity with Kubernetes networking.

Highly Available Control Plane

Konvoy ships with a highly available control plane, in case of multi-master Kubernetes deployment.

AWS

High availability is provided through the cloud provider’s load balancer.

On-premise

In on-premise deployments, Konvoy ships with Keepalived. Keepalived provides two main functionalities - high availability and load balancing. It uses the VRRP (Virtual Router Redundancy Protocol) to provide high availability. VRRP allows you assign a virtual IP (VIP) to participating machines, where it is active only on one of the machines.

VRRP provides high availability by ensuring that virtual IP is active as long as at least one of the participating machines is active. Konvoy uses Keepalived to maintain high availability of the control plane.

To use Keepalived:

  1. Identify and reserve a virtual IP (VIP) address from the networking infrastructure.

  2. Configure the networking infrastructure so that the reserved virtual IP address is reachable:

  • from all hosts specified in the inventory file.
  • from the computer that is used to deploy Kubernetes.

If the reserved virtual IP address is in the same subnet as the rest of the cluster nodes then nothing more needs to be configured. However, if it is in a different subnet you may need to configure appropriate routes to ensure connectivity with the virtual IP address. Further, the virtual IP address may share an interface with the primary IP address of the interface. In such cases, you must be able to disable any IP or MAC spoofing from the infrastructure firewall.

The following example illustrates the configuration if the reserved virtual IP address is 10.0.50.20:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  kubernetes:
    controlPlane:
      controlPlaneEndpointOverride: "10.0.50.20:6443"
      keepalived:
        interface: ens20f0 # optional
        vrid: 51           # optional

The IP address specified in spec.kubernetes.controlPlane.controlPlaneEndpointOverride is used for the Keepalived VIP. This value is optional if it is already specified in inventory.yaml as part of all.vars.control_plane_endpoint. You can set spec.kubernetes.controlPlane.keepalived.interface to specify the network interface for the Keepalived VIP. This field is optional; if not set, Konvoy automatically detects the network interface to use based on the route to the VIP.

Further, you could set spec.kubernetes.controlPlane.keepalived.vrid to specify the Virtual Router ID used by Keepalived. This field is optional; if not set, Konvoy will randomly pick a Virtual Router ID for you.

Keepalived is enabled by default for on-premise deployment. You can disable it by removing spec.kubernetes.controlPlane.keepalived from the cluster.yaml. This is usually done where there is an on-premise load balancer which could be used to maintain high availability of the control plane.

If you are not setting any of the optional values, set spec.kubernetes.controlPlane.keepalived: {} to enable the default values.

Pod-to-Pod connectivity

Konvoy ships with Calico as the default CNI plugin to provide pod-to-pod connectivity. The .yaml file for the default installation can be viewed here. Konvoy exposes two configurations in cluster.yaml, for Calico - Calico version and PodSubnet. Both these configurations are optional.

By default, Konvoy ships with the latest version of Calico available at the time of Konvoy release. However, you can configure it to a specific version as shown below:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  kubernetes:
    containerNetworking:
      calico:
        version: v3.13.5

Further, the Calico IPV4 pool CIDR can be set via spec.kubernetes.networking.podSubnet in cluster.yaml, as shown below:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  kubernetes:
    networking:
      podSubnet: 192.168.0.0/16
      serviceSubnet: 10.0.51.0/24

Konvoy ships with the default CIDR as 192.168.0.0/16. Make sure that podSubnet does not overlap with serviceSubnet.

Encapsulation

Two ways of encapsulating networking traffic are supported: IP-to-IP and VXLAN. By default, IP-to-IP is enabled:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  kubernetes:
    containerNetworking:
      calico:
        encapsulation: ipip

The following configuration switches it to VXLAN:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  kubernetes:
    containerNetworking:
      calico:
        encapsulation: vxlan

Network Policy

Calico supports a wide range of network policies. It is tightly integrated with Kubernetes network policy. You can use kubectl to configure Kubernetes network policy which would be enforced by Calico. Further, Calico extends Kubernetes network policy through custom CRDs which can be configured using calicoctl. More details about Calico network policy can be found here.

In-cluster BGP Route Reflectors

Calico advertises routes using BGP Protocol with the full node-to-node mesh configured by default. That means all nodes connect to each other which becomes a problem on big clusters. Konvoy has support for in-cluster BGP Route Reflectors. Route Reflectors are essential on clusters with more than 200 nodes. However, we recommend to have Route Reflectors on clusters with more than 100 nodes. When Route Reflector nodes are configured, the full mesh mode is disabled and each node connects only to in-cluster Route Reflectors. That reduces CPU and memory utilization on worker and control-plane nodes. More information about Calico BGP Peers and Route Reflectors can be found here.

Route Reflector node requires at least 2 CPU Core and 4Gb Memory. To enable in-cluster BGP Route Reflectors, add at least two nodes (three nodes are recommended) to the route-reflector node pool and add the following cluster configuration:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  nodePools:
  - name: route-reflector
    labels:
      - key: dedicated
        value: route-reflector
    taints:
      - key: dedicated
        value: route-reflector
        effect: NoExecute

Service Discovery

Konvoy ships with CoreDNS to provide a DNS based service discovery. The default CoreDNS configuration is as shown below:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       upstream
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    loop
    reload
    loadbalance
}

As shown in the above configuration, by default, CoreDNS is shipped with error, health, prometheus, forward, loop, reload, loadbalance plugins enabled. A detailed explanations for these plugins can be found here.

You can modify the CoreDNS configuration by updating the configmap named coredns in kube-system namespace.

Load Balancing

Load Balancing can be addressed in two ways:

  • Load balancing for the traffic within a Kubernetes cluster
  • Load balancing for the traffic coming from outside the cluster

Load balancing for internal traffic

Load balancing within a Kubernetes cluster is exposed through a service of type ClusterIP. ClusterIP is similar to a virtual IP (VIP) which presents a single IP address to the client and load balances the traffic to the backend servers. The actual load balancing happens via iptables rules or ipvs configuration, which are programmed by a Kubernetes component called kube-proxy. By default, kube-proxy runs in iptables mode. It configures iptables to intercept any traffic destined towards ClusterIP and send traffic to the real servers based on the probabilistic iptables rules. Kube-proxy configuration can be altered by updating the configmap named kube-proxy in the kube-system namespace.

Load balancing for external traffic

A Kubernetes service of type LoadBalancer requires a load balancer to connect an external client to your internal service.

AWS

In cloud deployments, the load balancer is provided by the cloud provider.

On-premise

For an on-premise deployment, Konvoy ships with MetalLB.

To use MetalLB for addon load balancing:

  1. Identify and reserve a virtual IP (VIP) address range from the networking infrastructure.

  2. Configure the networking infrastructure so that the reserved IP addresses is reachable:

  • from all hosts specified in the inventory file.
  • from the computer that is used to deploy Kubernetes.

If the reserved virtual IP addresses are in the same subnet as the rest of the cluster nodes, then nothing more needs to be configured. However, if it is in a different subnet then you may need to configure appropriate routes to ensure connectivity with the virtual IP address. Further, the virtual IP addresses may share an interface with the primary IP address of the interface. In such cases, you must disable any IP or MAC spoofing from the infrastructure firewall.

MetalLB can be configured in two modes - Layer2 and BGP.

The following example illustrates the Layer2 configuration in the cluster.yaml configuration file:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  addons:
    addonsList:
    - name: metallb
      enabled: true
      values: |-
        configInline:
          address-pools:
          - name: default
            protocol: layer2
            addresses:
            - 10.0.50.25-10.0.50.50

The number of virtual IP addresses in the reserved range determines the maximum number of services with a type of LoadBalancer that you can create in the cluster.

MetalLB in bgp mode implements only a minimal functionality of BGP. It only advertises the virtual IP to peer BGP agent.

The following example illustrates the BGP configuration in the cluster.yaml configuration file:

kind: ClusterConfiguration
apiVersion: konvoy.mesosphere.io/v1beta1
spec:
  addons:
    addonsList:
    - name: metallb
      enabled: true
      values: |-
        configInline:
          peers:
          - my-asn: 64500
            peer-asn: 64500
            peer-address: 172.17.0.4
          address-pools:
          - name: my-ip-space
            protocol: bgp
            addresses:
            - 172.40.100.0/24

In the above configuration, peers defines the configuration of the BGP peer such as peer ip address and autonomous system number (asn). The address-pools section is similar to layer2, except for the protocol.

Further, MetalLB supports advanced BGP configuration which can be found here.

NOTE: Making a configuration change in cluster.yaml for the metallb addon running konvoy deploy addons may not result in the config change applying. This is intentional behavior. MetalLB will refuse to adopt changes to the ConfigMap that will break existing Services¹. You may force MetalLB to load those changes by deleting the metallb controller pod:

kubectl -n kubeaddons delete pod -l app=metallb,component=controller

Make sure the MetalLB subnet does not overlap with podSubnet and serviceSubnet.

Ingress

Konvoy ships with Traefik as the default ingress controller. The default Traefik helm chart can be viewed here. Traefik creates a service of type Load Balancer. In the cloud, the cloud provider creates the appropriate load balancer. In on-premise deployment, by default, it uses MetalLB. MetalLB can be configured as discussed earlier.

Further, Traefik supports a lot of functionalities such as Name-based routing, Path-based routing, Traffic splitting etc. Details of these functionalities can be viewed here.

Custom Domain

Configure a custom domain for Konvoy…Read More