Kafka in a Workspace

Information about the Kafka Operator

Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. The Kafka Operator is a Kubernetes operator to automate provisioning, management, autoscaling, and operations of Apache Kafka clusters deployed to Kubernetes. It works by watching custom resources, such as KafkaClusters, KafkaUsers, and KafkaTopics, to provision underlying Kubernetes resources (i.e StatefulSets) required for a production-ready Kafka Cluster.

Install

Follow these steps to install the Kafka operator in a workspace. This procedure results in a Kafka operator running in a workspace namespace, ready to manage and create Kafka clusters in any project namespaces. See the Kafka in a Project custom resource documentation for more information on creating Kafka clusters.

Only install the Kafka operator once per workspace.

1. Follow the generic installation instructions for workspace catalog applications on the Application Deployment page.

2. Within the AppDeployment, update the appRef to specify the correct kafka-operator App. You can find the appRef.name by listing the available Apps in the workspace namespace:

   CODE

   kubectl get apps -n ${WORKSPACE_NAMESPACE}

Refer to the Kafka operator Helm Chart documentation for details on custom configuration for the operator.

If you use a custom version of KafkaCluster with cruise.control, ensure you use the custom resource image version 2.5.123 in the .cruiseControlConfig.image field for both air-gapped and non-air-gapped environments.

Using a custom image for a zookeeper cluster

To avoid the critical CVEs associated with the official kafka image in version v0.25.1, a custom image must be specified when creating a zookeeper cluster.

Specify the following custom values in KafkaCluster CRD:

• .spec.clusterImage to ghcr.io/banzaicloud/kafka:2.13-3.4.1

• .spec.cruiseControlConfig.initContainers[*].image to ghcr.io/banzaicloud/cruise-control:2.5.123

Uninstall via the CLI

Uninstalling the Kafka operator does not affect existing KafkaCluster deployments. After uninstalling the operator, you must manually remove any remaining Custom Resource Definitions (CRDs) from the operator.

1. Delete all of the deployed Kafka custom resources (see Deleting Kafka Custom Resources).

2. Uninstall a Kafka operator AppDeployment:

   CODE

   kubectl -n <workspace namespace> delete AppDeployment <name of AppDeployment>

3. Remove Kafka CRDs:

   NOTE: The CRDs are not finalized for deletion until you delete the associated custom resources.

   CODE

   kubectl delete crds kafkaclusters.kafka.banzaicloud.io kafkausers.kafka.banzaicloud.io kafkatopics.kafka.banzaicloud.io

Resources

• Kafka Operator Documentation

• Kafka Operator GitHub Repository

• Sample Kafka Operator Custom Resources

• Apache Kafka Documentation