Kafka in a Project

These pages refer to DKP Enterprise and DKP Gov Advanced products.

Deploying Kafka in a project

Prerequisites

To get started with creating and managing a Kafka Cluster in a project, you must:

Deploy the Kafka operator in the workspace where the project exists.
Deploy the ZooKeeper operator in the workspace where the project exists.
Deploy Zookeeper in a Project in the same project where you want to enable Kafka.

Get Started

After deploying the Kafka operator, create Kafka Clusters by applying a KafkaCluster custom resource on each attached cluster in a project’s namespace. Refer to the Kafka operator repository for examples of the custom resources and their configurations.

Example Deployment

If you need to manage these custom resources across all clusters in a project, it is recommended you use Project Deployments which enables you to leverage GitOps to deploy the resources. Otherwise, you will need to create the custom resources manually in each cluster.

This example deployment walks you through deploying a Kafka cluster in a project namespace. The result of this procedure is a running ZooKeeper cluster and Kafka cluster ready for use in your project’s namespace.

Ensure you deployed Zookeeper in a Project.
Set the PROJECT_NAMESPACE environment variable to the name of your project’s namespace:
CODE
```
export PROJECT_NAMESPACE=<project namespace>
```
Obtain the Kafka Operator version you deployed in the workspace:
Replace <target_namespace> with the namespace where you deployed Kafka:
CODE
```
kubectl get appdeployments.apps.kommander.d2iq.io -n <target_namespace> -o template="{{ .spec.appRef.name }}" kafka-operator
```
The output prints the Kafka Operator version.
Use the Kafka Operator version to download the simplekafkacluster.yaml file you require:

In the following URL, replace /v0.25.1/ with the Kafka version you obtained in the previous step and download the file.

https://raw.githubusercontent.com/banzaicloud/koperator/v0.25.1/config/samples/simplekafkacluster.yaml

In order to use a CVE-free kafka image, set clusterImage value to ghcr.io/banzaicloud/kafka:2.13-3.4.1 (similarly to workspace installation in Kafka in a Workspace | Using-a-custom-image-for-a-zookeeper-cluster).

Open and edit the downloaded file to use the correct Zookeeper Cluster address:
Replace <project_namespace> with the target project namespace.
CODE
```
zkAddresses:
    - "zookeeper-client.<project_namespace>:2181"
```

Apply the KafkaCluster configuration to your project’s namespace:
CODE
```
kubectl apply -n ${PROJECT_NAMESPACE} -f simplekafkacluster.yaml
```

Check the status of your Kafka cluster using kubectl:

CODE

kubectl -n ${PROJECT_NAMESPACE} get kafkaclusters

The output should look similar to this:

CODE

NAME    CLUSTER STATE    CLUSTER ALERT COUNT   LAST SUCCESSFUL UPGRADE   UPGRADE ERROR COUNT   AGE
kafka   ClusterRunning   0                                               0                     79m

With both the ZooKeeper cluster and Kafka cluster running in your project’s namespace, refer to the Kafka Operator documentation for information on how to test and verify they are working as expected in. When performing those steps, ensure you substitute: zookeeper-client.<project namespace>:2181 anywhere that the zookeeper client address is mentioned.

Delete Kafka Custom Resources

Follow these steps to delete the Kafka custom resources.

View all Kafka resources in the cluster:

CODE

kubectl get kafkaclusters -A
kubectl get kafkausers -A
Kubectl get kafkatopics -A

Delete a KafkaCluster example:

CODE

kubectl -n ${PROJECT_NAMESPACE} delete kafkacluster <name of KafkaCluster>