Kafka in a Project

Deploying Kafka in a project

Get Started

To get started with creating and managing a Kafka Cluster in a project, you first need to deploy the Kafka operator and the ZooKeeper operator in the workspace where the project exists.

After deploying the Kafka operator, create Kafka Clusters by applying a KafkaCluster custom resource on each attached cluster in a project’s namespace. Refer to the Kafka operator repository for examples of the custom resources and their configurations.

Example Deployment

If you need to manage these custom resources across all clusters in a project, it is recommended you use Project Deployments which enables you to leverage GitOps to deploy the resources. Otherwise, you will need to create the custom resources manually in each cluster.

This example deployment walks you through first deploying a ZooKeeper cluster and then a Kafka cluster in a project namespace. The result of this procedure is a running ZooKeeper cluster and Kafka cluster ready for use in your project’s namespace.

Set the PROJECT_NAMESPACE environment variable to the name of your project’s namespace:
CODE
```
export PROJECT_NAMESPACE=<project namespace>
```

Create a ZooKeeper Cluster custom resource in your project’s namespace:

CODE

kubectl apply -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: zookeeper
  namespace: ${PROJECT_NAMESPACE}
spec:
  replicas: 1
EOF

Check the status of your ZooKeeper cluster using kubectl:
CODE
```
kubectl -n ${PROJECT_NAMESPACE} get zookeeperclusters
```
Download the sample Kafka Cluster file.

Update the following attribute zkAddresses, replacing zookeeper-client.zookeeper:2181 with zookeeper-client.<project namespace>:2181. You can use sed to update the file:

MacOS sed

CODE

# If you are using sed that comes installed on macOS
sed -i '' "s/zookeeper-client.zookeeper:2181/zookeeper-client.${PROJECT_NAMESPACE}:2181/g" simplekafkacluster.yaml

GNU sed

CODE

# If you are using GNU sed
sed -i "s/zookeeper-client.zookeeper:2181/zookeeper-client.${PROJECT_NAMESPACE}:2181/g" simplekafkacluster.yaml

Verify the file contains the following line:

CODE

...
zkAddresses:
    - "zookeeper-client.<project namespace>:2181"
...

Apply the KafkaCluster to your project’s namespace:

CODE

kubectl apply -n ${PROJECT_NAMESPACE} -f simplekafkacluster.yaml

Check the status of your Kafka cluster using kubectl:
CODE
```
kubectl -n ${PROJECT_NAMESPACE} get kafkaclusters 
```

With both the ZooKeeper cluster and Kafka cluster running in your project’s namespace, refer to the Kafka Operator documentation for information on how to test and verify they are working as expected in. When performing those steps, ensure you substitute: zookeeper-client.<project namespace>:2181 anywhere that the zookeeper client address is mentioned.

Delete Kafka Custom Resources

Follow these steps to delete the Kafka custom resources.

View all Kafka resources in the cluster:

CODE

kubectl get kafkaclusters -A
kubectl get kafkausers -A
Kubectl get kafkatopics -A

Delete a KafkaCluster example:

CODE

kubectl -n ${PROJECT_NAMESPACE} delete kafkacluster <name of KafkaCluster>