Introduction

Monitoring your Kafka deployment is essential for tracking performance, detecting issues, and ensuring smooth operation. This guide walks through deploying Kafka on Kubernetes, setting up Prometheus to collect metrics via Kafka Exporter, and visualizing them in Grafana.

Steps We’ll Take:

• Deploy Kafka and Zookeeper using Helm
• Install and configure Kafka Exporter
• Set up Prometheus to scrape Kafka metrics
• Deploy Grafana for visualization
• Simulate Kafka traffic with sample producer and consumer apps

You can find the complete source code on GitHub.

Let’s get started with Kafka deployment.

Deploying Kafka and Zookeeper with Helm

Bitnami provides a well-maintained Kafka Helm chart that simplifies the deployment process. Run the following command to add the Bitnami repository:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Deploy Kafka and Zookeeper:

helm install kafka bitnami/kafka -f values_kafka.yaml

Note: We use a values_kafka.yaml file to configure Kafka with a predefined and visible username and password. This simplifies the setup and makes the tutorial easier to follow. However, in a production environment, sensitive information like usernames and passwords should be managed through secret management tools.

Check that the Kafka pods are running:

Now, let’s create the required Kafka topic for our demo. For this, we will use a Kubernetes Job that automates the topic creation process. The job will run a Kafka command inside a container to create the topic with the specified configuration.

Let’s apply the following job definition file yaml:

kubectl apply -f kafka-create-topic.yaml

You should see a pod for the job execution. Check its logs to verify the topic is created successfully:

Installing Kafka Exporter

Kafka Exporter collects and exposes Kafka metrics in a Prometheus-compatible format, allowing Prometheus to scrape them.

To install the Exporter, first add the Prometheus community repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Then run the installation:

helm upgrade --install kafka-exporter prometheus-community/prometheus-kafka-exporter -f values_kafka_exporter.yaml

Check out the content of the kafka-exporter-values.yaml file we’re using. We are setting the Kafka URL, authentication parameters, and some additional settings to enable the monitoring of consumer lag.

Ensure the Kafka Exporter pod is running:

Let’s verify that it is exposing the metrics properly.

To access the metrics endpoint, forward the Kafka Exporter service port to your local machine:

kubectl port-forward svc/kafka-exporter-prometheus-kafka-exporter 9308:9308

Now, visit http://localhost:9308/metrics in your browser to check the exposed metrics. If you can see the metrics page, Kafka Exporter is working correctly:

Simulating Kafka Activity with Producer and Consumer

To simulate Kafka activity, we will deploy the producer and consumer applications. These applications send and receive messages from a Kafka topic, generating metrics related to consumer groups, offsets, and message throughput.

Here are some code snippets of the producer and consumer implementations so you get an intuition of how the data simulation works.

Producer (producer.py):

producer = KafkaProducer(
	bootstrap_servers=broker,
	sasl_mechanism=sasl_mechanism,
	security_protocol="SASL_PLAINTEXT",
	sasl_plain_username=sasl_username,
	sasl_plain_password=sasl_password
)

event_types = ["INFO", "WARNING", "ERROR"]

while True:
	event_type = random.choice(event_types)
	message = f"{event_type} | Event-{random.randint(1, 1000)} | {datetime.now().isoformat()}"

	producer.send(topic, value=message.encode('utf-8'))
	print(f"Produced: {message}")

	time.sleep(random.uniform(0.2, 1.5))

Consumer (consumer.py):

consumer = KafkaConsumer(
        topic,
        group_id=consumer_group,
        bootstrap_servers=broker,
        enable_auto_commit=True,
        sasl_mechanism=sasl_mechanism,
        security_protocol="SASL_PLAINTEXT",
        sasl_plain_username=sasl_username,
        sasl_plain_password=sasl_password
    )

print(f"Consumer in group '{consumer_group}' consuming topic '{topic}'")

for message in consumer:
	print(f"Group {consumer_group} received: {message.value.decode('utf-8')}")
	time.sleep(random.uniform(0.5, 2))

Dockerizing Producer and Consumer Applications

Before deploying to Kubernetes, the producer and consumer applications must be containerized. Feel free to check the producer and consumer Dockerfiles in the repo.

To build and push these Docker images, go to the src directory and execute the commands below. Replace yourdockerhub with your own Docker Hub username or registry:

docker build -t yourdockerhub/kafka-k8s-monitoring-producer:latest -f producer/Dockerfile .
docker build -t yourdockerhub/kafka-k8s-monitoring-consumer:latest -f consumer/Dockerfile .

docker push yourdockerhub/kafka-k8s-monitoring-producer
docker push yourdockerhub/kafka-k8s-monitoring-consumer

Once the Docker images are ready, you can proceed with the Kubernetes deployments. For that, we use a single deployment YAML file, which includes configurations for the producer and multiple consumer groups.

The deployment defines three replicas for each consumer group, enabling parallel consumers, making the demo a bit more practical, and allowing us to explore statistics like the consumer lag for a particular consumer group and partition.

Running the Producer and Consumer in Kubernetes

Let’s apply the deployment file and check the logs to ensure the producer and consumers function as expected.

kubectl apply -f deployment.yaml

Verify that the pods are running:

To ensure the producer is sending messages and the consumers are receiving them, check the logs for the pods:

kubectl logs -l app=kafka-producer
kubectl logs -l app=kafka-consumer-group-1
kubectl logs -l app=kafka-consumer-group-2

Here is a sample output:

Export the Kafka Exporter port again and visit the metrics page. You should see some more metrics related to our new producers and consumers:

These metrics are handy to identify how far behind a consumer is from the latest message. Based on them, you can decide if scaling consumers or optimizing the processing logic is necessary.

Installing Prometheus

Prometheus collects and stores the metrics. It reads them from the Kafka Exporter and serves as the data source for Grafana dashboards later in the setup.

Use the following command to install Prometheus:

helm install prometheus prometheus-community/prometheus -f values_prometheus.yaml

Inspect the concrete Prometheus setup in the values_prometheus.yaml file.

Check if Prometheus pods are running:

Export Prometheus service port to view collected metrics:

kubectl port-forward svc/prometheus-server 9090:80

Visit the Prometheus URL and explore the metrics:

Installing Grafana

Install Grafana using the following commands:

helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -f values_grafana.yaml

The values_grafana.yaml file configures Grafana with Prometheus as the default data source.

Check if Grafana pods are running:

Check the exposed node port of the Grafana installation from the grafana service:

And visit the Grafana home page:

Login with the default credentials we set during the installation:

• Username: admin
• Password: admin123

Which will log you into Grafana:

Setting Up Grafana Dashboards for Kafka Monitoring

Once Grafana is up and running, the next step is to import some pre-built dashboards.

Once logged in, navigate to the Dashboards section and click New > Import. Import the following dashboard with ID 7589, which is the Kafka Exporter Overview dashboard:

After successful import, the Kafka Exporter dashboard should display Kafka metrics.

Feel free to add more dashboards that are publicly available. For example, one useful dashboard is the Node Exporter Dashboard (id = 1860), displaying various system metrics.

Summary

This article covered how to monitor Kafka on Kubernetes by setting up Prometheus and Grafana to track key metrics like message throughput and consumer lag. We deployed Kafka and Zookeeper using Helm and installed Kafka Exporter to expose the metrics. Prometheus was configured to scrape these metrics, and Grafana was set up to display them through dashboards.

Thanks for reading, and see you next time!

Resources

1. GitHub repo