Felpfe Inc.
Close this search box.
call 24/7

+484 237-1364‬

Close this search box.

Monitoring Kafka: Tools and Techniques for Maintaining a Healthy System


For any mission-critical application, monitoring is a non-negotiable requirement. Monitoring helps identify, diagnose, and resolve issues before they become catastrophic. When dealing with Apache Kafka, which is a distributed, high-throughput, and real-time processing system, it is all the more critical to have robust monitoring in place.

In this post, we will explore the tools and techniques to monitor Kafka effectively. You’ll get hands-on knowledge of Kafka’s built-in monitoring capabilities, the JMX interface, and popular open-source monitoring tools like Prometheus and Grafana.

Part 1: Built-In Monitoring Capabilities

Apache Kafka offers built-in command-line tools that allow you to check the status of your Kafka cluster.

Example 1: Check the Status of a Topic

You can use the kafka-topics.sh script to describe the topic and get the status:

bin/kafka-topics.sh --describe --topic my_topic --bootstrap-server localhost:9092

Example 2: Check the Status of the Consumer Group

Similarly, the kafka-consumer-groups.sh script allows you to check the status of consumer groups:

bin/kafka-consumer-groups.sh --describe --group my_group --bootstrap-server localhost:9092

Example 3: Check the Under-replicated Partitions

The kafka-topics.sh script can also be used to check for under-replicated partitions, which can be an indicator of failing brokers:

bin/kafka-topics.sh --describe --under-replicated-partitions --bootstrap-server localhost:9092

Part 2: JMX Interface

Kafka also exposes a Java Management Extensions (JMX) interface that can be used to monitor its internal metrics.

Example 4: Enable JMX Interface

You can enable the JMX interface by setting the KAFKA_JMX_OPTS environment variable:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname= -Dcom.sun.management.jmxremote.rmi.port=9999"

Then, you can start your Kafka server:

bin/kafka-server-start.sh config/server.properties

Example 5: Monitor Kafka Metrics using JConsole

JConsole, which comes with JDK, can be used to connect to the JMX interface and monitor Kafka metrics.

jconsole localhost:9999

Part 3: Monitoring Kafka with Prometheus and Grafana

Prometheus, an open-source monitoring system, and Grafana, an open-source metric analytics & visualization suite, are commonly used for Kafka monitoring.

Example 6: Prometheus Kafka Exporter

Prometheus provides a Kafka exporter, which is a service that scrapes Kafka metrics and makes them available to the Prometheus server. Here is a basic configuration:

  scrape_interval: 15s
  evaluation_interval: 15s

  - job_name: 'kafka'
      - targets: ['localhost:7071'] 

Then, you can start the Kafka exporter:

./kafka_exporter --kafka.server=localhost:9092

Example 7: Prometheus Server

You can then configure the Prometheus server to scrape the metrics exposed by the Kafka exporter. Here’s an example of a Prometheus server configuration:

  scrape_interval:     15s 
  evaluation_interval: 15s 

  - job_name: 'prometheus'
    - targets: ['localhost:9090']

  - job_name: 'kafka'
    - targets: ['localhost:7071']

Then, start the Prometheus server:

./prometheus --config.file=prometheus.yml

Example 8: Grafana Dashboard

Grafana can connect to the Prometheus server and visualize the collected metrics. First, add the Prometheus data source in Grafana:

  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://localhost:9090",
  "access": "proxy"

You can then import a pre-configured Kafka dashboard or create your own.


Apache Kafka, as an integral part of many complex data pipelines, requires effective monitoring to ensure its health and performance. Kafka provides various means for monitoring, from built-in tools to JMX metrics, to integration with open-source monitoring solutions like Prometheus and Grafana.

By now, you should have a solid understanding of the different tools and techniques available for monitoring Kafka. However, always remember that each Kafka deployment is unique, and it’s essential to understand what metrics are important for your particular use case.

About Author
Ozzie Feliciano CTO @ Felpfe Inc.

Ozzie Feliciano is a highly experienced technologist with a remarkable twenty-three years of expertise in the technology industry.

Stream Dream: Diving into Kafka Streams
In “Stream Dream: Diving into Kafka Streams,”...
Talking in Streams: KSQL for the SQL Lovers
“Talking in Streams: KSQL for the SQL Lovers”...
Stream Symphony: Real-time Wizardry with Spring Cloud Stream Orchestration
Description: The blog post, “Stream Symphony:...
Kafka Chronicles: Saga of Resilient Microservices Communication with Spring Cloud Stream
“Kafka Chronicles: Saga of Resilient Microservices...
Tackling Security in Kafka: A Comprehensive Guide on Authentication and Authorization
As the usage of Apache Kafka continues to grow in organizations...
1 2 3 58
90's, 2000's and Today's Hits
Decades of Hits, One Station

Listen to the greatest hits of the 90s, 2000s and Today. Now on TuneIn. Listen while you code.