What is the difference between RDD, Dataframe and Dataset in Spark?
August 16, 2021Error initializing SparkContext with java.net.BindException
August 20, 2021In Kafka, consumers connected to a topic are grouped into consumer groups. So a consumer group will have a list of consumers. So to get the list of consumers connected to a topic we need a list of all consumer groups for that topic.
So getting a list of consumers connected to a Kafka topic is a 2 step process.
- Get the list of consumer groups for a topic
- Get the list of consumers for a consumer group
Get the list of consumer groups for a topic
Use kafka-consumer-groups.sh to list all consumer groups. Note that the below command will list all the consumer groups for all topics managed by the cluster.
bin/kafka-consumer-groups.sh --list --bootstrap-server <kafka-broker>:9092
Get the list of consumers for a consumer group
Issue the below command once you narrow down the consumer group(s) you are interested in to list the consumers part of the consumer group.
bin/kafka-consumer-groups --bootstrap-server <kafka-broker>:9092 --describe --group sample-consumer-group
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.
