Read latest Message from kafka

Sometime it is good to read the latest messages from a kafka topic. The standard flag provides you with the from-beginning behavior. This can be problematic, because when the topic already has a huge number of messages received. So if you want only to create a consumer, which reads the last 10 messages or so, you need to tell the consumer from which offset the receiving should start.

Photo by Denys Nevozhai on Unsplash

Offset

Kafka stores for each consumer a specific offset. This is like a bookmark for the consumer. The broker tracks this for every consumer/consumer group. So when adding a new consumer this consumer by default can be set to latest, or first. Both of these options are not helping in our case.

The first will not show any old message and the other will start showing all, which can be quite many.

Getting the Offset for a Topic

So what we need is the max offset for a topic and a partition. If we want to find a specific message we might need to check on multiple partitions, because we might not have an idea to which the message was routed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$> ./kafka-run-class.sh kafka.tools.GetOffsetShell \
    --broker-list \
    localhost:9092 \
    --topic mytopic

mytopic:2:900020
mytopic:1:900040
mytopic:0:900060
mytopic:3:900080

Now we have the offset for all four partitions and we can create a consumer, which has a starting offset max - 10.

Reading messages from a specific Offset

The kafka console consumer provides a parameter to specify the starting point of the consumer. In our case this will be the max from the partition minus 10. Here we for example the last ten messages from partition 0.

1
2
3
4
5
6

$> ./kafka-console-consumer.sh \
    --bootstrap-server localhost:9092 \
    --topic mytopic \
    --offset 900050 \
    --partition 0