Kafka is basically a streaming data service, where one can publish/put/produce a set of data and multiple services can consume those data. In queue(rabbitmq or sqs) based systems, a message once consumed by a service cannot be consumed by others, basically a message is gone from the queue itself.

In Kafka, the messages stay as long as the retention period(period defined to remove messages) in the kafka server/cluster, this can be useful to project a data from throughout its history.

Publish Subscriber#

Kafka uses publisher subscriber pattern, Publisher is basically the one that produce messages and send them to a common place, its just send the message to a common place not to the subscriber/receiver directly, now the subscribers subscribes to specific set of messages that they are interested in with that common place. This common place is called broker.

simple-pub-sub

Kafka is not a simple pub-sub system, alone it involves some complex things like topics, partition, consumer group, replication etc to increase throughput and relability of our data.

How Kafka Works on high level#

lets consider a single system(server) where kafka is installed and we are using it.

At the core, kafka is just one single binary called the broker. This broker has a server(TCP) and a client.

  • The server basically listens on a TCP port and handles some request like ProduceRequest, FetchRequest, JoinGroup, OffsetCommit etc. these are some events that helps producer and consumer to write and read their messages.
  • The client also sits inside the broker, which is only used in a multi-server(broker) setup, where this client is responsible for leader election and replicating the messages across servers(brokers) etc, but in this example case, this is of no use.

This broker also owns a directory on the disk, where it stores all the topics, and each topic splits into multiple partitions(1 by default)

Topics and Partitions#

Topic(eg: log) is basically where all our msgs goes into, kafka basically has a .log file(kinda wal) where it appends end of the file, everytime it receives a new message from the producer. Topic is basically a directory, with name like <topic-name> - <partition-name>/append.log, below is the reason why partition is involved

Consider there is no partition itself, we have only topic log/append.log, now consider we have 5producers seneding messages at a given time for our topic log-events, in this case the broker will write to same append.log file, since we have only one for this topic, so at a given time broker would lock this append.log file in order to write a single msg, so in a 5 concurrent data sending event, at a given time only one can able to write while the others(4) could be waiting, which increase latency and delays in down stream(read systems).

Also single log file may affect the read concurrency as well, consider u have 5consumers all reading the same topic at the same time, now the msgs are read by all the consumers so duplicate processing can happen.

To solve the above problem, kafka has something called partitions, whenever u are configuring a topic, u may also configure the number of partitions for that topic, for eg: lets say we are configuring 3 partitions for our topic log, now our actual disk would look like,

 log-0/append.log
 log-1/append.log
 log-2/append.log

In the above if u can see we have 3 log files for three partitions under same topic, now if any producer coming to write msg into this topic, will basically routes to one of the log file, instead of same. this routing is done by hash(key)%noOfPartitions(3), here the key will be sent by the producer, so each time for the same key the messages would land on the same partition log file.

This key comes from the application(eg: user_id, product_id etc), this also guarantees the order, same key always goes into the same partition within the whole topic.

Producer and Consumer — Not Separate Binaries, Just Libraries#

Now who is producer and consumer, these are not available inside the broker, producer and consumer are libraries available or u can build ur own based on ur application.

  • Producer client: Takes ur key, hashes it to pick the right partition, batches records(messages) to send to the broker to increase throughput.
  • Consumer client: pulls messages from a particular partition for a topic.

Consumer Groups#

A consumer group contains a set of consumers all shares the same group.id, a single topic can be polled by multiple consumer group, within a single consumer group, there can exist n number of consumers each partition within a single topic is assigned exactly to one consumer.

How Consumer Group Works?

  1. Consider we have a consumer group called g1.
  2. Consumer c1 joins the consumer group g1, it first sends JoinGroup(group.id=g1) to broker, the broker now allocates a partition(p1) to this consumer.
  3. Now the Consumer c1 asks the offet(from which line do i need to read the messages from the partition p1) via OffsetFetch, broker returns the last commited offset(this data also available inside kafka logs), if there is no offset or its a brand new partition to read, then consumer can either read from start or the latest(last inserted) msg.
  4. Consumer now starts fetching msgs from that offset.

Without Consumer group, read concurrency between multiple consumers could be difficult. and this consumer group is mostly managed by consumer client library(things like the above 4), also think each consumer is a server/pod worker(in EKS case).