Today we are going to investigate how to set up a fabric network with kafka and zookeeper. As a beginner, first we need to understand what is kafka and zookeeper.

Background

Apache Kafka

Apache Kafka is an open-source stream-processing software platform⁰.

Kafka is a distributed, horizontally-saclable, fault-tolerant, commit log¹.

Since commit log is a record of transactions, Kafka can easily integrates with blockchain platform. Therefore, I think this is the reason why the first “consensus” of Fabric is utilising Kafka while the next would be based on Raft².

Zookeeper

Zookeeper is a general purpose distribtued process coordination system, it helps to coordinate tasks, manage state, configure, etc accross a distributed system.

Ordering Service

Ordering Service consists of orderers, providing a shared communication channel to clients and peers and offering a broadcast service for messages containing transactions.

OSN (Ordering Service Node/ Orderer Node): connected with the endorsers and peers

Understanding the role of ordering service in fabric network

Fabric transaction flow

Image Source: Hyperledger Fabric: A Distributed Operating System for Permissioned Blockchains

Configurations

For kafka & zookeep, refer to this

Deployment

Disclaimer: this is based on fabric-samples/first-network

FAQ

If kafka is a streaming-process software, why are we calling kafka-consensus in fabric?

First, we need to understand what is consensus. The definition of consensus is a general agreement³ and in the context of blockchain, it means all nodes must agree to the same order of transactions/blocks to achieve a universal shared ledger
Kafka provides the same order of transactions among multiple nodes by a shared queue as ordering nodes send to Kafka transactions and receive from Kafka transactions in the same order.