Kafka My Kafka Note v67

Introduction to Apache Kafka
Michael Mühlbeyer
BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Introduction to Apache Kafka
Overview
Architecture
Kafka Connect
Kafka Stream
KSQL
Cluster Installation
2 12.07.18 Introduction to Apache Kafka

Apache Kafka - Overview
Kafka was designed to solve both problems

– Simplifying data pipelines
– Handling streaming data
Created by Linkedin in 2010
Apache open source project since 2012
https://www.quora.com/What-is-the-relation-between-Kafka-the-writer-and-Apache-Kafka-the-
distributed-messaging-system

Apache Kafka - Overview
Kafka decouples data source and

destination systems
– Publish/subscribe architecture
All data sources write their data to the

Kafka cluster
All systems wishing to use the data

read from Kafka

Architecture

Apache Kafka - Architecture
Producer send data to the Producer Producer Producer

cluster
Consumers read data
Kafka Cluster
Brokers are the main Zookeeper
component for storage and Broker 1 Broker 2 Broker 3 Ensemble
messaging
Consumer Consumer Consumer

Basic unit in Kafka is a message

Producers write messages to
Brokers
Consumers read messages from
Brokers
A message is a key-value pair

http://kafka.apache.org/documentation.html#messageformat

Kafka maintains streams of messages in Topics

– Logical representation
– Categorize messages in groups
Developer decides which Topics exist

– Topic is auto created on first use
No limit in number of Topics
One or more Producer can write to the same Topic

Topics are split into partitions
Partitions are distributed across the Brokers
Broker1
Topic
Consumer
1 2 3
Producer Partition 0
Consumer
1 2 3
Partition 1

Apache Kafka - Architecture mytopic
Broker1
P0
1 2 3
Consumer
P2
1 2 3
mytopic
Broker2
P2 Partition 0
Consumer
1 2 3
Producer
P1
1 2 3
mytopic
Broker3 Consumer
P0
1 2 3
P1
1 2 3

Partition 0
Apache Kafka - Producers
Producers write data as messages to the Kafka cluster
Can be written in any language
A command line tool exists to send messages to the cluster

– For testing, debugging, etc
kafka-console-producer --broker-list localhost:9092 --topic test

Apache Kafka - Producers
Producers use a partitioning strategy to assign each message to a Partition

– Default is hash key of message key
Partitioning is used for

– Load balancing à share load across brokers
– semantic partioning àuser-specified key allows locality-sensitive message
processing

Apache Kafka - Brokers
Receive and store the messages send by Producers
A Kafka cluster has typically 3 or more brokers
– Each can handle hundreds of thousands, or millions, of messages per second

Apache Kafka - Brokers
Messages in a Topic are spread across Partitions in different Brokers
Each Partition is stored on the Broker’s disk as one or more log files
– Do not get messed up with log4j logfiles
Each message in the log is identified by its offset number
Retention policy for messages to manage log file growth

– Per Topic

Apache Kafka – Broker
Partitions can be replicated across multiple Brokers

Fault tolerance if one Broker goes down
– Automatically handled by Kafka

Apache Kafka - Consumer
Consumers pull messages from one or more Topics in the cluster
– Consumer will retrieve messages as they are written to the topic
The Consumer Offset keeps track of the latest message read

– Consumer Offset can be changed (if necessary) to reread messages
The Consumer Offset is stored in a special Kafka Topic
A command-line Consumer tool exists to read messages from the cluster
kafka-console-consumer --zookeeper localhost:2181 --topic test --from-beginning
kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic test

Apache Kafka - Consumers

Apache Kafka - Consumers
Different Consumers could read from the same

Topic
Multiple Consumers can be combined in a
Consumer Group
– Scaling
– Each Consumer uses a subset of partitions

Apache Kafka – Cluster overview

Apache Kafka - Zookeeper
Zookeeper is a centralized service which can be used by distributed applications

– Open source Apache project
– Distributed synchronisation
– Enables highly reliable distributed coordination
– Maintains configuration infos
Typically consists of 3 or 5 servers in a quorum

Apache Kafka - Zookeeper
Kafka Brokers use Zookeeper Producer Producer Producer

for important internal features
– Cluster Management
Kafka Cluster
– Failure detection and recovery Zookeeper
Broker 1 Broker 2 Broker 3 Ensemble
– Access Control List storage
Consumer Consumer Consumer

Apache Kafka – Replication and Durability
Each Partition can have replicas

Replicas will be placed on different Brokers
Replicas are spread evenly among the Brokers

Distributed partition leaders

– Ideally spreaded evenly across the cluster

Increase the replication factor for higher durability
For auto-created Topics
– default.replication.factor (Default: 1)
– Configuration on each broker (server.properties )
For manually created topics
kafka-topics --create --zookeeper zk_host:2181 --partitions 2 -

-replication-factor 3 --topic my_topic

Producers can control acknowledgement setting
Value Latency Durability Description

0 No network delay low Producer doesn‘t wait for leader
1 (default) 1 network roundtrip medium Producer waits for Leader

Leader sends ack
No wait for follower
all(-1) 2 network roundtrips high Producer waits for Leader

Leader sends ack when all In-Sync replicas
have send the ack

Kafka Connect

Kafka Connect
Source: https://www.confluent.io/

Kafka Connect
Framework for streaming data between Kafka and other systems
– Open source
Useful for
– Stream an entire SQL database to Kafka
– Stream Kafka topics into HDFS
Kafka Connect has benefits over „do-it-yourself“ Producers and Consumers
– Tested Connectors

Kafka Connect

Kafka Connect
Source: https://www.confluent.io/product/connectors/

Kafka Streams

Kafka Streams
Powerful easy-to use Java library

Part of open source Apache Kafka
Build your own stream processing applications that are
– Scalable
– Fault-tolerant
– Stateful
– Distributed
– able to handle late-arriving, out-of-order data

Kafka Streams

Kafka Streams – Unix
Kafka Connect Kafka Streams Kafka Connect
cat < in.txt | grep "apache“ | tr a-z A-Z > out.txt
Kafka

Kafka Streams

Kafka Streams - Topology

Kafka Streams – Partition and Tasks

Kafka Streams
Reading data from Kafka
Writing data to Kafka

Kafka Streams

KSQL

KSQL
Open Source
Enables stream processing with zero coding required
The simplest way to process streams of data in real-time
Powered by Kafka: scalable, distributed, battle-tested
All you need is Kafka

KSQL
Realtime Analytics with Kafka and KSQL
Swingbench
Orders Logins
Kafka
Oracle
Kafka 
Elasticsearch
Kafka Connect Connect
Txn log
Goldengate
Kafka Streams
API
KSQL

KSQL
Declare Table based on Topics
ksql> CREATE TABLE CUSTOMERS

(CUSTOMER_ID INT,
CUST_FIRST_NAME STRING,
CUST_LAST_NAME STRING, CUSTOMER_CLASS STRING)
WITH
(KAFKA_TOPIC=‚my_ora_stream',
VALUE_FORMAT='JSON');

KSQL
Query the live stream of Kafka
ksql> SELECT CUSTOMER_ID,

CUST_FIRST_NAME,
CUST_LAST_NAME,
CUSTOMER_CLASS
FROM CUSTOMERS
LIMIT 3;
75003 | karl | greene | Occasional
75010 | samuel | cook | Prime
75012 | paul | taylor | Occasional

Kafka powered ETL platform

Confluent Platform

Confluent
Confluent Platform
Platform: Enterprise Streaming based on Apache Kafka®
Database Changes Log Events loT Data Web Events …
Confluent Platform
Data  Real-time Applications
Integration
Monitoring & Administration
Confluent Control Center | Security
Confluent Platform
Hadoop Transformations
Operations
Replicator | Auto Data Balancing
Database Custom Apps
Data Compatibility
Schema Registry
Analytics
Data Warehouse
Development and Connectivity

Clients | Connectors | REST Proxy | KSQL | CLI Monitoring
CRM
Apache Kafka®
… …
Core | Connect API | Streams API
Apache Open Source Confluent Open Source Confluent Enterprise
Confidential 68

Cluster Installation

Cluster installation
Install current Java JDK
sudo yum install java-1.8.0-openjdk -y
Install Confluent
sudo rpm --import

http://packages.confluent.io/rpm/3.3/archive.key

Install Confluent Packages
sudo vi /etc/yum.repos.d/confluent.repo
[Confluent.dist]
name=Confluent repository (dist)
baseurl=http://packages.confluent.io/rpm/3.3/7
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/3.3/archive.key
enabled=1
[Confluent]
name=Confluent repository
baseurl=http://packages.confluent.io/rpm/3.3
gpgcheck=1
gpgkey=http://packages.confluent.io/rpm/3.3/archive.key
enabled=1

Install the Packages
sudo yum install confluent-platform-oss-2.11
For a lab environment the above setting should be fine no need to change anything.
Start zookeeper
sudo zookeeper-server-start /etc/kafka/zookeeper.properties

Start Kafka Broker
sudo kafka-server-start /etc/kafka/server.properties
Cluster is ready to use
sudo kafka-topics --list --zookeeper localhost:2181

_confluent.support.metrics
__consumer_offsets

http://kafka.apache.org/
https://www.confluent.io/
https://www.confluent.io/blog/
https://www.confluent.io/product/ksql/
http://blog.muehlbeyer.net/index.php/apache-kafka-installation-linux/

Michael Mühlbeyer
Senior Consultant
Tel. +49 162 295 96 96
michael.muehlbeyer@trivadis.com

Kafka My Kafka Note v67

Uploaded by

Kafka My Kafka Note v67

Uploaded by

Introduction to Apache Kafka

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

2 12.07.18 Introduction to Apache Kafka

Kafka was designed to solve both problems

Created by Linkedin in 2010

Apache open source project since 2012

3 12.07.18 Introduction to Apache Kafka

Kafka decouples data source and

All data sources write their data to the

All systems wishing to use the data

4 12.07.18 Introduction to Apache Kafka

5 12.07.18 Introduction to Apache Kafka

Producer send data to the Producer Producer Producer

Consumer Consumer Consumer

6 12.07.18 Introduction to Apache Kafka

Basic unit in Kafka is a message

7 12.07.18 Introduction to Apache Kafka

8 12.07.18 Introduction to Apache Kafka

Kafka maintains streams of messages in Topics

Developer decides which Topics exist

No limit in number of Topics

One or more Producer can write to the same Topic

9 12.07.18 Introduction to Apache Kafka

10 12.07.18 Introduction to Apache Kafka

11 12.07.18 Introduction to Apache Kafka

Producers write data as messages to the Kafka cluster

Can be written in any language

A command line tool exists to send messages to the cluster

kafka-console-producer --broker-list localhost:9092 --topic test

12 12.07.18 Introduction to Apache Kafka

Producers use a partitioning strategy to assign each message to a Partition

Partitioning is used for

13 12.07.18 Introduction to Apache Kafka

14 12.07.18 Introduction to Apache Kafka

Retention policy for messages to manage log file growth

15 12.07.18 Introduction to Apache Kafka

Partitions can be replicated across multiple Brokers

16 12.07.18 Introduction to Apache Kafka

The Consumer Offset keeps track of the latest message read

A command-line Consumer tool exists to read messages from the cluster

kafka-console-consumer --zookeeper localhost:2181 --topic test --from-beginning

kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic test

17 12.07.18 Introduction to Apache Kafka

18 12.07.18 Introduction to Apache Kafka

Different Consumers could read from the same

19 12.07.18 Introduction to Apache Kafka

20 12.07.18 Introduction to Apache Kafka

Zookeeper is a centralized service which can be used by distributed applications

Typically consists of 3 or 5 servers in a quorum

21 12.07.18 Introduction to Apache Kafka

Kafka Brokers use Zookeeper Producer Producer Producer

Consumer Consumer Consumer

22 12.07.18 Introduction to Apache Kafka

Each Partition can have replicas

23 12.07.18 Introduction to Apache Kafka

Distributed partition leaders

24 12.07.18 Introduction to Apache Kafka

kafka-topics --create --zookeeper zk_host:2181 --partitions 2 -

25 12.07.18 Introduction to Apache Kafka

Value Latency Durability Description

1 (default) 1 network roundtrip medium Producer waits for Leader

all(-1) 2 network roundtrips high Producer waits for Leader

26 12.07.18 Introduction to Apache Kafka

27 12.07.18 Introduction to Apache Kafka

28 12.07.18 Introduction to Apache Kafka

29 12.07.18 Introduction to Apache Kafka

30 12.07.18 Introduction to Apache Kafka

31 12.07.18 Introduction to Apache Kafka

32 12.07.18 Introduction to Apache Kafka

Powerful easy-to use Java library

33 12.07.18 Introduction to Apache Kafka

34 12.07.18 Introduction to Apache Kafka

Kafka Connect Kafka Streams Kafka Connect

cat < in.txt | grep "apache“ | tr a-z A-Z > out.txt

35 12.07.18 Introduction to Apache Kafka

36 12.07.18 Introduction to Apache Kafka