introduction-to-apache-kafka

1 © Hortonworks Inc. 2011 – 2017 All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
はじめよう！
Apache Kafkaでリアルタイムデータ処理
Yifeng Jiang
Solutions Engineering Lead
September 6, 2017

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About Me
Ã 蒋逸峰 (しょういつほう / Yifeng Jiang)
Ã Solutions Engineering Lead, NAPAC, Hortonworks
– Hadooper since 2009
– HBase book author
– Software engineer, cloud, PaaS, DevOps
Ã Jogger, hiker
Ã Twitter: @uprush

DATA AT REST
DATA IN
MOTION
ACTIONABLE
INTELLIGENCE
Modern IoT Data Applications
PERISHABLE
INSIGHTS
HISTORICAL
INSIGHTS
INTERNET
OF
ANYTHING
Hortonworks
DataFlow
Hortonworks
Data Platform
Hortonworks Delivers
Connected Data Platforms

Introduction to
Apache Kafka

Apache Kafka
Ã Distributed messaging systems
– Real-time
– Scalable to handle large data volume
– Low Latency
– Fault tolerant
Ã Originated at LinkedIn
– Aimed at solving data movement across systems
– Scala and Java
– Open Source (Apache 2.0)
– Adapted at many companies

Key Concepts and Terminology

Kafka: Anatomy of a Topic
Partition
0
Partition
1
Partition 2
0 0 0
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5
6 6 6
7 7 7
8 8 8
9 9 9
10 10
11 11
12
Writes
Old
New
Ã Messages (logs) are stored on broker’s
local disk
Ã Messages are appended to log file
Ã Log Retention – time and size based

Kafka Replication
Ã Partition has replicas – Leader replica, Follower replicas
Ã Replicas are distributed to multiple brokers
Ã Leader maintains in-sync-replicas (ISR)
https://www.slideshare.net/junrao/kafka-replication-apachecon2013

Kafka Producer
• Create a new message and publish to a Topic and Partition
• Original messages are partitioned and then split into batches
• Each split batch is sent to leader broker (and then replicated to ISR)
• Each send is acknowledged by either leader broker and/or all ISR
p3 p2 p1 p2 p1m5 m4 m3 m2 m1
Broker-0
P0.R0 (L)
P1.R0
Broker-1
P0.R1
P2.R1 (L)
Broker-2
P1.R2 (L)
P2.R2
Topic with 3 partition and Replica factor 2
App Producer Lib
partitioner Split
batch

Kafka Consumer
Ã Consumers pull data from brokers
Ã Consumer apps have to keep track of the topic-partition offset read
Ã Consumer Groups: Allow multiple hosts to form a group to access a topic
– Max parallelism – determined by topic partitions
Broker-0
P3
Broker-1
P1 P2
C1 C2
Consumer Group - 1
C3 C4
Consumer Group - 2
C5 C6
P0

Kafka – Why Kafka is fast
Fast Writes
Writes are appends to file system
Partitions improve performance and throughput
Uses OS buffer cache
Lots of memory on the machine helps
Fast Reads
Hot data sits in memory, most time data is served without disk I/O
File descriptor to socket descriptor efficient transfer
Linux sendfile(), JVM transferTo() implementation
Why Performance?
Disk flushes are delayed
Durability is guaranteed via replication
When consumers are reading the latest data, it reads from page cache

Kafka & Real-time System

1
3
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Kafka at Scale
http://events.linuxfoundation.jp/sites/events/files/slides/Kafka%20At%20Scale.pdf

1
4

1
5
Use Case: Connected Car
https://azure.microsoft.com/ja-jp/blog/announcing-public-preview-of-apache-kafka-on-
hdinsight-with-azure-managed-disks/

1
6
Real-time System Building Blocks
• Streams
– An unbounded sequence of messages, events, information packets or tuples (named list of
values)
• Data Pipe – Message/Information bus
– Decouple publishers (providers) and consumers (subscribers)
– Scalability, Centralized, Distributed
• Stream Processing
– Semantics (operations and processing primitives)
– Stateless or with state
• Low Latency Storage
– NoSQL database

1
7
Real-time System Building Blocks
• Streams -- Nifi, Fluentd, etc.
– An unbounded sequence of messages, events, information packets or tuples (named list of
values)
• Data Pipe – Message/Information bus -- Kafka
– Decouple publishers (providers) and consumers (subscribers)
– Scalability, Centralized, Distributed
• Stream Processing -- Storm, Spark Streaming, etc.
– Semantics (operations and processing primitives)
– Stateless or with state
• Low Latency Storage -- HBase, Redis, Druid, etc.
– NoSQL database

1
8
Apache Metron:
Real-time Big Data Cyber Security
powered By Kafka

Data Services and Integration Layer
ModulesReal-time Processing
Cyber Security Engine
Telemetry
Parsers
Apache Metron
Telemetry Ingest Buffer
Telemetry
Data Collectors
Real-time
Enrich / Threat
Intel Streams
Performance
Network
Ingest
Probes
/ OtherMachine Generated Logs
(AD, App / Web Server,
firewall, VPN, etc.)
Security Endpoint Devices
(Fireye, Palo Alto,
BlueCoat, etc.)
Network Data
(PCAP, Netflow, Bro, etc.)
IDS
(Suricata, Snort, etc.)
Threat Intelligence Feeds
(Soltra, OpenTaxi,
third-party feeds)
Telemetry
Data Sources
Data Vault
Real-Time Search
Evidentiary Store
Threat Intelligence
Platform
Model as a Service
Community
Models
Data Science
Workbench
PCAP Forensics
Threat
IntelligenceEnrichment
Indexers
and WriterProfiler Alert Triage
Cyber Security
Stream Processing Pipeline

Metron Architecture – Real-time System Built on Kafka
Parsers
Kafka enrichments topic
Kafka indexing topic

Metron Architecture – Real-time System Built on Kafka
Kafka indexing topic
Metron UI and Dashbaords

Understood,
but that looks difficult…

2
3
Hortonworks DataFlow (HDF) 3.0

2
5
Introducing Hortonworks Streaming Analytics Manager (SAM)
Streaming Analytics
Manager
Design, develop, deploy and
manage streaming analytics
app with a drag-and-drop
paradigm

2
6
Introducing Hortonworks Schema Registry
A shared repository for schemas
allowing applications to save,
retrieve and reuse schemas and
flexibly interact with each other

2
7
Centralized Security with Apache Ranger

Build Your First Streaming Analytics
App in Under 30 Minutes

2
9
Trucking company w/ large fleet of international trucks
A truck generates millions of events for a given
route; an event could be:
§ 'Normal' events: starting / stopping of the vehicle
§ ‘Violation’ events: speeding, excessive acceleration
and breaking, unsafe tail distance
§ ‘Speed’ Events: The speed of a driver that comes in
every minute.
Company uses an application that monitors truck
locations and violations from the truck/driver in real-
time
Route?
Truck?
Driver?
Analysts query a broad
history to understand if
today’s violations are
part of a larger problem
with specific routes,
trucks, or drivers

3
0

3
1
DEMO

Question?

Data
PlatformConference Tokyo 2017ビッグデータ x IoT / クラウド / AI（人工知能）を利用した
データ駆動型ビジネスの本格的な実現に向けて
2017年10月10日開催
主催：株式会社インプレス
共催：ホートンワークスジャパン株式会社
申し込み・詳細
dataplatform.jp

3
4
THANK YOU
Yifeng Jiang
@uprush

introduction-to-apache-kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to introduction-to-apache-kafka

Similar to introduction-to-apache-kafka (20)

More from Yifeng Jiang

More from Yifeng Jiang (20)

Recently uploaded

Recently uploaded (20)

introduction-to-apache-kafka