Kafka overview and use cases

•Download as PPTX, PDF•

3 likes•957 views

Overview of kafka, how it works, components of kafka, use cases. Kafka at LinkedIn. Download the slides to see animations explaining how the components fit. This was presented on kafka meetup help on June 11, 2016 @LinkedinBangalore ofc

Technology

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Kafka - Overview
Indrajeet Kumar
Site Reliability Engineer at LinkedIn

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved. 2
So what is it?
It is a high-throughput, low-latency messaging system

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
And who uses it?
3

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
What for?
•Messaging
•Website Activity Tracking
•Metrics
•Log Aggregation
•Stream Processing
•For fun ;)
4

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
So how does it work?
▪ Components
– Producer
– Broker
▪ Topic
▪ Partition
– Consumer
5

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Broker
producer producer producer
B2B1P1 P2P1
R P2
R
6
consumer consumer consumer

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
The consumer
7
consumer
B2B1P1 P2
B3P3
C1 C2
P1
R

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
The Producer
8
Producer
B2B1P1 P2
B3P3
P1 P2
P1
R

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Attributes of a Kafka Cluster
▪ Durable
▪ Scalable
▪ Low Latency
▪ Finite Retention
▪ No single point of failure
9

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Kafka At LinkedIn
▪ Multiple Datacenters, Multiple Clusters
▪ Mirroring between clusters
▪ Message Types
– Metrics
– Tracking
– Queuing
▪ Data transport from applications to Hadoop, and back
10

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Some numbers!
▪ 1800+ Broker machines
▪ 79K+ Topics
▪ 1.1M+ Partitions
▪ 1.3 Trillion messages per day
▪ 330 Terabytes in/day
▪ 1.2 Petabytes out/day
▪ Peak load for a single cluster
– 2 million messages/sec
– 4.7 Gigabits/sec inbound
– 15 Gigabits/sec outbound
11

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Questions
12

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.

What's hot

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...HostedbyConfluent

Best Practices for Streaming IoT Data with MQTT and Apache Kafka®confluent

More Datacenters, More ProblemsTodd Palino

Help, My Kafka is Broken! (Emma Humber & Gantigmaa Selenge, IBM) Kafka Summit...HostedbyConfluent

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, JupiterHostedbyConfluent

Deep Dive Series #3: Schema Validation + Structured Audit Logsconfluent

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...HostedbyConfluent

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...HostedbyConfluent

Creating an Elastic Platform Using Kafka and Microservices in OpenShift confluent

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...HostedbyConfluent

Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...HostedbyConfluent

How Zillow Unlocked Kafka to 50 Teams in 8 months | Shahar Cizer Kobrinsky, Z...HostedbyConfluent

Elastically Scaling Kafka Using Confluentconfluent

What is Apache Kafka and What is an Event Streaming Platform?confluent

How Much Can You Connect? | Bhavesh Raheja, Disney + HotstarHostedbyConfluent

From legacy systems to microservices and back | Andera Gioia, QuantycaHostedbyConfluent

How Yelp Leapt to Microservices with More than a Message Queueconfluent

Multi tier, multi-tenant, multi-problem kafkaTodd Palino

AWS Api Gateway by Łukasz Marchewka ScalaccScalac

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

What's hot (20)

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

Best Practices for Streaming IoT Data with MQTT and Apache Kafka®

More Datacenters, More Problems

Help, My Kafka is Broken! (Emma Humber & Gantigmaa Selenge, IBM) Kafka Summit...

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter

Deep Dive Series #3: Schema Validation + Structured Audit Logs

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...

Creating an Elastic Platform Using Kafka and Microservices in OpenShift

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...

Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...

How Zillow Unlocked Kafka to 50 Teams in 8 months | Shahar Cizer Kobrinsky, Z...

Elastically Scaling Kafka Using Confluent

What is Apache Kafka and What is an Event Streaming Platform?

How Much Can You Connect? | Bhavesh Raheja, Disney + Hotstar

From legacy systems to microservices and back | Andera Gioia, Quantyca

How Yelp Leapt to Microservices with More than a Message Queue

Multi tier, multi-tenant, multi-problem kafka

AWS Api Gateway by Łukasz Marchewka Scalacc

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

Viewers also liked

Event Driven Architectures with Apache Kafka on HerokuHeroku

Real time Analytics with Apache Kafka and Apache SparkRahul Jain

Apache kafkaRahul Jain

Introduction to Apache KafkaJeff Holoman

Developing Real-Time Data Pipelines with Apache KafkaJoe Stein

Introduction to Kafka and ZookeeperRahul Jain

Advanced Threat Detection on Streaming DataCarol McDonald

Apache Kafka DC Meetup: Replicating DB Binary Logs to KafkaMark Bittmann

I'm No Hero: Full Stack Reliability at LinkedInTodd Palino

Apache sqoop with an use caseDavin Abraham

Works of site reliability engineerShohei Kobayashi

Stephen McHenry - Chanecellor of Site Reliability Engineering, GoogleIE Group

Cassandra Summit 2014: Turkcell Curio, Real-Time Targeted Mobile Marketing Pl...DataStax Academy

Discover.hdp2.2.storm and kafka.finalHortonworks

Kafka for DBAsGwen (Chen) Shapira

SRE ToolsGurbakash Phonsa

Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014Chen-en Lu

SRE From ScratchGrier Johnson

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

SRE - drupal day aveiro 2016Ricardo Amaro

Viewers also liked (20)

Event Driven Architectures with Apache Kafka on Heroku

Real time Analytics with Apache Kafka and Apache Spark

Apache kafka

Introduction to Apache Kafka

Developing Real-Time Data Pipelines with Apache Kafka

Introduction to Kafka and Zookeeper

Advanced Threat Detection on Streaming Data

Apache Kafka DC Meetup: Replicating DB Binary Logs to Kafka

I'm No Hero: Full Stack Reliability at LinkedIn

Apache sqoop with an use case

Works of site reliability engineer

Stephen McHenry - Chanecellor of Site Reliability Engineering, Google

Cassandra Summit 2014: Turkcell Curio, Real-Time Targeted Mobile Marketing Pl...

Discover.hdp2.2.storm and kafka.final

Kafka for DBAs

SRE Tools

Apache Kafka: A high-throughput distributed messaging system @ JCConf 2014

SRE From Scratch

Apache Kafka - Scalable Message-Processing and more !

SRE - drupal day aveiro 2016

Similar to Kafka overview and use cases

Enterprise Kafka: Kafka as a ServiceTodd Palino

Kafka at Scale: Multi-Tier ArchitecturesTodd Palino

API Management - Practical Enterprise Implementation ExperienceCapgemini

5 Pillars of Building Enterprise0grade APIsWSO2

Anypoint new features_coimbatore_mule_meetupMergeStack

MuleSoft Meetup Valletta 1.0Anastasiia Linnas

Kafka at Peak PerformanceTodd Palino

The Muda, Mura and Muri of DevOpsSanjeev Sharma

Light Speed Integrations With Anypoint Flow DesignerAaronLieberman5

Linked in multi tier, multi-tenant, multi-problem kafkaNitin Kumar

Exposing Business Functionalities with SOA, Integration and API ManagementWSO2

MuleSoft Surat Meetup#51 - API Monitoring - Through a New LensJitendra Bafna

Monitoring Consumer Digital ExperienceThousandEyes

MWLUG 2017: BP107 Plotting the Path for your IBM Domino Applications Franziska Tanner

Effective API DesignBansilal Haudakari

First Monterrey MuleSoft Meetup - Introduction to MuleSoft.Ricardo Rodríguez

Deliver Unrivaled End-User Experience With Confidence - How Synthetic Monitor...DevOps.com

Building an API Factory: Turn your APIs into ProductsNuwan Dias

Scribe Online CDK & Connector DevelopmentCloudFronts Technologies LLP.

API Management: Unlock the Value of Your Unique Assets with a Robust APISoftware AG

Similar to Kafka overview and use cases (20)

Enterprise Kafka: Kafka as a Service

Kafka at Scale: Multi-Tier Architectures

API Management - Practical Enterprise Implementation Experience

5 Pillars of Building Enterprise0grade APIs

Anypoint new features_coimbatore_mule_meetup

MuleSoft Meetup Valletta 1.0

Kafka at Peak Performance

The Muda, Mura and Muri of DevOps

Light Speed Integrations With Anypoint Flow Designer

Linked in multi tier, multi-tenant, multi-problem kafka

Exposing Business Functionalities with SOA, Integration and API Management

MuleSoft Surat Meetup#51 - API Monitoring - Through a New Lens

Monitoring Consumer Digital Experience

MWLUG 2017: BP107 Plotting the Path for your IBM Domino Applications

Effective API Design

First Monterrey MuleSoft Meetup - Introduction to MuleSoft.

Deliver Unrivaled End-User Experience With Confidence - How Synthetic Monitor...

Building an API Factory: Turn your APIs into Products

Scribe Online CDK & Connector Development

API Management: Unlock the Value of Your Unique Assets with a Robust API

Recently uploaded

"ML in Production",Oleksandr BaganFwdays

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited

Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity

Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren

Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi

Search Engine Optimization SEO PDF for 2024.pdfRankYa

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays

WordPress Websites for Engineers: Elevate Your Brandgvaughan

DMCC Future of Trade Web3 - Special EditionDubai Multi Commodity Centre

Recently uploaded (20)

"ML in Production",Oleksandr Bagan

Scanning the Internet for External Cloud Exposures via SSL Certs

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...

DevoxxFR 2024 Reproducible Builds with Apache Maven

Human Factors of XR: Using Human Factors to Design XR Systems

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

Ensuring Technical Readiness For Copilot in Microsoft 365

Dev Dives: Streamline document processing with UiPath Studio Web

Advanced Test Driven-Development @ php[tek] 2024

Unleash Your Potential - Namagunga Girls Coding Club

DevEX - reference for building teams, processes, and platforms

Connect Wave/ connectwave Pitch Deck Presentation

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)

My Hashitalk Indonesia April 2024 Presentation

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024

Vertex AI Gemini Prompt Engineering Tips

Search Engine Optimization SEO PDF for 2024.pdf

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

WordPress Websites for Engineers: Elevate Your Brand

DMCC Future of Trade Web3 - Special Edition