From Ubu to kafka

•

5 likes•1,862 views

Yann Schwartz

A short history of log ingestion at Criteo and some kafka gotachas.

Data & Analytics

From Ubu to Kafka
ingesting logs at Criteo

Dramatis Personae
Yann Schwartz
@abolibibelot
SRE / DPS / Lake
/ Rivers

May you live in interesting times
● Rsyslog evil twin
● Who shaves the barber?
● Line based formatting for fun and profit
● The World according to ARP

● No one will ever need more than 4096 file descriptors
● Who needs Redis when you have Zookeeper?
● Big Data fits on an Integer
● Fault tolerance is overrated

Oh noes
● Use replication
● ...and understand ISRs
● ...and forbid unclean leader elections
● Your client hates you (BYOB)
● Storm hates you even more
●

Art credit: Mathieu Chataigner - François Visconte - Yann Schwartz

Recently uploaded

20240419 - Measurecamp Amsterdam - SAM.pdfHuman37

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)Universitat Politècnica de Catalunya

办理学位证中佛罗里达大学毕业证,UCF成绩单原版一比一F sss

Student Profile Sample report on improving academic performance by uniting gr...Seán Kennedy

Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhYasamin16

办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degreeyuu sss

Identifying Appropriate Test Statistics Involving Population MeanMYRABACSAFRA2

Predictive Analysis for Loan Default Presentation : Data Analysis Project PPTBoston Institute of Analytics

专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改yuu sss

Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics

Student profile product demonstration on grades, ability, well-being and mind...Seán Kennedy

Vision, Mission, Goals and Objectives ppt..pptxellehsormae

LLMs, LMMs, their Improvement Suggestions and the Path towards AGIThomas Poetter

GA4 Without Cookies [Measure Camp AMS]📊 Markus Baersch

RadioAdProWritingCinderellabyButleri.pdfgstagge

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024Timothy Spann

detection and classification of knee osteoarthritis.pptxAleenaJamil4

modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx

Biometric Authentication: The Evolution, Applications, Benefits and Challenge...GQ Research

NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...Boston Institute of Analytics

Recently uploaded (20)

20240419 - Measurecamp Amsterdam - SAM.pdf

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

办理学位证中佛罗里达大学毕业证,UCF成绩单原版一比一

Student Profile Sample report on improving academic performance by uniting gr...

Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh

办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree

Identifying Appropriate Test Statistics Involving Population Mean

Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT

专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改

Predicting Salary Using Data Science: A Comprehensive Analysis.pdf

Student profile product demonstration on grades, ability, well-being and mind...

Vision, Mission, Goals and Objectives ppt..pptx

LLMs, LMMs, their Improvement Suggestions and the Path towards AGI

GA4 Without Cookies [Measure Camp AMS]

RadioAdProWritingCinderellabyButleri.pdf

April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024

detection and classification of knee osteoarthritis.pptx

modul pembelajaran robotic Workshop _ by Slidesgo.pptx

Biometric Authentication: The Evolution, Applications, Benefits and Challenge...

NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...

Featured

Everything You Need To Know About ChatGPTExpeed Software

Product Design Trends in 2024 | Teenage EngineeringsPixeldarts

How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow

AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork

Skeleton Culture CodeSkeleton Technologies

PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley

Content Methodology: A Best Practices Report (Webinar)contently

How to Prepare For a Successful Job Search for 2024Albert Qian

Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)

Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal

5 Public speaking tips from TED - Visualized summarySpeakerHub

ChatGPT and the Future of Work - Clark Boyd Clark Boyd

Getting into the tech field. what next Tessa Mero

Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray

How to have difficult conversations Rajiv Jayarajah, MAppComm, ACC

Introduction to Data ScienceChristy Abraham Joy

Time Management & Productivity - Best PracticesVit Horky

The six step guide to practical project managementMindGenius

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36

Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools

Featured (20)

Everything You Need To Know About ChatGPT

Product Design Trends in 2024 | Teenage Engineerings

How Race, Age and Gender Shape Attitudes Towards Mental Health

AI Trends in Creative Operations 2024 by Artwork Flow.pdf

Skeleton Culture Code

PEPSICO Presentation to CAGNY Conference Feb 2024

Content Methodology: A Best Practices Report (Webinar)

How to Prepare For a Successful Job Search for 2024

Social Media Marketing Trends 2024 // The Global Indie Insights

Trends In Paid Search: Navigating The Digital Landscape In 2024

5 Public speaking tips from TED - Visualized summary

ChatGPT and the Future of Work - Clark Boyd

Getting into the tech field. what next

Google's Just Not That Into You: Understanding Core Updates & Search Intent

How to have difficult conversations

Introduction to Data Science

Time Management & Productivity - Best Practices

The six step guide to practical project management

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...

Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...

From Ubu to kafka

1. From Ubu to Kafka ingesting logs at Criteo

2. Dramatis Personae Yann Schwartz @abolibibelot SRE / DPS / Lake / Rivers

4. Evolutionary design

5. 2009

7. 2011

9. 2014

10.

11.

12.

13. 2015

14.

15.

16. Are we there yet?

17.

18.

19.

20.

21.

22. Wheel of misfortune

23. May you live in interesting times ● Rsyslog evil twin ● Who shaves the barber? ● Line based formatting for fun and profit ● The World according to ARP

24. ● No one will ever need more than 4096 file descriptors ● Who needs Redis when you have Zookeeper? ● Big Data fits on an Integer ● Fault tolerance is overrated

25. Mind the gap

26. Oh noes ● Use replication ● ...and understand ISRs ● ...and forbid unclean leader elections ● Your client hates you (BYOB) ● Storm hates you even more ●

27. Art credit: Mathieu Chataigner - François Visconte - Yann Schwartz

28. Join the fun. We’re hiring.

From Ubu to kafka

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

From Ubu to kafka