Fluentd is an open source data collector that allows for flexible and extensible logging. It provides a unified way to collect logs, metrics, and events from various sources and send them to multiple destinations. It handles concerns like buffering, retries, and failover to provide reliable data transfer. Fluentd uses a plugin-based architecture so it can support many use cases like simple forwarding, lambda architectures, stream processing, and logging for Docker and Kubernetes.

1. FLUENTD: UNIFIED LOGGING
LAYER
Kiyoto Tamura
July 23, 2015
Silicon Valley Data Engineering Meetup

2. WHOAMI
Kiyoto Tamura
GitHub/Twitter: kiyoto/kiyototamura
Treasure Data, Inc.
Vice President, Marketing
TRUST ME, I used to code daily =)

3. Tweet NOW!
“At #svde learning how to collect more event
data using #Fluentd”

4. WHAT’S FLUENTD?
An extensible & reliable data collection
tool
simple core + plugins
buffering, HA (failover),
load balancing, etc.
like syslogd

5. data collection tool

6. ✓ duplicated code for error handling...
✓ messy code for retrying mechnism...
Blueflood
MongoDB
Hadoop
Metrics
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Your system
bash scripts ruby scripts
rsync
log file
bash
python scripts
custom
loggger
cron
other custom
scripts...

7. (this is painful!!!)

8. Blueflood
MongoDB
Hadoop
Metrics
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Your system
filter / buffer / route

9. extensible

10. CORE PLUGINS
• Divide & Conquer
• Buffering & Retries
• Error Handling
• Message Routing
• Parallelism
• Read Data
• Parse Data
• Buffer Data
• Write Data
• Format Data
Common
Concerns
Use Case
Specific

11. architecture

12. INTERNAL ARCHITECTURE
“input-ish” “output-ish”
Input Parser Buffer Output FormatterFilter

13. reliable

14. reliable data transfer

15. DIVIDE & CONQUER & RETRY
error retry
error retry retry
retry

16. reliable process

17. THIS?

18. OR THIS?

19. M X N → M + N
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
buffer/filter/route

20. use cases

21. SIMPLE FORWARDING

22. # logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag backend.apache
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to ES and HDFS
<match backend.*>
type mongo
database fluent
collection test
</match>

23. LESS SIMPLE FORWARDING

24. LAMBDA ARCHITECTURE

25. # logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to ES and HDFS
<match backend.*>
type copy
<store>
type elasticsearch
logstash_format true
</store>
<store>
type webhdfs
host namenode
port 50070
path /path/on/hdfs/
</store>
</match>

26. CEP FOR STREAM PROCESSING

27. FLUENTD ON KUBERNETES (NOV 2015)

28. FLUENTD LOGGING DRIVER (APR 2015)

29. Tweet Again!
“Happy v1 #k8s and congrats #Fluentd for
becoming a #docker logging driver”

30. DEMO: FLUENTD + DOCKER

31. THANK YOU!
AND TREASURE DATA IS HIRING!
WWW.TREASUREDATA.COMC/CAREERS