Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fluentd and Docker - running fluentd within a docker container

Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.

Fluentd and Docker - running fluentd within a docker container

  1. 1. FLUENTD: UNIFIED LOGGING LAYER John Hammink August 11, 2015 Docker San Francisco Meetup
  2. 2. About Me • A recovering software & QA engineer turned digital artist once interested in fractals; • now into data visualization based on large datasets rendered directly to GPU (RGL, various Python GL libraries, etc.) • github: jammink2; twitter: rijksband
  3. 3. Tweet NOW! “At #svde learning how to collect more event data using #Fluentd”
  4. 4. WHAT’S FLUENTD? An extensible & reliable data collection tool simple core + plugins buffering, HA (failover), load balancing, etc. like syslogd
  5. 5. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
  6. 6. data collection tool
  7. 7. ✓ duplicated code for error handling... ✓ messy code for retrying mechnism... Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system bash scripts ruby scripts rsync log file bash python scripts custom
 loggger cron other custom
 scripts...
  8. 8. (this is painful!!!)
  9. 9. Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system filter / buffer / route
  10. 10. extensible
  11. 11. CORE PLUGINS • Divide & Conquer • Buffering & Retries • Error Handling • Message Routing • Parallelism • Read Data • Parse Data • Buffer Data • Write Data • Format Data Common Concerns Use Case Specific
  12. 12. architecture
  13. 13. INTERNAL ARCHITECTURE “input-ish” “output-ish” Input Parser Buffer Output FormatterFilter
  14. 14. Internal Architecture (Simplified) Input Buffer Output Plugin Plugin Plugin 2012-02-04 01:33:51 myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } time tag record
  15. 15. Architecture :: Input plugins Input HTTP+JSON (in_http) File tail (in_tail) Syslog (in_syslog) ... Plugin ✓ Receive logs ✓ Or pull logs from data sources ✓ in non-blocking manner
  16. 16. Architecture :: Output plugins Plugin ✓ Write or send event logs Output File (out_file) Amazon S3 (out_s3) MongoDB (out_mongo) ...
  17. 17. Architecture :: Buffer plugins Plugin ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety Buffer Memory (buf_memory) File (buf_file)
  18. 18. Architecture :: Buffer plugins Plugin ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety chunk chunk chunk output Input
  19. 19. reliable data transfer
  20. 20. DIVIDE & CONQUER & RETRY error retry error retry retry retry
  21. 21. reliable process
  22. 22. THIS?
  23. 23. OR THIS?
  24. 24. M X N → M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer/filter/route
  25. 25. use cases
  26. 26. SIMPLE FORWARDING
  27. 27. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match backend.*> type mongo database fluent collection test </match>
  28. 28. LESS SIMPLE FORWARDING
  29. 29. LAMBDA ARCHITECTURE
  30. 30. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match *.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
  31. 31. FLUENTD ON KUBERNETES (NOV 2015)
  32. 32. FLUENTD LOGGING DRIVER (APR 2015)
  33. 33. Tweet Again! “Happy v1 #k8s and congrats #Fluentd for becoming a #docker logging driver”
  34. 34. DEMO: FLUENTD + DOCKER
  35. 35. THANK YOU! AND TREASURE DATA IS HIRING! WWW.TREASUREDATA.COMC/CAREERS

    Be the first to comment

    Login to see the comments

  • terumba

    Aug. 12, 2015
  • omaruriel

    Aug. 13, 2015
  • akuwano

    Aug. 13, 2015
  • kenichiroukimura

    Aug. 14, 2015
  • kgtzsolt

    Aug. 20, 2015
  • Gonnector

    Feb. 3, 2017
  • diegobenedicto1

    Mar. 27, 2018
  • DrDamianIgbe

    Jun. 13, 2018

Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.

Views

Total views

5,318

On Slideshare

0

From embeds

0

Number of embeds

36

Actions

Downloads

36

Shares

0

Comments

0

Likes

8

×