Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Stream processing in Mercari - Devsumi 2015 autumn LT

Stream processing in Mercari - Devsumi 2015 autumn LT

  • Login to see the comments

Stream processing in Mercari - Devsumi 2015 autumn LT

  1. 1. Stream processing in Mercari Developers Summit 2015 Autumn LT @kazeburo Masahiro Nagano
  2. 2. Me •長野雅広(Masahiro Nagano) •@kazeburo •Mercari, Inc. •Principal Engineer, Site Reliability
  3. 3. Your Friendly Mobile Marketplace
  4. 4. Stream processing for Monitoring
  5. 5. Monitoring
  6. 6. collect metrics value • Just get current value/point • e.g. load avg, memory usage • Get counter/incremental value and calculate changes • e.g. traffic, cpu usage • Parse log and count value by your self • e.g. access_log, error_log, custom_log
  7. 7. Log monitoring Past and present
  8. 8. While ago • Make a script for parsing and count logs • Execute periodically from monitoring tool • Too many tail (io) and grep (cpu) (;_;) #!/bin/sh set -e LOG_WATCH=20000 TARGET=$(date -d '61 second ago' +%H:%M:[0-9][0-9]) tail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" | wc -l tail -$LOG_WATCH /var/log/httpd/service_access_log | grep "$TARGET" | grep '" 500 ' wc -l
  9. 9. fluentd + datacounter Web access_log in_tail Web access_log Web access_log aggregate datacounter <match mercari.access_log> type datacounter count_interval 1m count_key status pattern1 2xx ^2dd$ pattern2 3xx ^3dd$ pattern4 4xx ^4dd$ pattern5 5xx ^5dd$ tag status.datacount.web </match> <match status.datacount.web> type zabbix zabbix_server 10.x.x.x </match>
  10. 10. Cons of fluentd +datacounter •Gigantic fluend.conf •need a configuration file generator? •Write fluend plugin for responding various needs? •restart fluentd is required for adding a new tabulation
  11. 11. Norikra •Norikra is an awesome open source product by Mr. tagomori, provides “Stream processing” with SQL •We LOVE SQL!!
  12. 12. 219.109.xx.xx - - [09/Nov/2014:06:50:07 +0900] "GET /mt.js HTTP/1.1" 200 6339 "http://blog.nomadscafe.jp/2013/06/webdb-pressvol172.html" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" 50.116.xx.xx - - [09/Nov/2014:06:54:41 +0900] "GET /2014/09/line- isucon4-51192.html HTTP/1.1" 200 30194 "http://www.google.co.uk/url? sa=t&source=web&cd=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.92 Safari/535.2" 76.164.xx.xx - - [09/Nov/2014:07:09:21 +0900] "GET / HTTP/1.0" 200 75606 "http://blog.nomadscafe.jp/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/ 537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36" SELECT COUNT(1, status like “5%”) AS count_5x, COUNT(1, status like “2%”) AS count_2x FROM access_log.win:time_batch(1 min) { “count_2x”: 765, “count_5x”: 2 } Norikra
  13. 13. After Norika Web access_log in_tail Web access_log Web access_log aggregate SQL投入 Norikra
  14. 14. Pros of Norikra • Do not need a gigantic configuration file • Do not required restarting a daemon • Schema less • analysis any log data like error_log, application specific log • SQL • every engineer/producer can write SQL
  15. 15. Graph/Alert •zabbix •nagios •GrowthForecast (graph only) •Data Dog •mackerel
  16. 16. Worker/Batch WebWeb Stream Processing in Mercari Web/App access_log Worker/Batch create SQL Norikra error_log app_log error_log app_log define Alert Mackerel Generates Graph automatically!!
  17. 17. Alternatives and Competitors • PipelineDB • www.pipelinedb.com • Azure Stream Analytics • azure.microsoft.com/ja-jp/services/stream- analytics/ • Amazon Kinesis Analytics(coming soon) • aws.amazon.com/kinesis/analytics/
  18. 18. end

×