SlideShare a Scribd company logo
1 of 63
Download to read offline
Take my logs. Please.

Mike Brittain
Director of Engineering, Infrastructure
Etsy.com

mike@etsy.com      @mikebrittain
(hello?)
This sounds
boooooorrrrring...
No, no... hang in there!
Take My Logs. Please!
25 MM uniques/month
150 Countries
$300 MM+ sales last year
Apache, PHP, MySQL,
PostgreSQL,
Memcache, Gearman,
Solr, etc.
What’s working?
What’s working?
Performance
What’s working?
Performance
Operability
What’s working?
Performance
Operability
Simplicity
Logging + Trending
App logging
(Apache access and error logs)
“Common”
LogFormat "%h %l %u %t
 "%r" %>s %b
“Combined”
LogFormat "%h %l %u %t
 "%r" %>s %b
 "%{Referer}i"
 "%{User-agent}i""
mod_log_config
  %f         Filename requested

       # of keepalive requests served
  %k         on this connection
       Time taken to serve the request,
  %T              in seconds
mod_log_config
  %f         Filename requested

       # of keepalive requests served
  %k         on this connection



       Time taken to serve the request,
  %D           in microseconds
mod_log_config
   %f              Filename requested

             # of keepalive requests served
   %k              on this connection



             Time taken to serve the request,
   %D                in microseconds
             Contents of “note” foobar from
%{foobar}n           another module
apache_note()
apache_note(“foobar”, $whatever);
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
$GLOBALS['timer'] = microtime(true) * 1000000;
$GLOBALS['timer'] = microtime(true) * 1000000;

register_shutdown_function('pageStats');

function pageStats() {




}
$GLOBALS['timer'] = microtime(true) * 1000000;

register_shutdown_function('pageStats');

function pageStats() {
  $timer_end = microtime(true) * 1000000;
  $diff = $timer_end - $GLOBALS['timer'];




}
$GLOBALS['timer'] = microtime(true) * 1000000;

register_shutdown_function('pageStats');

function pageStats() {
  $timer_end = microtime(true) * 1000000;
  $diff = $timer_end - $GLOBALS['timer'];

    apache_note('php_microsec', $diff);

    apache_note('php_bytes',
                memory_get_peak_usage());
}
What about “%D”?
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
“Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n ...

easy_reg=1; personalize_widget=0;
icon_in_cornflower_blue=1;
Coming soon...
%{locale}n     (i18n)

%{platform}n   (desktop vs. mobile)
Coming soon...
%{locale}n     (i18n)

%{platform}n   (desktop vs. mobile)



OPS-1805, OPS-1827
etsy.com/careers
Using something else?
time, http method, request uri,
response code, referer, user-agent,
response time, response memory,
custom segmentation fields...
Quick averages
grep "GET /listing/" access.log | 

awk '{sum=sum+$(NF-1)} END {print sum/NR}'
Quick graphs
grep "GET /listing/" access.log | 

perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" | 

awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat




gives you...
Quick graphs
# /tmp/pagetimes.dat

18:37   251.0
18:38   252.1
18:39   253.5
18:40   251.0
18:45   250.0


and then...
Quick graphs
# GNUPLOT

set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points
Quick graphs
Error logs
PHP + Apache errors in one file
Simple logging interface
Error logs
Levels: error, info, debug
Namespace: perf, sql, __class__
Logger::error("Query exceeded 5 sec: $query",

             “sql_long_query”);
web0054 [Fri Mar 04 16:27:48 2011] [error]

[sql_long_query] [mk04gw1p71] Query exceeded

5 sec: SELECT * FROM ...
web0054 [Fri Mar 04 16:27:48 2011] [error]

[sql_long_query] [mk04gw1p71] Query exceeded

5 sec: SELECT * FROM ...
$ grep "16:27:48" access.log | wc -l

1527
web0054 [Fri Mar 04 16:27:48 2011] [error]

[sql_long_query] [mk04gw1p71] Query exceeded

5 sec: SELECT * FROM ...
iow
error.log -> request_uid -> access.log
request uri, ab selections, user id, locale,
platform, api key, etc.
Filtering
tail -f error.log | grep -v “sql_long_query” | ...
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Help me, Rhonda.
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Oh noooooo!
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Oh noooooo!
web0001   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0034   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201   [04:28:54   2011]   [error] [client 10.101.x.x] You've been eaten by a gr
web0055   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002   [04:28:54   2011]   [warning] [client 10.101.x.x] Sky is falling.
web0089   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020   [04:28:54   2011]   [error] [client 10.101.x.x] Sky is falling.
web1101   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0001   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087   [04:28:54   2011]   [fatal] [client 10.101.x.x] Sky is falling.
web0002   [04:28:54   2011]   [error] [client 10.101.x.x] Oh noooooo!
web0201   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0355   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0052   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003   [04:28:54   2011]   [error] [client 10.101.x.x] You've been eaten by a gr
web0066   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0001   [04:28:54   2011]   [warning] [client 10.101.x.x] Sky is falling
web0020   [04:28:54   2011]   [error] [client 10.101.x.x] Sky is falling.
web1101   [04:28:54   2011]   [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055   [04:28:54   2011]   [warning] [client 10.101.x.x] Gaaaaahhh!
web0001   [04:28:54   2011]   [warning] [client 10.101.x.x] Oh nooooooooooo
web0001   [04:28:54   2011]   [error] [client 10.101.x.x] Gaaaaahhh!!!
Trending
fatals   errors   warnings
Logster
Run by cron
Maintains a cursor on log files
Simple parsing & aggregation
Output to Ganglia or Graphite

                          github.com/etsy
web0054 [Fri Mar 04 16:27:48 2011] [error]
 [login] [mk04gw1p71] User login failed.
      Reason: wrong password for ...
^.+ [.+] [(?P<log_level>.+)]
if (fields['log_level'] == “fatal”):
   self.fatals += 1

elif (fields['log_level'] == “error”):
   self.errors += 1

elif (fields['log_level'] == “warning”):
   self.warnings += 1

...
MetricObject("fatals",
  (self.fatals / self.duration), "per sec")

MetricObject("errors",
  (self.errors / self.duration), "per sec")

MetricObject("warning",
  (self.warnings / self.duration), "per sec")
fatals   errors   warnings
Logster

          Signed-in vs. Signed-out
github.com/etsy
Log a plethora of data.
Don’t be afraid to use one file.
Use custom fields to segment data.
Correlate errors to specific requests.
Make f#@k!ng graphs.
Convert rates to trend lines.
Take my logs. Please!
Thank you.
                  codeascraft.etsy.com
                  github.com/etsy

Mike Brittain
Director of Engineering, Infrastructure
Etsy.com

mike@etsy.com      @mikebrittain

More Related Content

What's hot

Using Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an analytics product on App EngineUsing Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an analytics product on App EngineRiver of Talent
 
Anatomy of an Addon Ecosystem - EmberConf 2019
Anatomy of an Addon Ecosystem - EmberConf 2019Anatomy of an Addon Ecosystem - EmberConf 2019
Anatomy of an Addon Ecosystem - EmberConf 2019Lisa Backer
 
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]Iakiv Kramarenko
 
Oracle APEX Performance
Oracle APEX PerformanceOracle APEX Performance
Oracle APEX PerformanceScott Wesley
 
PHPUnit Episode iv.iii: Return of the tests
PHPUnit Episode iv.iii: Return of the testsPHPUnit Episode iv.iii: Return of the tests
PHPUnit Episode iv.iii: Return of the testsMichelangelo van Dam
 
Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13Stephan Hochdörfer
 
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...apidays
 
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...apidays
 
Rails 3: Dashing to the Finish
Rails 3: Dashing to the FinishRails 3: Dashing to the Finish
Rails 3: Dashing to the FinishYehuda Katz
 
Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13Stephan Hochdörfer
 
A Journey with React
A Journey with ReactA Journey with React
A Journey with ReactFITC
 
Incremental Type Safety in React Apollo
Incremental Type Safety in React Apollo Incremental Type Safety in React Apollo
Incremental Type Safety in React Apollo Evans Hauser
 
You do not need automation engineer - Sqa Days - 2015 - EN
You do not need automation engineer  - Sqa Days - 2015 - ENYou do not need automation engineer  - Sqa Days - 2015 - EN
You do not need automation engineer - Sqa Days - 2015 - ENIakiv Kramarenko
 
Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13Stephan Hochdörfer
 
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)Mike Schinkel
 
Bowtie: Interactive Dashboards
Bowtie: Interactive DashboardsBowtie: Interactive Dashboards
Bowtie: Interactive DashboardsJacques Kvam
 
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014Mike Schinkel
 
The Best (and Worst) of Django
The Best (and Worst) of DjangoThe Best (and Worst) of Django
The Best (and Worst) of DjangoJacob Kaplan-Moss
 

What's hot (20)

Using Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an analytics product on App EngineUsing Task Queues and D3.js to build an analytics product on App Engine
Using Task Queues and D3.js to build an analytics product on App Engine
 
Anatomy of an Addon Ecosystem - EmberConf 2019
Anatomy of an Addon Ecosystem - EmberConf 2019Anatomy of an Addon Ecosystem - EmberConf 2019
Anatomy of an Addon Ecosystem - EmberConf 2019
 
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]
 
Oracle APEX Performance
Oracle APEX PerformanceOracle APEX Performance
Oracle APEX Performance
 
PHPUnit Episode iv.iii: Return of the tests
PHPUnit Episode iv.iii: Return of the testsPHPUnit Episode iv.iii: Return of the tests
PHPUnit Episode iv.iii: Return of the tests
 
Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13Real World Dependency Injection - oscon13
Real World Dependency Injection - oscon13
 
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
APIdays Helsinki 2019 - Specification-Driven Development of REST APIs with Al...
 
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
APIdays Helsinki 2019 - API Versioning with REST, JSON and Swagger with Thoma...
 
Rails 3: Dashing to the Finish
Rails 3: Dashing to the FinishRails 3: Dashing to the Finish
Rails 3: Dashing to the Finish
 
KISS Automation.py
KISS Automation.pyKISS Automation.py
KISS Automation.py
 
Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13Phing for power users - dpc_uncon13
Phing for power users - dpc_uncon13
 
A Journey with React
A Journey with ReactA Journey with React
A Journey with React
 
Incremental Type Safety in React Apollo
Incremental Type Safety in React Apollo Incremental Type Safety in React Apollo
Incremental Type Safety in React Apollo
 
You do not need automation engineer - Sqa Days - 2015 - EN
You do not need automation engineer  - Sqa Days - 2015 - ENYou do not need automation engineer  - Sqa Days - 2015 - EN
You do not need automation engineer - Sqa Days - 2015 - EN
 
Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13Your Business. Your Language. Your Code - dpc13
Your Business. Your Language. Your Code - dpc13
 
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014 (PPT)
 
Bowtie: Interactive Dashboards
Bowtie: Interactive DashboardsBowtie: Interactive Dashboards
Bowtie: Interactive Dashboards
 
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
Hardcore URL Routing for WordPress - WordCamp Atlanta 2014
 
The Best (and Worst) of Django
The Best (and Worst) of DjangoThe Best (and Worst) of Django
The Best (and Worst) of Django
 
GAEO
GAEOGAEO
GAEO
 

Viewers also liked

How to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNHow to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNMike Brittain
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCMike Brittain
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsMike Brittain
 
Simple Log Analysis and Trending
Simple Log Analysis and TrendingSimple Log Analysis and Trending
Simple Log Analysis and TrendingMike Brittain
 
On Failure and Resilience
On Failure and ResilienceOn Failure and Resilience
On Failure and ResilienceMike Brittain
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsMike Brittain
 
From Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsFrom Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsMike Brittain
 
The Real Life Social Network v2
The Real Life Social Network v2The Real Life Social Network v2
The Real Life Social Network v2Paul Adams
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyMike Brittain
 
26 Disruptive & Technology Trends 2016 - 2018
26 Disruptive & Technology Trends 2016 - 201826 Disruptive & Technology Trends 2016 - 2018
26 Disruptive & Technology Trends 2016 - 2018Brian Solis
 

Viewers also liked (10)

How to Get to Second Base with Your CDN
How to Get to Second Base with Your CDNHow to Get to Second Base with Your CDN
How to Get to Second Base with Your CDN
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
 
Simple Log Analysis and Trending
Simple Log Analysis and TrendingSimple Log Analysis and Trending
Simple Log Analysis and Trending
 
On Failure and Resilience
On Failure and ResilienceOn Failure and Resilience
On Failure and Resilience
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
 
From Building a Marketplace to Building Teams
From Building a Marketplace to Building TeamsFrom Building a Marketplace to Building Teams
From Building a Marketplace to Building Teams
 
The Real Life Social Network v2
The Real Life Social Network v2The Real Life Social Network v2
The Real Life Social Network v2
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
26 Disruptive & Technology Trends 2016 - 2018
26 Disruptive & Technology Trends 2016 - 201826 Disruptive & Technology Trends 2016 - 2018
26 Disruptive & Technology Trends 2016 - 2018
 

Similar to Take My Logs. Please!

The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010Fabien Potencier
 
Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기JeongHun Byeon
 
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...dantleech
 
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac..."Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac...Fwdays
 
Joomla! security 101
Joomla! security 101Joomla! security 101
Joomla! security 101vdrover
 
Action Controller Overview, Season 2
Action Controller Overview, Season 2Action Controller Overview, Season 2
Action Controller Overview, Season 2RORLAB
 
Introducing Neo4j 3.1: New Security and Clustering Architecture
Introducing Neo4j 3.1: New Security and Clustering Architecture Introducing Neo4j 3.1: New Security and Clustering Architecture
Introducing Neo4j 3.1: New Security and Clustering Architecture Neo4j
 
Building Cloud Castles
Building Cloud CastlesBuilding Cloud Castles
Building Cloud CastlesBen Scofield
 
OWASP Top 10 - DrupalCon Amsterdam 2019
OWASP Top 10 - DrupalCon Amsterdam 2019OWASP Top 10 - DrupalCon Amsterdam 2019
OWASP Top 10 - DrupalCon Amsterdam 2019Ayesh Karunaratne
 
Re-Design with Elixir/OTP
Re-Design with Elixir/OTPRe-Design with Elixir/OTP
Re-Design with Elixir/OTPMustafa TURAN
 
idea: talk about the Active Cache
idea: talk about the Active Cacheidea: talk about the Active Cache
idea: talk about the Active CacheChing Yi Chan
 
What's new in Rails 4
What's new in Rails 4What's new in Rails 4
What's new in Rails 4Fabio Akita
 
Neo4j Network security
Neo4j Network securityNeo4j Network security
Neo4j Network securityssusera3acdc
 
Brian hogg word camp preparing a plugin for translation
Brian hogg   word camp preparing a plugin for translationBrian hogg   word camp preparing a plugin for translation
Brian hogg word camp preparing a plugin for translationwcto2017
 
Malware Detection with OSSEC HIDS - OSSECCON 2014
Malware Detection with OSSEC HIDS - OSSECCON 2014Malware Detection with OSSEC HIDS - OSSECCON 2014
Malware Detection with OSSEC HIDS - OSSECCON 2014Santiago Bassett
 
Preparing a WordPress Plugin for Translation
Preparing a WordPress Plugin for TranslationPreparing a WordPress Plugin for Translation
Preparing a WordPress Plugin for TranslationBrian Hogg
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerElixir Club
 
Review unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpReview unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpDamien Seguy
 
Workshop quality assurance for php projects tek12
Workshop quality assurance for php projects tek12Workshop quality assurance for php projects tek12
Workshop quality assurance for php projects tek12Michelangelo van Dam
 

Similar to Take My Logs. Please! (20)

The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010The state of Symfony2 - SymfonyDay 2010
The state of Symfony2 - SymfonyDay 2010
 
Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기Node.js API 서버 성능 개선기
Node.js API 서버 성능 개선기
 
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
Building and Incredible Machine with Pipelines and Generators in PHP (IPC Ber...
 
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac..."Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
"Full Stack frameworks or a story about how to reconcile Front (good) and Bac...
 
Joomla! security 101
Joomla! security 101Joomla! security 101
Joomla! security 101
 
Action Controller Overview, Season 2
Action Controller Overview, Season 2Action Controller Overview, Season 2
Action Controller Overview, Season 2
 
Introducing Neo4j 3.1: New Security and Clustering Architecture
Introducing Neo4j 3.1: New Security and Clustering Architecture Introducing Neo4j 3.1: New Security and Clustering Architecture
Introducing Neo4j 3.1: New Security and Clustering Architecture
 
Building Cloud Castles
Building Cloud CastlesBuilding Cloud Castles
Building Cloud Castles
 
OWASP Top 10 - DrupalCon Amsterdam 2019
OWASP Top 10 - DrupalCon Amsterdam 2019OWASP Top 10 - DrupalCon Amsterdam 2019
OWASP Top 10 - DrupalCon Amsterdam 2019
 
Re-Design with Elixir/OTP
Re-Design with Elixir/OTPRe-Design with Elixir/OTP
Re-Design with Elixir/OTP
 
idea: talk about the Active Cache
idea: talk about the Active Cacheidea: talk about the Active Cache
idea: talk about the Active Cache
 
What's new in Rails 4
What's new in Rails 4What's new in Rails 4
What's new in Rails 4
 
Neo4j Network security
Neo4j Network securityNeo4j Network security
Neo4j Network security
 
Brian hogg word camp preparing a plugin for translation
Brian hogg   word camp preparing a plugin for translationBrian hogg   word camp preparing a plugin for translation
Brian hogg word camp preparing a plugin for translation
 
Malware Detection with OSSEC HIDS - OSSECCON 2014
Malware Detection with OSSEC HIDS - OSSECCON 2014Malware Detection with OSSEC HIDS - OSSECCON 2014
Malware Detection with OSSEC HIDS - OSSECCON 2014
 
Symfony 1, mi viejo amigo
Symfony 1, mi viejo amigoSymfony 1, mi viejo amigo
Symfony 1, mi viejo amigo
 
Preparing a WordPress Plugin for Translation
Preparing a WordPress Plugin for TranslationPreparing a WordPress Plugin for Translation
Preparing a WordPress Plugin for Translation
 
WebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan WintermeyerWebPerformance: Why and How? – Stefan Wintermeyer
WebPerformance: Why and How? – Stefan Wintermeyer
 
Review unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphpReview unknown code with static analysis - bredaphp
Review unknown code with static analysis - bredaphp
 
Workshop quality assurance for php projects tek12
Workshop quality assurance for php projects tek12Workshop quality assurance for php projects tek12
Workshop quality assurance for php projects tek12
 

Recently uploaded

9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 

Recently uploaded (20)

9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 

Take My Logs. Please!

  • 1. Take my logs. Please. Mike Brittain Director of Engineering, Infrastructure Etsy.com mike@etsy.com @mikebrittain
  • 5. 25 MM uniques/month 150 Countries $300 MM+ sales last year
  • 12. App logging (Apache access and error logs)
  • 13. “Common” LogFormat "%h %l %u %t "%r" %>s %b
  • 14. “Combined” LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i""
  • 15. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %T in seconds
  • 16. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %D in microseconds
  • 17. mod_log_config %f Filename requested # of keepalive requests served %k on this connection Time taken to serve the request, %D in microseconds Contents of “note” foobar from %{foobar}n another module
  • 19. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 20. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 22. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { }
  • 23. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { $timer_end = microtime(true) * 1000000; $diff = $timer_end - $GLOBALS['timer']; }
  • 24. $GLOBALS['timer'] = microtime(true) * 1000000; register_shutdown_function('pageStats'); function pageStats() { $timer_end = microtime(true) * 1000000; $diff = $timer_end - $GLOBALS['timer']; apache_note('php_microsec', $diff); apache_note('php_bytes', memory_get_peak_usage()); }
  • 26. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 27. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 28. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n %{request_uid}n %{api_consumer_key}n %{api_method_name}n %{php_bytes}n %{php_microsec}n %D
  • 29. “Steroids” LogFormat %{True-Client-IP}i %l %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %V %{user_id}n %{shop_id}n %{uaid}n %{ab_selections}n ... easy_reg=1; personalize_widget=0; icon_in_cornflower_blue=1;
  • 30. Coming soon... %{locale}n (i18n) %{platform}n (desktop vs. mobile)
  • 31. Coming soon... %{locale}n (i18n) %{platform}n (desktop vs. mobile) OPS-1805, OPS-1827 etsy.com/careers
  • 32. Using something else? time, http method, request uri, response code, referer, user-agent, response time, response memory, custom segmentation fields...
  • 33. Quick averages grep "GET /listing/" access.log | awk '{sum=sum+$(NF-1)} END {print sum/NR}'
  • 34. Quick graphs grep "GET /listing/" access.log | perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" | awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat gives you...
  • 35. Quick graphs # /tmp/pagetimes.dat 18:37 251.0 18:38 252.1 18:39 253.5 18:40 251.0 18:45 250.0 and then...
  • 36. Quick graphs # GNUPLOT set terminal png set output 'listings.png' set yrange [0:2000] set xdata time set timefmt "%d/%B/%Y:%H:%M:%S" set format x "%H:%M" plot '/tmp/pagetimes.dat' using 1:2 with points
  • 38. Error logs PHP + Apache errors in one file Simple logging interface
  • 39. Error logs Levels: error, info, debug Namespace: perf, sql, __class__
  • 40. Logger::error("Query exceeded 5 sec: $query", “sql_long_query”);
  • 41. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
  • 42. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
  • 43. $ grep "16:27:48" access.log | wc -l 1527
  • 44. web0054 [Fri Mar 04 16:27:48 2011] [error] [sql_long_query] [mk04gw1p71] Query exceeded 5 sec: SELECT * FROM ...
  • 45. iow error.log -> request_uid -> access.log request uri, ab selections, user id, locale, platform, api key, etc.
  • 46. Filtering tail -f error.log | grep -v “sql_long_query” | ...
  • 47. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling. web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling. web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo! web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!! web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling. web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh! web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh! web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
  • 48. Trending fatals errors warnings
  • 49. Logster Run by cron Maintains a cursor on log files Simple parsing & aggregation Output to Ganglia or Graphite github.com/etsy
  • 50. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] User login failed. Reason: wrong password for ...
  • 52. if (fields['log_level'] == “fatal”): self.fatals += 1 elif (fields['log_level'] == “error”): self.errors += 1 elif (fields['log_level'] == “warning”): self.warnings += 1 ...
  • 53. MetricObject("fatals", (self.fatals / self.duration), "per sec") MetricObject("errors", (self.errors / self.duration), "per sec") MetricObject("warning", (self.warnings / self.duration), "per sec")
  • 54. fatals errors warnings
  • 55. Logster Signed-in vs. Signed-out
  • 57. Log a plethora of data. Don’t be afraid to use one file.
  • 58. Use custom fields to segment data.
  • 59. Correlate errors to specific requests.
  • 61. Convert rates to trend lines.
  • 62. Take my logs. Please!
  • 63. Thank you. codeascraft.etsy.com github.com/etsy Mike Brittain Director of Engineering, Infrastructure Etsy.com mike@etsy.com @mikebrittain