Details on how we capture application data in our access and error logs, as well as how to generate quick reports and graphs from these logs. This talk was presented at O'Reilly's Velocity Online Conference on October 26, 2011.

1. Take my logs. Please.
Mike Brittain
Director of Engineering, Infrastructure
Etsy.com
mike@etsy.com @mikebrittain

2. (hello?)

3. This sounds
boooooorrrrring...
No, no... hang in there!

5. 25 MM uniques/month
150 Countries
$300 MM+ sales last year

6. Apache, PHP, MySQL,
PostgreSQL,
Memcache, Gearman,
Solr, etc.

7. What’s working?

8. What’s working?
Performance

9. What’s working?
Performance
Operability

10. What’s working?
Performance
Operability
Simplicity

11. Logging + Trending

12. App logging
(Apache access and error logs)

13. “Common”
LogFormat "%h %l %u %t
"%r" %>s %b

14. “Combined”
LogFormat "%h %l %u %t
"%r" %>s %b
"%{Referer}i"
"%{User-agent}i""

15. mod_log_config
%f Filename requested
# of keepalive requests served
%k on this connection
Time taken to serve the request,
%T in seconds

16. mod_log_config
%f Filename requested
# of keepalive requests served
%k on this connection
Time taken to serve the request,
%D in microseconds

17. mod_log_config
%f Filename requested
# of keepalive requests served
%k on this connection
Time taken to serve the request,
%D in microseconds
Contents of “note” foobar from
%{foobar}n another module

18. apache_note()
apache_note(“foobar”, $whatever);

19. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D

20. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D

21. $GLOBALS['timer'] = microtime(true) * 1000000;

22. $GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
}

23. $GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
$timer_end = microtime(true) * 1000000;
$diff = $timer_end - $GLOBALS['timer'];
}

24. $GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
$timer_end = microtime(true) * 1000000;
$diff = $timer_end - $GLOBALS['timer'];
apache_note('php_microsec', $diff);
apache_note('php_bytes',
memory_get_peak_usage());
}

25. What about “%D”?

26. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D

27. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D

28. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D

29. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n ...
easy_reg=1; personalize_widget=0;
icon_in_cornflower_blue=1;

30. Coming soon...
%{locale}n (i18n)
%{platform}n (desktop vs. mobile)

31. Coming soon...
%{locale}n (i18n)
%{platform}n (desktop vs. mobile)
OPS-1805, OPS-1827
etsy.com/careers

32. Using something else?
time, http method, request uri,
response code, referer, user-agent,
response time, response memory,
custom segmentation fields...

33. Quick averages
grep "GET /listing/" access.log |
awk '{sum=sum+$(NF-1)} END {print sum/NR}'

34. Quick graphs
grep "GET /listing/" access.log |
perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" |
awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat
gives you...

35. Quick graphs
# /tmp/pagetimes.dat
18:37 251.0
18:38 252.1
18:39 253.5
18:40 251.0
18:45 250.0
and then...

36. Quick graphs
# GNUPLOT
set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points

37. Quick graphs

38. Error logs
PHP + Apache errors in one file
Simple logging interface

39. Error logs
Levels: error, info, debug
Namespace: perf, sql, __class__

40. Logger::error("Query exceeded 5 sec: $query",
“sql_long_query”);

41. web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...

42. web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...

43. $ grep "16:27:48" access.log | wc -l
1527

44. web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...

45. iow
error.log -> request_uid -> access.log
request uri, ab selections, user id, locale,
platform, api key, etc.

46. Filtering
tail -f error.log | grep -v “sql_long_query” | ...

47. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr
web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!

48. Trending
fatals errors warnings

49. Logster
Run by cron
Maintains a cursor on log files
Simple parsing & aggregation
Output to Ganglia or Graphite
github.com/etsy

50. web0054 [Fri Mar 04 16:27:48 2011] [error]
[login] [mk04gw1p71] User login failed.
Reason: wrong password for ...

51. ^.+ [.+] [(?P<log_level>.+)]

52. if (fields['log_level'] == “fatal”):
self.fatals += 1
elif (fields['log_level'] == “error”):
self.errors += 1
elif (fields['log_level'] == “warning”):
self.warnings += 1
...

53. MetricObject("fatals",
(self.fatals / self.duration), "per sec")
MetricObject("errors",
(self.errors / self.duration), "per sec")
MetricObject("warning",
(self.warnings / self.duration), "per sec")

54. fatals errors warnings

55. Logster
Signed-in vs. Signed-out

56. github.com/etsy

57. Log a plethora of data.
Don’t be afraid to use one file.

58. Use custom fields to segment data.

59. Correlate errors to specific requests.

60. Make f#@k!ng graphs.

61. Convert rates to trend lines.

62. Take my logs. Please!

63. Thank you.
codeascraft.etsy.com
github.com/etsy
Mike Brittain
Director of Engineering, Infrastructure
Etsy.com
mike@etsy.com @mikebrittain