More Related Content Similar to Take My Logs. Please! (20) Take My Logs. Please!1. Take my logs. Please.
Mike Brittain
Director of Engineering, Infrastructure
Etsy.com
mike@etsy.com @mikebrittain
15. mod_log_config
%f Filename requested
# of keepalive requests served
%k on this connection
Time taken to serve the request,
%T in seconds
16. mod_log_config
%f Filename requested
# of keepalive requests served
%k on this connection
Time taken to serve the request,
%D in microseconds
17. mod_log_config
%f Filename requested
# of keepalive requests served
%k on this connection
Time taken to serve the request,
%D in microseconds
Contents of “note” foobar from
%{foobar}n another module
19. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
20. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
23. $GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
$timer_end = microtime(true) * 1000000;
$diff = $timer_end - $GLOBALS['timer'];
}
24. $GLOBALS['timer'] = microtime(true) * 1000000;
register_shutdown_function('pageStats');
function pageStats() {
$timer_end = microtime(true) * 1000000;
$diff = $timer_end - $GLOBALS['timer'];
apache_note('php_microsec', $diff);
apache_note('php_bytes',
memory_get_peak_usage());
}
26. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
27. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
28. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n %{request_uid}n
%{api_consumer_key}n
%{api_method_name}n
%{php_bytes}n %{php_microsec}n %D
29. “Steroids”
LogFormat %{True-Client-IP}i %l %t "%r"
%>s %b "%{Referer}i"
"%{User-Agent}i" %V
%{user_id}n %{shop_id}n %{uaid}n
%{ab_selections}n ...
easy_reg=1; personalize_widget=0;
icon_in_cornflower_blue=1;
32. Using something else?
time, http method, request uri,
response code, referer, user-agent,
response time, response memory,
custom segmentation fields...
34. Quick graphs
grep "GET /listing/" access.log |
perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" |
awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat
gives you...
36. Quick graphs
# GNUPLOT
set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points
41. web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...
42. web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...
44. web0054 [Fri Mar 04 16:27:48 2011] [error]
[sql_long_query] [mk04gw1p71] Query exceeded
5 sec: SELECT * FROM ...
47. web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppp
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a gr
web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
50. web0054 [Fri Mar 04 16:27:48 2011] [error]
[login] [mk04gw1p71] User login failed.
Reason: wrong password for ...
52. if (fields['log_level'] == “fatal”):
self.fatals += 1
elif (fields['log_level'] == “error”):
self.errors += 1
elif (fields['log_level'] == “warning”):
self.warnings += 1
...
53. MetricObject("fatals",
(self.fatals / self.duration), "per sec")
MetricObject("errors",
(self.errors / self.duration), "per sec")
MetricObject("warning",
(self.warnings / self.duration), "per sec")
55. Logster
Signed-in vs. Signed-out
63. Thank you.
codeascraft.etsy.com
github.com/etsy
Mike Brittain
Director of Engineering, Infrastructure
Etsy.com
mike@etsy.com @mikebrittain