Metrics Driven Engineering42. Time, remote address, http method, request uri, referrer, user-agent,
response size, response code, execution time, memory consumed,
plus custom fields...
• Signed-in/out (user_id vs. “-”)
• display mode (“desktop” vs. “mobile”)
• i10n/i18n (“en-US”)
• etc.
AccessLogs
43. LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...
apache_note(“custom_field”, $whatever);
44. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /
listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1"
200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://
www.google.com/bot.html)" desktop - 13399576 505780 554876
45. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /
listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1"
200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://
www.google.com/bot.html)" desktop - 13399576 505780 554876
46. Logger::error("User login failed. Reason:
$msg for $email_addr", “login”);
Methodnamedenoteslog“level”—
error,fatal,warning,notice,debug.
A“namespace”parameterisprovided
sowecanaggregatelogentrieswith
similarconcerns.
47. Logger::error("User login failed. Reason:
$msg for $email_addr", “login”);
web0054 [Fri Mar 04 16:27:48 2011] [error] [login]
[mk04gw1p71] User login failed. Reason: wrong
password was submitted for mike@etsy.com
UniquerequestID
Servername
Dateandtime Level
Namespace
48. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conve
web0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.
web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.
web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user ID
web0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset co
web0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user ID
web0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user ID
web0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.
web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Re
web0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID enc
web0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Re
web0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faile
web0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faile
web0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faile
web0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failed
web0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Re
web0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.
web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failed
web0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset c
web0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user ID
web0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.
web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.
web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.
web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. Re
49. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conve
web0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.
web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.
web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user ID
web0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset co
web0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user ID
web0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user ID
web0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.
web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Re
web0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID enc
web0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Re
web0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faile
web0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faile
web0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faile
web0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failed
web0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Re
web0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.
web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failed
web0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset c
web0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user ID
web0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.
web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.
web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.
web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. Re
50. web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conve
web0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.
web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.
web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user ID
web0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset co
web0003 [Fri Mar 04 16:27:49 2011] [error] [register] [39e08e6692] Duplicate user ID
web0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user ID
web0054 [Fri Mar 04 16:27:49 2011] [warning] [login] [mk04gw1p71] User login failed.
web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Re
web0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID enc
web0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Re
web0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faile
web0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faile
web0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faile
web0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failed
web0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Re
web0054 [Fri Mar 04 16:27:50 2011] [warning] [login] [mk04gw1p71] User login failed.
web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failed
web0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset c
web0012 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:50 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user ID
web0025 [Fri Mar 04 16:27:50 2011] [warning] [login] [32976da59c] User login failed.
web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.
web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.
web0035 [Fri Mar 04 16:27:51 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:51 2011] [error] [login] [2f297b40a5] User login failed. Re
FATALS ERRORS WARNINGS
Logster
52. web0054 [Fri Mar 04 16:27:48 2011] [error] [login]
[mk04gw1p71] User login failed. Reason: wrong
password was submitted for mike@etsy.com
^.+ [.+] [(?P<log_level>.+)]
1.Patternmatchonfieldsofinterest
53. if (fields['log_level'] == “fatal”):
self.fatals += 1
elif (fields['log_level'] == “error”):
self.errors += 1
elif (fields['log_level'] == “warning”):
self.warnings += 1
...
2.Aggregatevalues (sum,average,percentile,etc.)
71. $g = new Graphite($time);
$g->setTitle('File Not Found');
$g->addMetric('webs.errorLog.notExist', '#00cc00');
echo $g->getDashboardHTML(280, 220);
SuperEasy!
github.com/etsy/dashboard
74. http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1
webs.errorLog.warning,1318444930,1318448530,60|
5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,
1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,
1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.
0,1.0,1.0,None
82. Merci!
These slides will be available at
mikebrittain.com/talks
codeascraft.etsy.com
github.com/etsy
Say“Hello!”
mike@etsy.com
@mikebrittain
Metrics-DrivenEngineering