Metrics Driven Engineering

Metrics-DrivenEngineering
Mike Brittain
ENGINEERING DIRECTOR, ETSY
@mikebrittain

PROCESSANDTOOLS
SupportingacultureofContinuousDeployment

Howmanynewvisitors?
Howmanylistingscreated?
Howmanyregistrations?
HowdopeopleuseEtsy?
Howmanymessagessent?
Howmanypurchases?
Howmanynewshops?

Searchindexing?
Howfastarepagesgenerating?
Asynctaskscurrentlyinqueue?
Howistheapplicationbehaving?
DeveloperAPIauthandratelimiting?
Imagesresizedandstored?
Errorandwarningrates?

Replicationslavelag?
Memcachehits/misses?
Availableconnections?
AretheserversandnetworkOK?
Databasequeriespersecond?
Totaloutgoingbandwidth?
CPU,Memory,I/O?

Metricshelpyouidentifygoals
...butalsotellyouwhenyou’vebrokensomething.

AlwaysBeShipping
credit: ibailemon (ﬂickr)

1st day
Putyourselfonthewebsite.

2nd day
Completetax,insurance,and
beneﬁtsforms.
credit: ktpupp (ﬂickr)

DevSandbox Trunk/master Production
You!
Test

7e9a814 -> 63a2bb3
Deploy to Production

50+Deploys/day
200+Committers
15Productteams
8Infrastructureteams
50+Deploys/day
credit: misswired (ﬂickr)

PeerReview
Codereviews,Architecturereviews,Operabilityreviews

AutomatedTests
Staticanalysis,Unittests,Integrationtests,Functionaltests

May2013
$102.9Millioningoodsold
1.37Billionpageviews
https://www.etsy.com/blog/news/2013/etsy-statistics-may-2013-weather-report/

Failureisnotanoption
inevitable

Failureisnotanoption
anddetectable!
inevitable

Soundslikealotofwork,who’sgoingto
buildallofthis?
Q:

Well,theOpsteammanagesthenetwork,racks
theservers,installedthemonitoringtools,wears
thepagers,blah,blah,blah...
A:
Soundslikealotofwork,who’sgoingto
buildallofthis?
Q:

OPS
Logging
Graphing
Trending
Alerting
ENG

Metricsarepartofeveryfeature
(andsoareconﬁgﬂags)

Ganglia (application,servers,network)
Logster* (application,servers)
Cacti (network,SNMP)
FITB* (network)
*github.com/etsy
Simple,open-sourcetools
Graphite (application)
Statsd* (application)
Logformats (application,servers)
Nagios (alerting)

Cluster-oriented
Hugecommunitycontributedrecipes
Custommetrics(gmetad)
Ganglia

Single-instance
Createnewmetricson-the-ﬂy
CustomizeviaURLsanddisplayfunctions
http://www.aosabook.org/en/graphite.html
Graphite

Time, remote address, http method, request uri, referrer, user-agent,
response size, response code, execution time, memory consumed,
plus custom ﬁelds...
• Signed-in/out (user_id vs. “-”)
• display mode (“desktop” vs. “mobile”)
• i10n/i18n (“en-US”)
• etc.
AccessLogs

LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...
apache_note(“custom_ﬁeld”, $whatever);

LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /
listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1"
200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://
www.google.com/bot.html)" desktop - 13399576 505780 554876

Logger::error("User login failed. Reason:
$msg for $email_addr", “login”);
Methodnamedenoteslog“level”—
error,fatal,warning,notice,debug.
A“namespace”parameterisprovided
sowecanaggregatelogentrieswith
similarconcerns.

Logger::error("User login failed. Reason:
$msg for $email_addr", “login”);
web0054 [Fri Mar 04 16:27:48 2011] [error] [login]
[mk04gw1p71] User login failed. Reason: wrong
password was submitted for mike@etsy.com
UniquerequestID
Servername
Dateandtime Level
Namespace

web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conve
web0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.
web0012 [Fri Mar 04 16:27:48 2011] [warning] [login] [mk04gw1p71] User login failed.
web0081 [Fri Mar 04 16:27:48 2011] [error] [register] [39e08e6692] Duplicate user ID
web0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset co
web0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user ID
web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Re
web0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID enc
web0012 [Fri Mar 04 16:27:49 2011] [warning] [login] [32976da59c] User login failed.
web0041 [Fri Mar 04 16:27:49 2011] [fatal] [login] [mk04gw1p71] Invalid charset conve
web0012 [Fri Mar 04 16:27:49 2011] [error] [login] [2f297b40a5] User login failed. Re
web0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faile
web0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faile
web0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faile
web0035 [Fri Mar 04 16:27:50 2011] [warning] [login] [2f297b40a5] User login failed.
web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failed
web0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Re
web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failed
web0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset c
web0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user ID
web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.
web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.

web0054 [Fri Mar 04 16:27:48 2011] [error] [login] [mk04gw1p71] Invalid charset conve
web0102 [Fri Mar 04 16:27:48 2011] [warning] [login] [47dd608551] User login failed.
web0100 [Fri Mar 04 16:27:49 2011] [fatal] [register] [f9c2b23702] Invalid charset co
web0050 [Fri Mar 04 16:27:49 2011] [error] [register] [2e468a9bb6] Duplicate user ID
web0200 [Fri Mar 04 16:27:49 2011] [error] [login] [f9c2b23702] User login failed. Re
web0064 [Fri Mar 04 16:27:49 2011] [error] [login] [47dd608551] Duplicate user ID enc
web0025 [Fri Mar 04 16:27:49 2011] [warning] [register] [32976da59c] User login faile
web0088 [Fri Mar 04 16:27:49 2011] [warning] [register] [2e468a9bb6] User login faile
web0050 [Fri Mar 04 16:27:50 2011] [warning] [register] [39e08e6692] User login faile
web0072 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [2f297b40a5] User login failed
web0050 [Fri Mar 04 16:27:50 2011] [error] [login] [2e468a9bb6] User login failed. Re
web0200 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [f9c2b23702] User login failed
web0064 [Fri Mar 04 16:27:50 2011] [error] [subscribe] [47dd608551] Invalid charset c
web0012 [Fri Mar 04 16:27:50 2011] [error] [register] [2f297b40a5] Duplicate user ID
web0088 [Fri Mar 04 16:27:50 2011] [warning] [login] [2e468a9bb6] User login failed.
web0050 [Fri Mar 04 16:27:51 2011] [warning] [login] [39e08e6692] User login failed.
FATALS ERRORS WARNINGS
Logster

github.com/etsy/logster
Runbycron(e.g.1mintervals)
Keeps acursoronyourlogﬁle
Parseandaggregatevalueshoweveryouwant
OutputtoGanglia,Graphite,AmazonCloudWatch
Simpleparsers
Logster

web0054 [Fri Mar 04 16:27:48 2011] [error] [login]
[mk04gw1p71] User login failed. Reason: wrong
password was submitted for mike@etsy.com
^.+ [.+] [(?P<log_level>.+)]
1.Patternmatchonﬁeldsofinterest

if (fields['log_level'] == “fatal”):
self.fatals += 1
elif (fields['log_level'] == “error”):
self.errors += 1
elif (fields['log_level'] == “warning”):
self.warnings += 1
...
2.Aggregatevalues (sum,average,percentile,etc.)

MetricObject("fatals",
(self.fatals / self.duration), "per sec")
MetricObject("errors",
(self.errors / self.duration), "per sec")
MetricObject("warning",
(self.warnings / self.duration), "per sec")
3.Sendthevaluesas“metricobjects”tothecollectors

github.com/etsy/logster
FATALS ERRORS WARNINGS
Logster

github.com/etsy/statsd
StatsD
Networkdaemon(node.js)
AcceptsdataoverUDP
FlushestoGraphiteevery10sec
One-lineofcode

StatsD::increment("logins.success");

StatsD::increment("logins.success");
Logins

StatsD::timing("profile.time", $msec);

StatsD::timing("profile.time", $msec);
90thpct
average
lower

echo "events.deploy.site 1 `date +%s`"
| nc graphite.etsycorp.com 2003

VerticalLineTechnology!
target=drawAsInfinite(events.deploy.site)

250,000+metricsatEtsy
Systems,Applications,Business

github.com/etsy/dashboard
Dashboards

<a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or
+Script+Not+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite
%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production
%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
<img src="http://graphite.etsycorp.com/render?
from=-1hours&width=280&height=220&title=File+or+Script+Not
+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite
%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production
%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
</a>
KindofHard :-/

$g = new Graphite($time);
$g->setTitle('File Not Found');
$g->addMetric('webs.errorLog.notExist', '#00cc00');
echo $g->getDashboardHTML(280, 220);
SuperEasy!

But,yousaid...
“250,000+metricsatEtsy”
Systems,Applications,Business

http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1

http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1
webs.errorLog.warning,1318444930,1318448530,60|
5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,
1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,
1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.
0,1.0,1.0,None

Holt-WintersConﬁdenceBands
lower
upper

Businessmetrics
+ Conﬁdencebands
_____________
Alertablemetrics

Metrics!
Metrics+Events
Metrics+Alerts
Metrics+Metrics

High-level,real-timevisibility

Detectproblemsearly,
andresolvethemquickly.

Makethemaccessible
Makethemrequiredfeatures
Makethemdeadsimple

Merci!
These slides will be available at
mikebrittain.com/talks
codeascraft.etsy.com
github.com/etsy
Say“Hello!”
mike@etsy.com
@mikebrittain
Metrics-DrivenEngineering

Metrics Driven Engineering

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (10)

Recently uploaded

Recently uploaded (20)

Metrics Driven Engineering