Web Performance Culture and Tools at Etsy

Web Performance
Culture and Tools
at Etsy
Mike Brittain
Dir. of Engineering, Infrastructure
Etsy
Boston Web Performance Meetup
Aug 24, 2011

Overview
Etsy and Engineering
Make Performance
Matter
Tools and Process

Monthly
Weather Report
$38 MM in sales
1.9 MM items sold

http://etsy.me/weather-report-june-2011

Monthly
Weather Report
990 MM page views

http://etsy.me/weather-report-june-2011

Engineering
Technologies
Linux, Apache, MySQL,
PHP, Memcache
Solr, Squid, Hadoop, Amazon S3, EC2, EMR
Ganglia, Cacti, Nagios, Graphite, Splunk, and
some of our own...

Engineering
Teams
90 Engineers
3-6 engineers
+ product developer and/or designer

Engineering
Continuous
Deployment
~40 releases per day
inc. app code and conﬁg changes

1. Write code
2. Code review
3. Automated tests

1. Write code
2. Code review
3. Automated tests
4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod

1. Write code
2. Code review
3. Automated tests
4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
5. Monitor!
6. Monitor!
7. Monitor!

Engineering
Data-Driven
Development
45,000+ metrics
50+ dashboards

Make
Performance
Matter
Have a story

Business Impact
Measure
“Our own years of testing have conclusively
shown that when speed of a feature or
product improves, usage, quite simply,
goes up.”
~Google

Abundance of research from Google, Bing,
AOL, Amazon, Shopzilla, etc.
http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

Business Impact
Measure
Bounce rate
Search conversion

Business Impact
Measure
Purchase funnel
Ad impressions, page views and
tracking discrepancies
Social, engagement

Operations
Site Stability
Measure perf for discrete pieces
of your infrastructure
Contention for shared resources like
database, memcache, solr, or even web
server processes

Getting to work
Where to
Start?
Focus your eﬀorts where it makes
sense for your business

Getting to work
Where to
Optimize?
Tiers of service
SLAs

Getting to work
Focus
Beware of CTS:
Constant Tweaking Syndrome

Getting to work
Focus
Friends don’t let friends
tweak without graphing

Make
Performance
Matter
Have your story

Process
Etsy’s
Perf Team
Standardize patterns
Create tools and reports
Coordinate eﬀorts

Process
Server-side
Performance
There is nothing worse
than a blank page or a spinning
thing-a-ma-jigger

Process
Server-side
Performance
95th Percentile
> 800 ms is a P2 bug

Process
Server-side
Performance

Tools
Logging
Perf-related data belongs in
your server logs
apache_note()

Tools
Logging
$timer_start = microtime(true);
...
$timer_diff = microtime(true) - $timer_start;

register_shutdown_function()

apache_note('php_microsec', $timer_diff);

Tools
Logging
LogFormat "%{True-Client-IP}i %l %u %t "%r"
%>s %b "%{Referer}i" "%{User-Agent}i"
%{etsy_user_id}n %{php_bytes}n %{php_microsec}n
%D" combined

Tools
Logging
web0060 66.249.71.110 - - [24/Aug/2011:04:16:52
+0000] "GET /listing/12189259/tropical-etched-
pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://
www.google.com/bot.html)" - 13399576 505780
554876

Tools
Analyze
grep ... access.log | awk ...
grep "/listing/" access.log |
awk '{sum=sum+$(NF-1)} END {print sum/NR}'

grep "/listing/" access.log |
awk 'BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)}
END {print max}'

Tools
Analyze
1. Capture perf data in logs
page gen., Boomerang, Gomez
2. Aggregate
Splunk, Logster, StatsD
3. Record in Graphite
http://graphite.wikidot.com

Tools
Logster
1. Read new log entries every
minute
2. Parse and aggregate useful
numbers
3. Send to Graphite
http://github.com/etsy

web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllpp
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a g
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a g
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.

Tools
StatsD
Collects metrics from your app
code and sticks them in Graphite
StatsD::increment("logins.success");
StatsD::timing("gearman.time", $msec);

Tools
StatsD
90th pct

average
lower

Tools
I/O Proﬁler
Lightweight, inline proﬁling
Start and end times wrapped
around service calls
databases, memcache, apc, etc.

Tools
Client-side
Testing
Gomez (API), Boomerang,
WebPagetest

Tools
ShowSlow
Tracks front-end best practices

Tools
ShowSlow
Internal instance
Automated testing
End-point for I/O Proﬁler
Trending on individual rules
http://showslow.com

Other thoughts
YSlow &
Page Speed
Mind your development process
3rd-party content and ga.js
“Use a CDN”

Other thoughts
Device-Speciﬁc
Design
The mobile web is very much
about designing for performance
Screen size, pixel density, connection speed,
wi-ﬁ vs. cellular, browser cache size, local
storage, connections per host, metered
pricing, etc.

Make
Performance
Matter
Have your story
Use tools and process
to focus your eﬀorts

We are
Hiring
http://etsy.com/careers
Software Engineering positions available in
a number of teams, including Analytics,
Operations, Web Performance, Payments,
Core Platform, Front-End, Internal Apps,
Search, Security, and more...

Thank
You
Mike Brittain
mike@etsy.com
@mikebrittain
CodeAsCraft.etsy.com
github.com/etsy
graphite.wikidot.com
showslow.com

Web Performance Culture and Tools at Etsy

Recommended

Recommended

More Related Content

What's hot

What's hot (16)

Viewers also liked

Viewers also liked (19)

Similar to Web Performance Culture and Tools at Etsy

Similar to Web Performance Culture and Tools at Etsy (20)

Recently uploaded

Recently uploaded (20)

Web Performance Culture and Tools at Etsy