Measuring what matters

© AKAMAI - EDGE 2016
Introductions
Cliff Crocker
Akamai
Buddy Brewer
SOASTA

Three eras of modern performance measurement
2001-2006 Basically one browser
2007-2010 Transitional period
2011-2016 WTF just happened

2001-2006: Internet Explorer 6
2001 - 2006

2007-2010: Transitional period
2007
IE7
iPhone
Firefox 2
2008 2009 2010

2007
IE7
iPhone
Firefox 2
2008
IE7
iPhone 3G
Firefox 3
Android
Chrome 1
2009 2010

2007
IE7
iPhone
Firefox 2
2008
IE7
iPhone 3G
Firefox 3
Android
Chrome 1
2009
IE8
iPhone 3GS
Firefox 3
Android 2.1
Chrome 1
2010

2007
IE7
iPhone
Firefox 2
2008
IE7
iPhone 3G
Firefox 3
Android
Chrome 1
2009
IE8
iPhone 3GS
Firefox 3
Android 2.1
Chrome 1
2010
IE8
iPhone 3GS
Firefox 3
Android 2.1
Chrome 1
iPad
Nav Timing

2011-2016: WTF just happened
?
How many browsers in a
typical Thanksgiving
weekend?

2011-2016: WTF just happened
572
How many browsers in a
typical Thanksgiving
weekend?

Meanwhile...

So what can I measure?

What is available today - Synthetic Tooling
Pros
CPU metrics
Test environments & other places without real users
Access to everything including headers
Cons
Gaps in: Geos, browsers, pages
Impossible to precisely match user demo- & technographics
No way to realistically measure browser cache impact

What is available today - RUM
Pros
Near perfect coverage of browsers, geos, pages
Ability to understand impact to performance
Realistic measurement of browser cache impact
Cons
Gaps in: Geos, browsers, pages
Impossible to precisely match user demo- & technographics
No access to HTTP header data

Engagement
Revenue
Coverage
Caching
CPU
Pre-prod
Availability
Page details
Resources
Headers
SyntheticReal

https://www.w3.org/wiki/Web_Performance/Publications
W3C - Performance Working Group

Navigation Timing API

Resource Timing API

Coming soon to a browser
near you! Measure the size of
assets in addition to
performance.
Resource Timing (2.0) - Transfer Size

● Measure performance using high-precision
timestamps
● Consists of “marks” and “measures”
● PerformanceMark: Timestamp
● PerformanceMeasure: Duration between two given
marks
Browser Support - User Timing API

Browser Support - Navigation Timing API

Browser Support - Resource Timing API

Browser Support - User Timing API

So what should I measure?

What matters depends on you

What matters to an ops engineer?
● Preference for binary metrics
● Looks for indicators of system performance
● Affinity to network and back-end metrics
MTTR

Network
Server

Network
Server
function getPerf() {
var timing = window.performance.timing;
return {
dns: timing.domainLookupEnd -
timing.domainLookupStart,
connect: timing.connectEnd -
timing.connectStart
};
}
ttfb: timing.responseStart -
timing.connectEnd};

Help answers questions like...
● Do I have an issue with an upstream DNS resolver?
● Am I having Tier1 connectivity issues?
● How fast is my application server responding?

Don’t be misled by “super fast” responses
● DNS Caching
● Prefetching
● Connection reuse
● Pages served from browser cache

Object Level
Timing
● Resource timing API
● Binary metric for testing infrastructure
● Measure performance of a third-party
● CDN healthcheck

Object Timing
var rUrl =
‘https://edge.akamai.com/ec/us/multimed
ia/images/edge/akamai-edge-conference-s
peaker-buddy-brewer.jpg’;
var me =
performance.getEntriesByName(rUrl)[0];
var timings = {
loadTime: me.duration,
dns: me.domainLookupEnd -
me.domainLookupStart,
tcp: me.connectEnd - me.connectStart,
waiting: me.responseStart -
me.requestStart,
fetch: me.responseEnd -
me.responseStart
}

What matters to a front-end developer?
Optimized
Load Times
● “Do no harm”
● Looks for indicators of application performance
● Looks to measure a “fully loaded” page
● Affinity for front-end metrics

Client

function getPerf() {
var timing = window.performance.timing;
return {
Basepage: timing.responseEnd - timing.responseStart,
DLoading: timing.domLoading – timing.navigationStart,
DInt: timing.domIneractive – timing.navigationStart,
DContLoaded: timing.domContentLoadedEventEnd –
timing.navigationStart,
DContLoadTime: timing.domContentLoadedEventEnd –
timing.domContentLoadedEventStart,
DComplete: timing.domComplete - timing.navigationStart,
PLoad: timing.loadEventStart - timing.navigationStart
};
}
Client

Be wary of onload...
~1s OnLoad ~3.5s visibly complete

What matters to a designer?
UX
● Performance by design (D4P)
● Looks for indicators of human perception
● How quickly the page renders
● Affinity for custom metrics and render metrics

RUM first paint and start render
Chrome - “First Paint” Visual Start Render

User perception
Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html

Custom Metrics

Custom Metrics - User Timing API
performance.mark(“startTask”);
//Some stuff you want to measure happens here
performance.mark(“stopTask”);
//Measure the duration between the two marks
performance.measure(“taskDuration”,“startTask”,“stopTask”);
//Measure the load of an image
<img src=”image1.gif” onload=”performance.mark(‘image1’)”>

What matters to a business executive?
$$$$$$$
● Business impact analysis
● Simplified/lean metrics
● Actionable data

What matters to a business executive?

Don’t create unintended silos
Biz
Ops
Dev
Design

Don’t create unintended silos
The success of my
business is a
reflection of...
The digital experience I
provide, which depends
on...
how quickly I can render
content in the browser,
which is impacted by...
the performance and
stability of my infrastructure.
1
2
3
4
Biz
Design
Ops
Dev

“I just want to buy my Mom’s
birthday present.”
“I just want to know if it will
rain today.”
“I just want to share pictures
of my vacation”

How do I collect it?

Real users are not normal
What do I do with it?

2.9s - p50 (median)
4.4s - Average
10.4s - p95

What should I throw out?

Load Time =
3500 years

Load Time =
-(10years)

What should I keep?

Other opportunities for bots...
1. How does performance impact bot behavior?
2. Are bots skewing the numbers I’m reporting?
3. Am I providing an optimal experience for search bots?
4. Does any of this impact SEO?

Summary
● We have amazing tools at our disposal - your users are
wired for sound!
● Measuring what matters is situation independent
● Important to look at all metrics from the same context
● Use robust statistics when looking at your data

Measuring what matters

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Measuring what matters

Similar to Measuring what matters (20)

More from Cliff Crocker

More from Cliff Crocker (7)

Recently uploaded

Recently uploaded (20)

Measuring what matters