AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018

https://www.ﬂickr.com/photos/9422878@N08/6884612488
AB Testing, Ads & other 3rd-Party Tags
@AndyDavies · Smashing Conf · Feb ‘18

Infrastructure and code
managed by someone else
Third-party =
Guy Podjarny, Velocity Santa Clara, 2014

We’re including more and more of them
0
25
50
75
100
125
150
175
200
225
01-Jan-13 01-Jul-13 01-Jan-14 01-Jul-14 01-Jan-15 01-Jul-15 01-Jan-16 01-Jul-16 01-Jan-17 01-Jul-17 01-Jan-1
50th 75th 90th 95th HTTPArchive Jan ’13 to Jan ‘18Percentile

Well… we’ve got plenty of choice!

And we’re told they’re going to make us more money
www.ﬂickr.com/photos/59937401@N07/5856660723

EVERYTHING SHOULD
HAVE VALUE, BECAUSE
EVERYTHING HAS A COST
Tim Kadlec

How can we measure their cost?
1. Test a page in WebPageTest
2. Repeat with selected 3rd-Parties blocked
3. Compare the two results

Use the filmstrip view to compare visual experience

Feed the timings into RUM to understand the business impact
Conversions
(At least three RUM products support ‘What-If’ type analysis)

MedianPageLoadTime(s)
0
2
4
6
8
10
12
14
W
eek
0
W
eek
1
W
eek
2
W
eek
3
W
eek
5
W
eek
5
Android
iOS
3rd-Parties really can cost you money…

MedianPageLoadTime(s)
0
2
4
6
8
10
12
14
W
eek
0
W
eek
1
W
eek
2
W
eek
3
W
eek
5
W
eek
5
Android
iOSRemoved BazaarVoice
for Android visitors
3rd-Parties really can cost you money…

Understand how 3rd-parties affect your key milestones
Working? Useful? Usable?
What’s happening here? Or here?

Analysing activity before firstMeaningfulPaint

How do we do this for many pages?
1. Grab Chrome timelines from HTTPArchive
2. Analyse with modiﬁed version of Jean-Pierre
Vincent’s 3rd-party-cpu-abuser
3. Aggregate the results

URL Count Mean (ms) SD (ms)
www.google-analytics.com 115 232 85
cdn.optimizely.com 110 440 329
pagead2.googlesyndication.com 68 375 266
assets.adobedtm.com 62 335 243
ajax.googleapis.com 59 259 114
securepubads.g.doubleclick.net 57 380 367
connect.facebook.net 47 238 89
platform.twitter.com 22 241 127
s7.addthis.com 22 360 218
tags.tiqcdn.com 20 294 137
nexus.ensighten.com 17 368 230
www.googletagmanager.com 15 317 413
apis.google.com 13 317 153
cdn.onesignal.com 12 270 136cdnjs.cloudﬂare.com 12 244 87
Top 15 CPU consumers (by frequency) - HTTPArchive rank < 5000

Let’s look at testing, using Optimizley as an example
https://www.ﬂickr.com/photos/pinkangelbabe/427255017

Optimizely’s cost varies by implementation
0
1.25
2.5
3.75
5
Duration of script execution before ﬁrstMeaningfulPaint (ms)
0 500 1000 1500 2000 2500
Optimizely
(n=110)
Circle size = number occurrences

On the critical rendering path

Improving performance
Reducing number of
• URL Matches
• Experiments
• Plugins (Widgets)
reduces snippet size, download and execution times
Removing jQuery from the bundle and using your own (if you
already have it) can shrink bundle further

Add metrics from Optimizely to Analytics or RUM?
optimizely.get("state").getActiveExperimentIds().length
Some products have documented APIs, why not extract key
metrics and send them to analytics or RUM?
Number of experiments in snippet:
Number of experiments active for this page view:
Object.keys(optimizely.get('data').experiments).length

Use Performance APIs to build a picture
Resource Timing
• When was script requested, how long did it take to fetch, was it
cached?
User Timing
• When did the initial script execute, how long did it take?
LongTasks
• Were there any script elements that took longer that 50ms?

Make sure tests are equivalent from a performance viewpoint
Some things seen in the the wild:
• One variant redirected to a new page, the other didn’t
• Optimised vs unoptimised images
(Can be good reasons why tests can’t be equivalent)

Could you do it server-side or or at CDN level?
https://www.ﬂickr.com/photos/kewl/8475764430

Know how to turn it off in an emergency
https://www.ﬂickr.com/photos/chrysics/4365802409

https://www.ﬂickr.com/photos/gviciano/3128575291
What if I told you…

https://www.ﬂickr.com/photos/gviciano/3128575291
a script loaded async, and even
injected after page load can delay a
page rendering?
…

What was the first request in this navigation?

beforeUnload event
took 400ms to execute!
cloning DOM triggered
extra network requests

beforeUnload event
handlers took over
1,800ms to execute!!!

The examples aren’t isolated cases
0
3
6
9
12
15
Duration of beforeUnload event handlers (s)
0.0 0.5 1.0 1.5 2.0 2.5 3.0
Moat Ads
(n=481)
TeaLeaf
(n=271)

Not always easy to spot these delays
https://www.ﬂickr.com/photos/fernandosanchez/2441716057

Navigation Timing API excludes beforeUnload
Prompt
for
unload
App
Cache
DNS TCP Request Response Processing onLoadRedirect
unload
navigation start

Navigation Timing API excludes beforeUnload
Prompt
for
unload
App
Cache
DNS TCP Request Response Processing onLoadRedirect
unload
Before
Unload
navigation start

Instrument addEventListener?
!(function() {
const callbacks = []
window.addEventListener = (function(_addEventListener){
_addEventListener.call(window, 'beforeunload', function(){
const args = Array.prototype.slice.call(arguments)
let total = -1 * performance.now()
callbacks.forEach(function(callback) {
callback.apply(undefined, args)
})
total += performance.now()
console.info(`total: ${total}`)
})
return function() {
if (arguments[0] === 'beforeunload') {
callbacks.push(arguments[1])
return
}
_addEventListener.apply(this, arguments)
}
})(window.addEventListener)
})() This is just a sketch of an idea - don’t deploy it without improvements!

https://gist.github.com/cvazac/de18cf4d045b01d179d8fdc2902d4997

WebPageTest?
1. WebPageTest script
navigate https://www.game.co.uk/ 
navigate https://www.game.co.uk/
2. Download Timeline for
step #2
3. Pipe through jq cat timeline.json | jq -r '[.[] |
select(.name=="EventDispatch"
and .args.data.type=="beforeunload")
| .dur] | add / 1000')

https://www.ﬂickr.com/photos/wnieke/4078161933

Do you include adverts?
https://www.ﬂickr.com/photos/helldini/6190516041

https://news.sky.com/story/hackers-behind-a-billion-dodgy-web-adverts-11219278

https://blog.trendmicro.com/trendlabs-security-intelligence/malvertising-campaign-abuses-googles-doubleclick-to-deliver-cryptocurrency-miners

https://discuss.httparchive.org/t/the-performance-impact-of-cryptocurrency-mining-on-the-web/1126/8

https://www.adlightning.com/ad-quality-report
“…41% of the ads tracked exceeded the industry-approved
maximum. Nearly 10% of the ads were larger than 5MB.
Some ads are as large as 30 MBs.”
“…average number of network requests and tracking scripts
per ad was 56:
3.7x greater than the maximum suggested by the IAB.”

“…one third (32%) of the ads were overly processor-
intensive.
The study also found it common for processor-intensive ads,
often video ads, to consume more than 3 seconds of CPU
time to render a single ad within the user’s browser.”
4% of video ads were in Flash format and 19% of in-banner
video ads were auto-play.
https://www.adlightning.com/ad-quality-report

And we wonder why
people use adblockers?

https://www.iab.com/newadportfolio/
Advertisers are trying to improve quality

Performance APIs can help to build a picture (within limits)
Resource Timing
• What was requested and when, how long did it take to fetch, was it
cached?
LongTasks
• Were there any script elements that took longer that 50ms?
But there are still plenty of blind spots

Can some of the security features of the web help?
Tried using
• Content-Security-Policy
• Subresource Integrity
to help a client police ads last year but didn’t have much success

TagManagers… A blessing or a curse?
https://www.ﬂickr.com/photos/jdhancock/9004372146

Makes it really easy to add new tags
https://www.ﬂickr.com/photos/lantzilla/29842854

Makes it easy to remove them too!
https://www.ﬂickr.com/photos/alesimages/4215559895

Provides a shared source of truth and audit trail
https://www.ﬂickr.com/photos/tim_ellis/3919090513

Help control loading order
https://www.ﬂickr.com/photos/rusty_clark/32889333561

Ultimately it’s about finding the balance
https://www.ﬂickr.com/photos/gilgamesh/183037111

VALUE
COST
EVERYTHING SHOULD
HAVE VALUE, BECAUSE
EVERYTHING HAS A COST
Tim Kadlec

Understand the COST
Establish the VALUEVALUE
COST

https://www.ﬂickr.com/photos/basheertome/4762529213

Thanks!
@AndyDavies
www.slideshare.net/andydavies

AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018

Similar to AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018 (20)

More from Andy Davies

More from Andy Davies (19)

Recently uploaded

Recently uploaded (20)

AB Testing, Ads and other 3rd party tags - SmashingConf London - 2018