Native advertising is a specific form of online advertising where ads replicate the look-and-feel of their serving platform. In such context, providing a good user experience with the served ads is crucial to ensure a positive user experience and hence long-term user engagement. In this talk, I will describe work at Yahoo aiming at understanding the user experience on ads in the mobile context and building learning frameworks to identify and account for ads of low quality while ensuring a return of investment to advertisers.
Slides for the Invited Talk at BigData Innovators Gathering (BIG), co-located with WWW 2017, Perth 2017 (https://big2017.org). Earlier versions of this talk were given at various venues in London.
2. This talk
4-year effort across research, engineering and product at Yahoo to
measure the quality of ads served on Gemini, Yahoo native advertising
network
Not just measuring but taking actions to improve user experience as well
as providing feedbacks to advertisers
à no deep learning but large scale predictive analytics
Focus of the talk: the post-click experience on native ads
à the quality of the landing page… if it is seen
5. Advertising is how Yahoo (and many other Internet companies) makes
money… and what keeps Yahoo services free for its customers
90% of Yahoo’s revenue is from advertising:
2016 search advertising revenue – $2.67B (52% of total revenue)
2016 display including native advertising – $1.98B (38% of total)
Scale: billion ads served daily
(Source: Yahoo 2016 10k annual report)
6. Online advertising is about connecting supply & demand
Search
Native
Display
Video
Brand
Direct
Response
Yahoo own &
operated sites
Publisher Partners
SUPPLY
(publishers)
DEMAND
(advertisers)
7. Advertising (ad) quality science
Develop predictive models that characterise the quality of ads shown to and clicked by users.
Maximise revenue and guaranteeing ROI to advertisers without
negatively impacting user experience.
Publishers
Advertisers
Users
Ad
inventory
ad network
Being able to help
advertisers improve the
quality of their ads
11. The quality of the post-click experience
the quality of the landing page on mobile
12. The post-click experience: Dwell time
dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
13. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
14. Quality of the post-click experience
Best experience is when conversion happens
No conversion does not mean a bad experience
Proxy metric of post-click experience: dwell time on the ad landing page
tad-click tback-to-publisher
dwell time = tback-to-publisher – tad-click
Positive post-click experience (“long” clicks)
has an effect on users clicking on ads again
15. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
16. Optimise for high quality ads
Estimating P(hq|click) = quality score
P(dwell time > t)
Build predictive models that predict if an ad is of high quality
= predicted dwell time above a given threshold t
➔ high quality = high dwell time
revenue = 𝓕 (bid, CTR, quality)
P(hq|click)
logistic regression, gradient descent boosting, random forest, survival random forest
17. Landing page features
● window_size
● view_port
● media_support
content
mobile
support
requested
information
multimedia
mobile
optimized
out-going
connectivity
interactivity
textual
content
in-coming
connectivity
● description
● keywords
● title
meta
information
● num_forms
● num_input_radio
● num_input_string
● ...
readability multimedia
significant effect of text readability and page structure
18. A/B testing
dwell time increased by 20%
bounce rate decreased by 7%
revenue = bid x CTR x quality
(Lalmas etal, 2015; Barbieri, Silvestri & Lalmas, 2016)
19. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
20. Landing page rating: Low, Average or High
landing
pages quality score q
…
L H
L and H are customisable:
e.g., LOW=[0,25%), AVG=[25%,75%], HIGH=(75%,100%]
2 cut-off points (L, H) that
divide distribution of quality
scores q into 3 regions:
- LOW: q < L
- AVG: L <= q <= H
- HIGH: q > H
(L, H)
ad ratingq LOW
21. Improving landing pages
Exploiting the features for recommending improvements
mobile
optimized
out-going
connectivity
interactivity textual
content
in-coming
connectivity
meta
information
readability
multimedia
● num_forms
● num_input_radio
● num_input_string
● ...
interactivity
● mediannum_forms ±ε
● mediannum_input_radio ±ε
● mediannum_input_string ±ε
● ...
for each feature
compute median and
confidence interval
for each ad feature
compute the distance from
the confidence interval
given an ad
num_input_radio
num_forms
num_input_string
...
22. There might be too few/much textual content
There might be too few/many entities
There might be too few/many images
Landing Page Content
n. of words
n. of Wikipedia en;;es
n. of images
Landing Page Layout
height/width
resizability (fit to mul;ple screen size)
Landing Page Structure
n. of drop-down menus
n. of checkboxes
n. of input strings
Landing Page Readability
content summarizability
The height/width of the landing page might be too small/large
The landing page might not be adapted to different screen
sizes
There might be too many drop-down menus
There might be too many checkboxes
There might be too much information requested from the
users
The textual content might be further summarised to make it more
readable
Examples of recommendations
23. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
24. peak on app X
● accidental clicks do not
reflect post-click
experience
● not all clicks are equal
app X
The quality of a click on mobile apps
peak on app Y
dwell time distribution of apps X and Y for a
given ad
app Y
25. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
The post-click experience journey
26. Fitting the data into a mixture model
The number of mixture components is determined using the BIC criterion
which selects the model that fits best the data while avoiding overfitting
Time period 1
Time period 2 (after UI change)
Bayesian information criterion (BIC)
bouncy clicks
accidental clicks
27. Accidental clicks threshold for app X
Min
1st Quartile
Median
Mean
3rd Quartile
Max
Distribution of the medians as computed on the first
component of each ad
Applications
- discount accidental clicks
using economics models
- train click models
discarding accident clicks
- input to UI design
ads with all three components
28. dwell time
proxy of post-click ad
quality experience
predictive post-click ad
quality models
ad quality ratings &
recommendations
accidental clicks
identification
proxy of accidental
clicks
Metric
Ad serving
User
engagement
Advertiser
ROI
Ad quality: The post-click experience journey
Acknowledgments: Marc Bron, Ayman Farahat, Andy Haines, Miriam Redi, Gabriele Tolomei, Guy Shaked,
Ke (Adam) Zhou, Fabrizio Silvestri, Michele Trevisiol, Ben Shahshahani, Puneet M Sangal and many others