1. User Engagement:
from Sites to a Network of Sites
or
The Network Effect Matters!
Ricardo Baeza-Yates
Mounia Lalmas
Yahoo! Labs Barcelona
Joint work with Janette Lehmann and Elad Yom-Tov
and many others at Yahoo! Labs -1-

2. Outline
o Motivation, definition and scope
o Models of user engagement
o Networked user engagement
-2-

3. Motivation, Definition and Scope
o Definition and scope
o Characteristics of user engagement
o Measures of user engagement
o Our research agenda
-3-

4. User Engagement – connecting three sides
Quality of the user experience that emphasizes the positive aspects of
the interaction, and in particular the phenomena associated with
users wanting to use a web application longer and frequently.
Successful technologies are not just used, they are
engaged with
user feelings: happy, sad, user mental states: concentrated, user interactions: click, read
excited, bored, … challenged, lost, interested … comment, recommend, buy, …
The emotional, cognitive and/or behavioural connection
that exists, at any point in time and over time, between a
user and a technological resource
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),
WSDM Workshop on User Modelling for Web Applications, 2011.
-4-

5. Would a user engage with this web site?
http://www.nhm.ac.uk/ -5-

6. Would a user engage with this web site?
http://www.amazingthings.org/ (art event calendar)
-6-

7. Would a user engage with this web site?
http://www.lowpriceskates.com/ (e-commerce – skating)
-7-

8. Would a user engage with this web site?
http://chiptune.com/ (music repository) -8-

9. Would a user engage with this web site?
http://www.theosbrinkagency.com/ (photographer)
-9-

10. Characteristics of user engagement (I)
• Users must be focused to be engaged
Focused attention • Distortions in the subjective perception of time used to
measure it
• Emotions experienced by user are intrinsically motivating
Positive Affect • Initial affective hook can induce a desire for exploration, active
discovery or participation
• Sensory, visual appeal of interface stimulates, promote
Aesthetics focused attention
• Linked to design principles (e.g. symmetry, balance, saliency)
• People remember enjoyable, useful, engaging experiences
and want to repeat them
Endurability • Reflected in e.g. the propensity of users to recommend an
experience/a site/a product
- 10 -

11. Characteristics of user engagement (II)
• Novelty, surprise, unfamiliarity and unexpected
Novelty • Appeal to user curiosity, encourages inquisitive
behavior and promotes repeated engagement
• Richness captures the growth potential of an activity
Richness and control • Control captures the extent to which a person is able
to achieve this growth potential
• Trust is a necessary condition for user engagement
Reputation, trust and
• Implicit contract among people and entities which is
expectation more than technological
Motivation, interests,
incentives, and • Difficulties in setting up “laboratory” style experiments
benefits
- 11 -

12. Forrester Research – The four I’s
• Presence of a user
Involvement • Measured by e.g. number of visitors, time spent
• Action of a user
Interaction • Measured by e.g. CTR, online transaction, uploaded
photos or videos
• Affection or aversion of a user
Intimacy • Measured by e.g. satisfaction rating, sentiment
analysis in blogs, comments, surveys, questionnaires
• Likelihood a user advocates
Influence • Measured by e.g. forwarded content, invitation to join
Measuring Engagement, Forrester Research, June 2008-
- 12

13. Peterson et al Engagement measure - 8 indices
Click Depth Index: page views
Duration Index: time spent
Recency Index: rate at which users return over time
Loyalty Index: level of long-term interaction the user has
with the site or product (frequency)
Brand Index: apparent awareness of the user of the brand,
site, or product (search terms)
Feedback Index: qualitative information including
propensity to solicit additional information or supply direct
feedback
Interaction Index: user interaction with site or product
(click, upload, transaction)
Peterson etal. Measuring the immeasurable: visitor engagement, WebAnalyticsDemystified, September 2008 - 13 -

14. Measuring user engagement
Measures
Characteristics
Self-reported Questionnaire, interview, report, Subjective,
engagement
product reaction cards
user study (lab/online)
Mostly qualitative
Cognitive Task-based methods (time spent, Objective,
engagement
follow-on task) user study (lab/online)
Neurological measures (e.g. EEG) Mostly quantitative
Physiological measures (e.g. eye Scalability an issue?
tracking, mouse-tracking)
Interaction Web analytics + “data science” Objective,
engagement
(CTR, bounce rate, dwell time, etc) data study
Metrics and user models
Quantitative
Large scale
- 14 -

15. Interaction engagement – Online metrics
Proxy of user engagement
- 15 -

16. Diagnostic and what we can do
Diagnostic: work exists, but fragmented.
In particular:
o What and how to measure depend on services and goals
o Going beyond site engagement
What we have done:
1. Models of user engagement
2. Networked user engagement
3. Complex networks analysis
Future: Economic model for networked UE
- 16 -

17. Models of User Engagement
Online sites differ concerning their engagement!
Games Search
Users spend Users come
much time per frequently and
visit do not stay long
Social media Special
Users come Users come on
frequently and average once
stay long
Service News
Users visit site, Users come
when needed periodically
is it possible to model these differences and
compare different classes of sites? - 17 -

18. Data and Metrics
Interaction data, 2M users, July 2011, 80 USA sites
Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.
- 18 -

19. Diversity in User Engagement
Engagement of a site depends on users and time
Users and Loyalty Time and Popularity
Sites have different user groups Site engagement can be periodic or
contains peaks
Proportion of user groups is site-
dependent
mail, social
media
media
(special events)
media,
entertainment
daily activity, shopping,
navigation entertainment
- 19 -

20. Methodology
General models User-based models Time-based models
Dimensions 5 user groups weekdays, weekend
8 metrics 8 metrics per user 8 metrics per time span
group
#Dimensions 8 40 16
Kernel k-means with
Kendall tau rank correlation kernel
Num. of clusters based on eigenvalue distribution of kernel matrix
Significant metric values with Kruskal-Wallis/Bonferonni
#Clusters
(Models) 6 7 5
Analyzing cluster centroids = models
- 20 -

21. Models of user engagement (I)
Models based on engagement metrics
• 6 general models
• Popularity, activity and loyalty are independent from each other
• Popularity and loyalty are influenced by external and internal factors
 e.g. frequency of publishing new information, events, personal
interests
• Activity depends on the structure of the site
interest-
specific
periodic
media
e-commerce,
configuration
- 21 -

22. Models of user engagement (II)
Models based on engagement metrics, user and time
User-based [7 models] Time-based [5 models]
Models based on engagement per user Models based on engagement over
group weekdays and weekend
navigation game, sport
hobbies, daily news
interest-specific
 Sites of the same type (e.g. mainstream media) do not necessarily belong to
the same model
 The groups of models describe different aspects of engagement, i.e. they are
independent from each other
- 22 -

23. Relationships between models
 Groups of models are independent from each other
General User Time
General 0.00 3.50 4.23
User 3.50 0.00 4.25 Variance of Information
Time 4.23 4.25 0.00 [0,5.61]
 Example:
 Model mu2
[high popularity and activity in all user groups, increasing loyalty]
 50% to model mt2
[high popularity on weekends and high loyalty on weekdays]
 50% to model mt3
[high activity and loyalty on weekends]
- 23 -

24. Recap & Next
User engagement is complex and standard
metrics capture only a part of it
User engagement depends on users and time
First step towards a taxonomy of models of user
engagement … and associated metrics
Next
Interaction between models
User demographics, time of the day, geo-location, etc
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
- 24 -

25. Understanding the problem:
Users on Yahoo! network of sites
- 25 -

26. Networked user engagement:
engagement across a network of sites
Large online providers (AOL, Google, Yahoo!, etc.) offer
not one service (site), but a network of services (sites)
Each service is usually optimized individually, with some
effort to direct users between them
 Success of a service depends on itself, but also on how
it is reached from other services (user traffic)
Measuring user engagement across a network
of sites should account for user traffic
between sites.
- 26 -

27. Online multi-tasking
leaving a site is
not a “bad thing!”
users spend more and more of their online session multi-tasking, e.g. emailing,
reading news, searching for information  ONLINE MULTI-TASKING
navigating between sites, using browser tabs, bookmarks, etc
seamless integration of social networks platforms into many services
- 27 -

28. Online multi-tasking
Users switch between sites within an online
session (several sites are visited and the
same site is visited several times)
Navigation
Back Link Other1 Browser usage changed
button Less usage of back button
1995 35.7% 45.7% 18.6%
1997 31.7% 43.4% 24.9%
2006 14.3% 43.5% 42.2%
Oberdorf et al
1) Usage of tabs, bookmarks, typing the URL directly,…
2) http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/
- 28 -

29. Online multi-tasking
Users switch between sites within an online
session (several sites are visited and the
same site is visited several times)
Navigation Browser usage changed
More and more usage of tabs
Back Link Other1
button
1995 35.7% 45.7% 18.6%
1997 31.7% 43.4% 24.9%
2006 14.3% 43.5% 42.2%
Dubroy et al
Oberdorf et al] UX Movement2:
External links affect your site and
users
Links that take users to different
websites should open in new tabs.
1) Usage of tabs, bookmarks, typing the URL directly,…
2) http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/
- 29 -

30. Online multi-tasking
Users switch between sites within an online session
(several sites are visited and the same site is
visited several times)
Navigation
Back Link Other1
button
1995 35.7% 45.7% 18.6%
1997 31.7% 43.4% 24.9%
2006 14.3% 43.5% 42.2%
 A short visit does not mean less engagement
Measuring user engagement across a network
of sites should account for multi-tasking
- 30 -

31. Networked user engagement - Two studies
o Is there a network effect?
o study of 50 Yahoo! sites
o downstream engagement as a measure of networked user
engagement
o effect of stylistics (layout and structure)
o Can we quantify the network effect?
o study of 728 Yahoo! sites and traffic between them
o use metrics from the complex network area together with
engagement metrics to characterize networked user
engagement
- 31 -

32. Is there a network effect?
The success of a web site largely depends on itself, but
also on the network effect
This is particularly relevant for the
Yahoo! network of properties
o Can we measure the network effect?
o Downstream engagement
o Can we influence downstream engagement?
o Session types
o Link types
- 32 -

33. Measuring networked user engagement:
Downstream engagement
Downstream engagement
for site A
(% remaining session time)
Site A
tes
o! si
o
Yah
User session
- 33 -

34. Downstream engagement
70%
Varies significantly across sites 60%
50%
40%
30%
Exhibits different distributions 20%
10%
according to site type 0%
0.12
0.1
0.08
Is not highly correlated with other 0.06
0.04
engagement measures such as 0.02
dwell time 0
1%
9%
17%
25%
33%
41%
49%
58%
66%
74%
82%
90%
98%
140
120
100
Optimizing downstream engagement 80
60
will have little effect on user 40
20
engagement within that site 0
0% 20% 40% 60% 80%
- 34 -

35. What causes engagement to change?
Web page
style?
Web page
content?
- 35 -

36. Methodology
Front pages
of 50 popular
Y! properties Defining
crawled every downstream
hour engagement
measure Influencing
Studying the
Sample user network
effect of links
data from Measuring downstream
on front
toolbar data downstream engagement
pages on
engagement with front
(19.4M downstream
sessions, measure with page
engagement
265K users) web page stylistics
stylistics and
content
May 2011
- 36 -

37. Data
Page attributes were defined for 50 popular (by page views) Yahoo! sites:
Sampled the front page once every hour during the month of May 2011.
Generate two types of attributes for each site at each time and date:
Stylistics (layout and structure) of a page
General such as time of day, date, weekday or not
Downstream engagement values were measured using Yahoo! toolbar
data:
A total of 19.4M sessions recorded from approximately 265,000 users.
User and front page datasets joined by site, date and time, such that for
each site and each date and time combination we have:
average downstream engagement
average dwell time
vector of corresponding style attributes collected around the time that user
engagement was measured
- 37 -

38. Page stylistics provide good information to predict
downstream engagement for many Yahoo! sites
The top-10 sites for which downstream engagement (DE) could be “accurately” predicted
based on their stylistics
Accuracy Precision Average DE
site 1 0.80 0.54 0.07
site 2 0.76 0.52 0.11
site 3 0.72 0.43 0.21
site 4 0.71 0.40 0.18
site 5 0.65 0.42 0.20
site 6 0.63 0.34 0.19
site 7 0.63 0.38 0.26
site 8 0.63 0.44 0.14
site 9 0.61 0.34 0.18
site 10 0.60 0.31 0.13
Downstream engagement of a number of sites of not particular types
(models) could not be predicted from their stylistics.
- 38 -

39. Influential features
o Time of day
o Number of (non-image/non-video) links to Yahoo! sites in HTML body
o Average rank of Yahoo! links on page
o Number of (non-image/non-video) links to non-Yahoo! sites in HTML body
o Number of span tags (tags that allow adding style to content or manipulating
content, e.g. JavaScript)
o Link
placements
and
number
of
Yahoo!
links
can
influence
downstream
engagement
o Not
new,
but
here
shown
to
hold
also
across
sites
o Links
to
non-­‐Yahoo!
sites
have
a
posi>ve
effect
on
downstream
engagement
o Possibly
because
when
users
are
faced
with
abundance
of
outside
links
they
decide
to
focus
their
aBen>on
on
a
central
content
provider,
rather
than
visi>ng
mul>tude
of
external
sites
- 39 -

40. Three case studies
Here we look at three different Yahoo! sites, and the effect of
their stylistics for downstream engagement
Sites Average Accuracy Precision
downstream
engagement
e-commerce 0.26 (+/- 0.31) 0.63 0.38
news 0.15 (+/- 0.02) 0.65 0.37
women-interests 0.21 (+/- 0.06) 0.72 0.53
Number of unique Yahoo! links (-)
Time of day (+) Number of image links to non-Yahoo! Number of (non-image/non-video) links
Weekend (-) sites in the body of the HTML (-) to Yahoo sites in the body of the
Number of unique Yahoo! links (-) Number of table elements (-) HTML (+)
Average rank of Yahoo! links on page (-) Average rank of Yahoo! links on page (-) Number of (non-image/non-video) links
Number of paragraph tags (+) Number of (non-image/non-video) links to non-Yahoo! sites in the body of
to non-Yahoo! sites in the body of the HTML (+)
the HTML (+) Number of video links within the page (+)
Time of day (+) Number of Java scripts on the page (-)
- 40 -

41. Influencing engagement through links
The correlations between the number of various links and the values of
downstream engagement and dwell time
• e-commerce:
• links have little effect on e-commerce news women-
downstream engagement, but interests
have on dwell time Downstream engagement
• news:
• more news stories lead to more Same site 0.03 -0.31 -0.27
time spent on news Other Y! site -0.09 0.20 0.22
• external links do not affect Non Y! site -0.10 0.04 -0.25
downstream engagement, but Dwell time
affect dwell time Same site 0.51 0.78 0.82
• women-interests: Other Y! site -0.61 0.38 -0.68
• links to other Yahoo! sites can Non Y! site -0.51 0.04 0.80
help increase engagement, but
they may decrease dwell time
- 41 -

42. Users are more amenable to enhancing
downstream engagement during certain sessions
Goal-specific sessions are those sessions where users have a
specific goal in mind: do email, read news, check FB
Sessions when at least 50% of visited sites belonged to the five
most common sites (for that user) were classified as goal-
specific
Goal-specific sessions accounted for 38% of sessions.
Approximately 92% of users had sessions of both kinds.
Average downstream engagement in goal-specific sessions was
0.16 vs. 0.2 for other sessions.
Accuracy of predicting downstream engagement was 0.76 for
goal-specific sessions vs. 0.81 for other sessions.
When users do not have specific goals in mind, they may be
more ready to accept suggestions (e.g. more links) for
additional browsing
- 42 -

43. Further Work: Quantifying the Network Effect
Previously using one metric (downstream engagement), we
showed that there is a network effect, and that the network
effect can be influenced.
We go one step further and propose a methodology to account
for the traffic between sites when measuring user
engagement on a network of sites.
o Engagement networks
o Metrics from complex networks area (network-level and node-
level)
o Application on 728 Yahoo! sites
- 43 -

44. Thank you
Questions?
Thanks to many people at Yahoo! Labs
- 44 -