This is the presentation of my Ph.D. defense in February 2015.
The aim of my thesis is to provide a deeper understanding of how users engage with websites and how to measure this engagement. We start with studying online behaviour metrics, which are commonly employed as a proxy for user engagement, and we propose new metrics that expose so far unconsidered aspects of user engagement. We then conduct several case studies that demonstrate how these metrics provide a deeper understanding of user engagement. Within each case study we also examine how the characteristics of a website influence user engagement.
1. photo
credit
donsolo,
CC
BY-‐NC-‐SA
2.0
From
Site
to
Inter-‐site
User
Engagement
Jane;e
Lehmann
Barcelona,
February
26,
2015
Advisors:
Ricardo
Baeza-‐Yates
Co-‐Advisor:
Mounia
Lalmas
2. • User
engagement
is
a
quality
of
the
user
experience
that
emphasizes
the
posiLve
aspects
of
interacLon
with
a
website
–
in
parLcular
the
fact
of
being
capLvated
by
the
website.
• In-‐the-‐moment
engagement
Users
stay
on
a
website
over
a
long
Lme.
• Long-‐term
engagement
Users
come
back
frequently
and
over
a
long-‐term.
IntroducLon
2
User
Engagement
DefiniLon
Successful
websites
are
not
just
used,
they
are
engaged
with.
3. User
Engagement
Measuring
3
IntroducLon
Before
we
can
design
engaging
websites,
it
is
crucial
that
we
are
able
to
measure
engagement.
“If
you
can
measure
it,
you
can
improve
it.”
Sir
William
Thomson
Analysis/Planning
Design
Changes
Measuring
4. Main
Research
Goals
4
IntroducLon
Primary
goal
Can
we
define
new
engagement
metrics
that
Measuring
enhance
our
understanding
of
engagement?
Secondary
goal
Can
we
idenLfy
ways
to
influence
engagement?
Analysis/Planning
Analysis/Planning
Design
Changes
Measuring
5. IntroducLon
5
Analysis/
Planning
Design
Changes
Measuring
Online
mulLtasking
Inter-‐site
engagement
Site
engagement
Effect
of
providing
off-‐site
content
Effect
of
hyperlinks
6. IntroducLon
6
Analysis/
Planning
Design
Changes
Measuring
Site
engagement
7. Measuring
Engagement
InteracLon
data
7
Site
engagement
Data
Browsing
events
provided
by
Yahoo
toolbar
(client-‐side).
Engagement
Analysing
the
data
using
online
behaviour
metrics.
Online
session:
Visit
on
Yahoo
News
8. Site
engagement
8
Measuring
Engagement
Online
behaviour
metrics
K.
Rodden,
H.
Hutchinson,
X.
Fu.
Measuring
the
user
experience
on
a
large
scale:
User-‐centered
metrics
for
web
applicaHons.
CHI,
2010.
E.
Peterson,
J.
Carrabis.
Measuring
the
immeasurable:
Visitor
engagement.
Web
AnalyHcs
DemysHfied,
2008.
B.
Haven,
S.
ViWal.
Measuring
engagement.
Forrester
Research,
2008.
B.
Weischedel
and
E.
Huizingh.
Website
opHmizaHon
with
web
metrics:
A
case
study.
Conference
on
Electronic
commerce,
2006.
9. Site
engagement
9
Measuring
Engagement
Online
behaviour
metrics
Popularity
#Users
Number
of
users.
#Visits
Number
of
visits.
#Clicks
Number
of
clicks.
AcCvity
(within
a
visit)
In-‐the-‐moment
engagement
PageViews
Avg.
number
of
page
views
per
visit.
DwellTime
Avg.
Lme
on
site
per
visit.
Loyalty
(across
visits)
Long-‐term
engagement
ReturnRate
Number
of
Lmes
a
user
visited
the
site.
AcLveDays
Number
of
days
a
user
visited
the
site.
10. Site
engagement
10
Measuring
Engagement
Differences
in
engagement
ComScore,
Alexa,
GoogleAnalyHcs,…
Shopping
Users
do
not
come
frequently,
but
stay
long
Games
Not
many
users,
but
they
stay
long
News
Users
come
frequently
and
stay
long
11. Measuring
Engagement
Problem
11
Site
engagement
Isolated
view:
The
metrics
focus
on
engagement
with
a
single
site.
RelaLonships
to
other
sites
are
not
considered.
13. Online
mulLtasking
13
MoCvaCon
In-‐the-‐moment
engagement
ComScore,
Alexa,
GoogleAnalyHcs,…
What
web
analyCcs
think
we
do…
1
visit
with
4
page
views.
14. Online
mulLtasking
14
MoCvaCon
In-‐the-‐moment
engagement
ComScore,
Alexa,
GoogleAnalyHcs,…
…
and
what
we
really
do:
3
visit
with
on
average
1.3
page
views.
15. Online
mulLtasking
15
MoCvaCon
Online
mulLtasking.
Problem
• Engagement
metrics
do
not
capture
such
behaviour.
• Measuring
acLvity
on
a
site
can
lead
to
incorrect
conclusions.
Online
mulCtasking
Users
visit
several
sites
and
switch
between
them
during
an
online
session,
to
perform
related
or
totally
unrelated
tasks.
16. Research
QuesCon
16
Online
mulLtasking
How
can
we
measure
engagement
by
accounLng
for
user
mulLtasking
behaviour?
Analysis/Planning
Design
Changes
Measuring
17. Extent
of
mulCtasking
• 10.2
disLnct
sites,
2
visits
per
site.
Absence
Cme
• 50%
of
sites
are
revisited
aker
<
1min.
InterrupHon
of
a
task
• There
are
revisits
aker
long
breaks.
Performing
a
new
task
Online
mulLtasking
17
Online
MulCtasking
CharacterisLcs
0.00
0.25
0.50
0.75
1.00
10
2
10
1
10
0
10
1
10
2
Cumulativeprobability Absence time [min]
news (finance)
news (tech)
social media
mail
2.09
1.76
2.28
2.09
#Visits Absence
time [min]
3.85
3.95
4.47
6.86
Absence time: Time between two visits
18. AcCvity
paPerns
• Four
types:
Decreasing,
increasing,
constant,
complex.
• Successive
visits
can
belong
together
(i.e.
to
the
same
task).
• Complex
cases
refer
to
no
specific
pa;ern
or
repeated
pa;ern.
Online
mulLtasking
18
Online
MulCtasking
CharacterisLcs
1 2 3 4
ith
visit on site
1 2 3 4
ith
visit on site
1 2 3 4
ith
visit on site
1 2 3 4
ith
visit on site
Proportionoftotal
dwelltimeonsite
0.23
0.28
0.33 p-value = 0.09
m = -0.01
p-value = 0.07
m = -0.02
p-value = 0.79
m = 0.00
news (finance) sitesmail sites social media sites news (tech) sites
decreasing attention increasing attention constant attention complex attention
19. Online
mulLtasking
19
Measuring
Engagement
Online
mulLtasking
metrics
Extent
of
mulCtasking
SessSites
Total
number
of
sites
accessed
(#tasks).
SessVisits
Number
of
visits
to
site
(site
switching).
Absence
Cme
CumAct
Aggregates
the
dwell
Lmes
of
the
visits
with
accounLng
for
the
Lme
between
the
visits.
AcCvity
paPern
A;Shik
A;Range
Describe
the
four
cases
of
a;enLon
shiks.
20. 20
CASE
STUDY:
MulCtasking
PaPerns
• ObjecCve:
Analyse
mulLtasking
acLvity
on
sites;
idenLfy
mulLtasking
pa;erns
(clustering).
• Metrics:
Site
DwellTime,
MulLtasking
metrics.
• Data:
July
2012,
2.5M
users,
760
sites
(shopping,
news,
search,
etc.).
21. 21
Case
Study:
MulCtasking
PaPerns
Results
No
mulCtasking
MulCtasking
Quick
Focused
Rapid
ConCnuous
Recurring
Checking
weather
Reading
mails
Following
link
to
off-‐site
content
Purchasing
an
item
Performing
search
Site
DwellTime
-‐-‐
++
++
++
-‐-‐
Extent
of
mulCtasking
-‐-‐
-‐-‐
++
++
++
Absence
Cme
-‐-‐
++
++
ImplicaCons
Provide
interesHng
off-‐
site
content
Shopping
takes
more
than
one
visit
Support
user
by
finishing
tasks
quickly
Online
mulLtasking
-- low value ++ high value
22. 22
Case
Study:
MulCtasking
PaPerns
Results
No
mulCtasking
MulCtasking
Quick
Focused
Rapid
ConCnuous
Recurring
Checking
weather
Reading
mails
Following
link
to
off-‐site
content
Purchasing
an
item
Performing
search
Site
DwellTime
-‐-‐
++
++
++
-‐-‐
Extent
of
mulCtasking
-‐-‐
-‐-‐
++
++
++
Absence
Cme
-‐-‐
++
++
AcCvity
paPern
Online
mulLtasking
De In CmCn
60%
0%
De In CmCn
60%
0%
De In CmCn
60%
0%
Activity pattern: De – Decreasing In – Increasing Cn – Constant Cm - Complex
-- low value ++ high value
23. 23
CASE
STUDY:
Wikipedia
(on-‐site
mulCtasking)
• ObjecCve:
Analyse
reading
acLvity
on
Wikipedia
arLcles;
idenLfy
reading
pa;erns
(clustering).
• Metrics:
ArLcle
DwellTime,
#ArLcles
in
session,
#Views
to
focal
arLcle.
• Data:
Sep
2011
–
Sep
2012,
500K
users,
10K
biography
arLcles.
24. 24
Case
Study:
Wikipedia
Approach
Online
mulLtasking
Users’
reading
behaviour
on
an
Wikipedia
arCcle
ArLcle
DwellTime
How
much
Lme
do
users
spend
on
an
arLcle?
#ArLcles
in
session
Do
users
view
also
other
arLcles
during
an
online
session?
#Views
on
focal
arLcle
How
oken
do
users
view
the
arLcle?
25. 25
Case
Study:
Wikipedia
Results
No
mulCtasking
MulCtasking
Focus
ExploraCon
Passing
Focus
is
on
focal
arHcle
Exploring
topic
around
the
focal
arHcle
Exploring
topic
and
pass
through
the
focal
arHcle
ArCcle
DwellTime
++
-‐-‐
#ArCcles
in
session
-‐-‐
++
++
#Views
to
focal
arCcle
++
-‐-‐
ImplicaCons
Content
quality
is
important
Links
to
addiHonal
content
are
important
ArHcles
might
need
to
be
extended
Online
mulLtasking
26.
On-‐site
mulCtasking
• MulLtasking
between
news
arLcles
of
a
provider.
• MulLtasking
between
different
tasks
on
a
social
media
site
(e.g.
sharing,
chapng,
updaLng
profile).
• …
Inter-‐site
mulCtasking
• MulLtasking
when
purchasing
items
online
(comparing
offers,
product
reviews,
search,
etc.)
• …
Online
mulLtasking
26
Further
Use
Cases
27. Take
Aways
• AccounLng
for
mulLtasking
leads
to
a
be;er
understanding
on
how
users
engage
with
sites.
• Leaving
a
site
does
not
necessarily
entail
less
engagement,
as
users
oken
return
to
the
site
later
on.
Publications
J. Lehmann, M. Lalmas, G.
Dupret, and R. Baeza-Yates.
Online multitasking and user
engagement. CIKM 2013.
J. Lehmann, C. Müller-Birn, D.
Laniado, M. Lalmas, and A.
Kaltenbrunner. Reader
preferences and behavior on
Wikipedia. HT 2014, Ted
Nelson Newcomer Paper
Award.
J. Lehmann, C. Müller-Birn, D.
Laniado, M. Lalmas, and A.
Kaltenbrunner. What and
how users read: Transforming
reading behavior into
valuable feedback for the
Wikipedia community.
Wikimania 2014.
Online
mulLtasking
27
30. Inter-‐site
engagement
30
MoCvaCon
Large
online
service
providers
frontpage
tv
sports
shopping
autos
search
daLng
jobs
news
shine
groups
maps
local
health
answer
weather
games
mail
omg
homes
travel
flickr
finance
Large
online
service
providers
(AOL,
Google,
Yahoo,
etc.)
have
not
only
one
site,
but
many
sites.
tumblr
31. Inter-‐site
engagement
31
MoCvaCon
Large
online
service
providers
frontpage
tv
sports
shopping
autos
search
daLng
jobs
news
shine
groups
maps
local
health
answer
weather
games
mail
omg
homes
travel
flickr
finance
Providers
want
that
users
engage
with
many
of
their
sites.
tumblr
32. Inter-‐site
engagement
32
MoCvaCon
Online
mulLtasking
Problem
• Engagement
metrics
do
not
measure
engagement
across
sites.
• How
to
adapt
them
is
not
obvious.
Inter-‐site
engagement
Users
visit
sites
that
belong
to
the
same
network
of
sites.
33. Research
QuesCon
33
Inter-‐site
engagement
How
can
we
measure
engagement
by
also
considering
the
relaLonships
between
sites?
Analysis/Planning
Design
Changes
Measuring
34. Inter-‐site
engagement
34
Traffic
Networks
Modelling
We
model
sites
(nodes)
and
user
traffic
(edges)
between
them
as
a
network.
Provider
network
G=(N,
E,
λ)
N:
Sites
E:
User
traffic
λ(e):
Traffic
volume
(#Clicks)
4
clicks
2
clicks
50
clicks
10
clicks
35. Inter-‐site
engagement
35
Measuring
Engagement
Inter-‐site
engagement
metrics:
Network-‐level
Traffic
distribuCon
Flow
Extent
to
which
users
navigate
between
sites.
Density1
Diversity
of
inter-‐site
engagement.
Reciprocity2
Homogeneity
of
traffic
between
sites.
External
traffic
EntryDisparity
Variability
of
in-‐going
traffic
to
the
network.
ExitDisparity
Variability
of
out-‐going
traffic
from
the
network.
[1]
S.
Wasserman.
Social
network
analysis:
Methods
and
applicaHons,
1994.
[2]
T.
SquarHni,
F.
Picciolo,
F.
RuzzenenH,
and
D.
Garlaschelli.
Reciprocity
of
weighted
networks.
Nature:
ScienHfic
reports,
2013.
36. Inter-‐site
engagement
36
Measuring
Engagement
Inter-‐site
engagement
metrics:
Node-‐level
Traffic
distribuCon
PageRank1
Probability
that
a
user
will
visit
the
site.
Downstream
Probability
that
a
user
will
conLnue
browsing
to
other
sites.
External
traffic
EntryProb
Probability
that
a
user
enters
the
network
in
this
site.
ExitProb
Probability
that
a
user
leaves
the
network
in
this
site.
[1]
L.
Page,
S.
Brin,
R.
Motwani,
T.
Winograd.
The
pagerank
citaHon
ranking:
Bringing
order
to
the
web.
Technical
report,
Stanford
InfoLab,
1999.
37. 37
CASE
STUDY:
Yahoo
Provider
Networks
• ObjecCve:
Compare
networks;
characterise
the
sites
in
a
network.
• Metrics:
Network
DwellTime,
Site
DwellTime,
Inter-‐
site
engagement
metrics.
• Data:
February
2014,
3.2M
clicks/network,
4
country-‐based
networks,
31
sites
per
network.
38. 38
Case
Study:
Yahoo
Comparing
provider
networks
Network
1
Network
2
Network
3
Network
4
High
engaging
Users
engage
quickly
with
many
sites
Users
engage
to
a
subset
of
sites
Low
engaging
Network
DwellTime
++
-‐-‐
++
-‐-‐
Traffic
DistribuCon
++
++
Flow
++
Density
-‐-‐
-‐-‐
Entry
Disparity
++
-‐-‐
++
ImplicaCons
The
network
is
performing
well.
This
should
be
looked
into.
MoHvate
users
to
visit
other
sites.
This
should
be
looked
into.
Inter-‐site
engagement
-- low value ++ high value
39. 39
Case
Study:
Yahoo
Sites
within
a
provider
network
Traffic
Hub
Supporter
Focused
Engagement
Shared
Engagement
Search,
front
pages
Support,
services
Leisure,
support
News,
leisure
Site
DwellTime
-‐-‐
-‐-‐
++
++
Traffic
DistribuCon
++
-‐-‐
-‐-‐
++
Entry
Probability
++
-‐-‐
++
-‐-‐
ImplicaCons
The
sites
forward
traffic
to
other
sites.
Users
visit
sites
for
specific
needs
and
support.
MoHvate
users
to
visit
other
sites.
The
sites
are
performing
well.
Inter-‐site
engagement
-- low value ++ high value
40.
Comparing
networks
• Device,
Lme,
upstream
traffic,
user.
• SimulaLons
(effect
of
adding/removing
sites).
• …
Network
types
• Network
of
pages
(e.g.
compare
language-‐based
Wikipedia
networks)
• Network
of
sites
from
different
providers
(e.g.
shopping
sites,
news
providers)
• …
Inter-‐site
engagement
40
Further
Use
Cases
41. Take
Aways
• Inter-‐site
engagement
allows
for
a
more
comprehensive
look
at
user
engagement
by
also
considering
the
relaLonships
between
sites.
• Deeply
engaged
users
do
not
only
engage
with
one
site,
but
with
many
sites
in
a
network.
Publications
J. Lehmann, M. Lalmas, and R. Baeza-
Yates. Measuring Inter-Site Engagement.
Handbook of Statistics, Elsevier, 2015. To
appear.
J. Lehmann, M. Lalmas, R. Baeza-Yates,
and E. Yom-Tov. Networked User
Engagement. ACM Workshop on User
engagement optimization at CIKM, 2013.
J. Lehmann, M. Lalmas, and R. Baeza-
Yates. Temporal Variations in Networked
User Engagement. TNETS Satellite at ECCS,
2013.
Some of the metrics were employed to
characterise online news reading across
news sites:
J. Lehmann, C. Castillo, M. Lalmas, and R.
Baeza-Yates. Story-Focused Reading in
Online News. Submitted for publication.
Inter-‐site
engagement
41
42. IntroducLon
42
Analysis/
Planning
Design
Changes
Measuring
Online
mulLtasking
Inter-‐site
engagement
Site
engagement
Effect
of
providing
off-‐site
content
43. 43
CASE
STUDY:
Online
News
• Hypothesis:
It
may
be
beneficial
(long-‐term)
to
enLce
users
to
leave
a
site
by
offering
interesLng
off-‐site
content.
• Data:
October
2013,
57K
users,
50
news
sites,
26K
news
arLcles.
44. Types
of
reading
sessions
No
click
Did
not
follow
a
related
link.
Off-‐site
click
Followed
a
related
link
to
content
on
another
site.
Effect
on
engagement
Short-‐term
Dwell
Lme
per
reading
session.
Long-‐term
Probability
that
user
starts
next
reading
session
within
the
next
12h.
44
Case
Study:
Online
News
Related
off-‐site
content
Approach
Effect
of
providing
off-‐site
content
45. Providing
links
to
related
off-‐site
content
has
a
no
short-‐term
effect,
but
a
posiCve
long-‐term
effect.
45
Case
Study:
Online
News
Results
Effect
of
providing
off-‐site
content
News provider
Dwelltimepersession
News provider
p(absence12h)
No Click Off-site click
46. IntroducLon
46
Analysis/
Planning
Design
Changes
Measuring
Online
mulLtasking
Inter-‐site
engagement
Site
engagement
Effect
of
providing
off-‐site
content
Effect
of
hyperlinks
47. 47
CASE
STUDY:
Yahoo
Provider
Network
• Hypothesis:
We
can
use
hyperlinks
to
influence
inter-‐site
engagement
in
a
provider
network.
• Data:
February
2014,
235M
clicks,
Yahoo
US
network,
73
sites.
48.
Hyperlink
vs.
traffic
network
On-‐site
Links/Traffic
to
pages
within
the
same
site.
Inter-‐site
Links/Traffic
to
pages
to
other
sites
in
the
network.
External
Links/Traffic
to
somewhere
else
on
the
Web.
48
Case
Study:
Yahoo
Approach
frontpage
sports
search
news
shine
groups
answer
weather
mail
omg
homes
flickr
Effect
of
hyperlinks
49. Hyperlinks
can
be
used
to
influence
site
and
inter-‐site
engagement
in
a
provider
network.
However,
both
types
of
engagement
influence
each
other.
49
Case
Study:
Yahoo
Results
Effect
of
hyperlinks
Traffic
On-site Inter-site External
Hyperlinks
On-site
Inter-site
External
0.54
-0.40
-
-0.45
0.50
-
-0.38
-
0.39
50. IntroducLon
50
Analysis/
Planning
Design
Changes
Measuring
Online
mulLtasking
Inter-‐site
engagement
Site
engagement
Effect
of
providing
off-‐site
content
Effect
of
hyperlinks
51. Two
new
perspecHves
for
measuring
engagement
which
consider
the
relaLonships
between
sites.
Online
mulCtasking
Accounts
for
user
mulLtasking
behaviour.
Inter-‐site
engagement
Accounts
for
the
traffic
between
sites.
ContribuLons
and
future
work
51
Main
ContribuCons
Measuring
engagement
Analysis/
Planning
Design
Changes
Measuring
52. AccounLng
for
the
new
perspecLves
when
influencing
engagement.
Online
news
Providing
related
off-‐site
content
influences
long-‐term
engagement.
Provider
network
Hyperlinks
affect
site
and
inter-‐site
engagement,
but
both
influence
each
other.
ContribuLons
and
future
work
52
Main
ContribuCons
Analysis/Planning
Analysis/
Planning
Design
Changes
Measuring
53. Wikipedia
Providing
informaLon
about
readers’
engagement
to
the
editor
community.
Yahoo
Using
inter-‐site
engagement
metrics
to
make
informed
decisions
about
design
changes
(hyperlinks).
Spiegel
Online
Measuring
and
improving
engagement
by
providing
interesLng
off-‐site
content.
ContribuLons
and
future
work
53
What
next?
Ongoing
and
future
work
Analysis/
Planning
Design
Changes
Measuring
54. photo
credit
donsolo,
CC
BY-‐NC-‐SA
2.0
Thank
you!
Jane;e
Lehmann
Barcelona,
February
26,
2015
lehmannj@acm.org
Acknowledgements
Ricardo
Baeza-‐Yates
Mounia
Lalmas
Claudia
Müller-‐Birn
Carlos
CasLllo
David
Laniado
Andreas
Kaltenbrunner
Elad
Yom-‐Tov
Georges
Dupret
Guy
Shaked
Fabrizio
Silvestri
Gabriele
Tolomei
Ethan
Zuckerman
John
Agapiou
Andy
Haines
Diego
Sáez-‐Trumper
Hemant
Purohit
Noora
Al
Emadi
Mohammed
El-‐Haddad
Nasir
Khan
55. • Mounia Lalmas and Janette Lehmann. “Models of User Engagement”. In H. L. O’Brien and M. Lalmas (Eds.), Why Engagement
Matters: Cross-disciplinary Perspectives and Innovations on User Engagement with Digital Media. Springer, 2015, in progress.
• Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. “Models of user engagement.” International
Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), pp. 164-175, Montreal, Canada, July, 2012.
• Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. “Online multitasking and user engagement.”
ACM International Conference on Information and Knowledge Management (CIKM 2013), pp. 519-528, San Francisco, United
States, October, 2013.
• Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Measuring Inter-Site Engagement.”. In V. Govindaraju, V. V.
Raghavan, and C. R. Rao (Eds.), Handbook of Statistics, Elsevier, 2015.
• Janette Lehmann, Mounia Lalmas, Ricardo Baeza-Yates, and Elad Yom-Tov. “Networked User Engagement.”, ACM Workshop
on User engagement optimization at CIKM, pp. 7-10, San Francisco, United States, October, 2013.
• Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Temporal Variations in Networked User Engagement.”, TNETS
Satellite at European Conference on Complex Systems (ECCS), Barcelona, Spain, September, 2013.
• Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. “Measuring Post-click User Experience
with Mobile Native Advertising on Streams.”, submitted for publication.
• Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “Reader preferences and
behavior on Wikipedia.”, ACM International Conference on Hypertext and Social Media (HT 2014), pp. 88-97, Santiago, Chile,
September, 2014, Ted Nelson Newcomer Paper Award.
• Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “What and how users
read: Transforming reading behavior into valuable feedback for the Wikipedia community.”, Presentation at Wikimania,
London, UK, August, 2014.
• Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. “Story-Focused Reading in Online News.”,
submitted for publication.
• Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Transient News Crowds in Social Media.”
International AAAI Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA, July, 2013.
• Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Finding News Curators in Twitter.” ACM
International Conference on World Wide Web Companion (WWW 2013 Companion), 863-870, Rio de Janeiro, Brazil, May,
2013.
55
PublicaCons
56. User engagement
• Mounia Lalmas, Heather L O’Brien, and Elad Yom-Tov. Measuring user engagement. Synthesis Lectures on Sample Series #1.
Morgan and cLaypool publishers, 2014.
• Heather L O’Brien and Elaine G Toms. What is user engagement? a conceptual framework for defining user engagement
with technology. American Society for Information Science and Technology (ASIS&T), 59(6):938–955, 2008.
• Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. Towards a science of user engagement (position
paper). In Proc. Workshop on User Modelling for Web Applications, WSDM, 2011.
• Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web
applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.
Online behaviour metrics
• Brian Haven and Suresh Vittal. Measuring engagement. Forrester Research, 2008.
• Eric T Peterson and Joseph Carrabis. Measuring the immeasurable: Visitor engagement. Web Analytics Demystified, 2008.
• Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web
applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.
• Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. In Proc.
Conference on Web Search and Data Mining, WSDM, pages 173–182. ACM, 2013.
• Randolph E Bucklin and Catarina Sismeiro. A model of web site browsing behavior estimated on clickstream data. Journal of
Marketing Research, 40(3):249–267, 2003.
• Birgit Weischedel and Eelko KRE Huizingh. Website optimization with web metrics: a case study. In Proc. Conference on
Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to
conducting successful business on the internet, pages 463–470. ACM, 2006.
• Peifeng Yin, Ping Luo, Wang-Chien Lee, and Min Wang. Silence is also evidence: interpreting dwell time for recommendation
from psychological perspective. In Proc. Conference on Knowledge Discovery and Data Mining, SIGKDD, pages 989–997.
ACM, 2013.
56
Selected
References
57. Online multitasking
• Qing Wang and Huiyou Chang. Multitasking bar: prototype and evaluation of introducing the task concept into a browser. In
Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 103–112. ACM, 2010.
• Hartmut Obendorf, Harald Weinreich, Eelco Herder, and Matthias Mayer. Web page revisitation revisited: implications of a
long-term click-stream study of browser usage. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages
597–606. ACM, 2007.
• Jeff Huang and Ryen W White. Parallel browsing behavior on the web. In Proc. Conference on Hypertext and Hypermedia,
HT, pages 13–18. ACM, 2010.
• Patrick Dubroy and Ravin Balakrishnan. A study of tabbed browsing among mozilla firefox users. In Proc. Conference on
Human Factors in Computing Systems, SIGCHI, pages 673–682. ACM, 2010.
Inter-site engagement
• Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167–256, 2003. 76, 77, 165
• Anna Chmiel, Kamila Kowalska, and Janusz A Hołyst. Scaling of human behavior during portal browsing.
• Mark R Meiss, Filippo Menczer, Santo Fortunato, Alessandro Flammini, and Alessandro Vespignani. Ranking web sites with real
user traffic. In Proc. Conference on Web Search and Data Mining, WSDM, pages 65–76. ACM, 2008.
• Young-Hoon Park and Peter S Fader. Modeling browsing behavior at multiple websites. Marketing Science, 23(3):280–303,
2004.
• Qiqi Jiang, Chuan-Hoo Tan, and Kwok-Kee Wei. Cross-website navigation behavior and purchase commitment: A pluralistic
field research. In Proc. Pacific Asia Conference on Information Systems, PACIS, 2012.
• Kevin Koidl, Owen Conlan, and Vincent Wade. Cross-site personalization: assisting users in addressing information needs that
span independently hosted websites. In Proc. Conference on Hypertext and Hypermedia, HT, pages 66–76. ACM, 2014.
• The PEW Research Center. Understanding the participatory news consumer. http://www.pewinternet.org/~/media/Files/
Reports/ 2010/PIP_Understanding_the_Participatory_News_Consumer. pdf, 2010.
• Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.
57
Selected
References
58. Link economy
• Joseph Turow and Lokman Tsui. The hyperlinked society. The University of Michigan Press, 2008.
• Juliette De Maeyer. Hyperlinks and journalism: where do they connect? In Proc. Future of Journalism Conference, 2011.
• Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink
formation in content networks. Management Science, 59(10):2360–2379, 2013.
• Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.
• Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink
formation in content networks. Management Science, 59(10):2360–2379, 2013.
• Hakan Ceylan, Ioannis Arapakis, Pinar Donmez, and Mounia Lalmas. Automatically embedding newsworthy links to articles. In
Proc. Conference on Information and Knowledge Management, CIKM, pages 1502–1506. ACM, 2012.
Recommendation
• Richard McCreadie, Craig Macdonald, and Iadh Ounis. News vertical search: when and what to display to users. In Proc.
Conference on Research and Development in Information Retrieval, SIGIR, pages 253–262. ACM, 2013.
• Samuel Ieong, Mohammad Mahdian, and Sergei Vassilvitskii. Advertising in a stream. In Proc. Conference on World Wide
Web, WWW, pages 29–38. ACM, 2014.
• Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. A predictive model for advertiser value-per-click in sponsored search. In
Proc. Conference on Information and Knowledge Management, CIKM, pages 1179–1190. ACM, 2013.
• Narongsak Thongpapanl and Abdul Rehman Ashraf. Enhancing online performance through website content and
personalization. Journal of Computer Information Systems, 52(1):3, 2011.
• Jian Wang and Yi Zhang. Utilizing marginal net utility for recommendation in e-commerce. In Proc. Conference on Research
and Development in Information Retrieval, SIGIR, pages 1003–1012. ACM, 2011.
• Joshua Porter. Designing for the social web. Peachpit Press, 2010.
58
Selected
References
60. • “In
a
world
full
of
choices
where
the
fleeCng
aPenCon
of
the
user
becomes
a
prime
resource,
it
is
essenLal
that
[...]
providers
do
not
just
design
[websites]
but
that
they
design
engaging
experiences.”
[A}ield].
• In
addiLon
to
uLlitarian
factors,
such
as
usability
and
usefulness,
we
must
consider
other
factors
of
interacLng
with
websites,
such
as
fun,
fulfillment,
play,
and
user
engagement.
Successful
websites
are
not
just
used,
they
are
engaged
with.
• In
order
to
design
engaging
websites,
it
is
crucial
to
understand
what
user
engagement
is
and
how
to
measure
it.
IntroducLon
60
MoCvaCon
Why
is
it
important
to
engage
users?
61. Methodology
InteracLon
data,
online
sessions
and
site
visits.
61
IntroducLon
t0
t1
t2
t3
t4
t5
t6
t7
session
end
session
start
time
Online session
Browsing activity on Wikipedia
https://ie-mg42.mail.yahoo.com
http://en.wikipedia.org/wiki/Freddie˙Mercury
http://www.bbc.com/news/uk-29149115
http://www.bbc.com/news/uk-england-nottinghamshire-29643802
http://en.wikipedia.org/wiki/Star Wars
http://en.wikipedia.org/wiki/Yoda
http://en.wikipedia.org/wiki/Albert˙Einstein
https://www.facebook.com/janette.lehmann.5
t0
t1
t2
t3
t4
t5
t6
t7
bc0
bc0
bc0
bc0
bc0
bc0
bc0
bc0
BCookie Timestamp URL
-
-
-
http://www.bbc.com/news/uk-29149115
-
http://en.wikipedia.org/wiki/Star Wars
http://en.wikipedia.org/wiki/Yoda
-
ReferrerURL
Interaction data
Page view on Wikipedia Page view on other site
62. IntroducLon
62
Thesis
structure
Metrics that account for site
popularity, activity and loyalty
Advertising
Chapter 7
Site engagement
How users experience
ads on desktop and
mobile devices?
Does ad quality
affect the engagement
with the publisher?
How can we identify
high quality ads?
Site engagement
Chapter 4
Multitasking
Chapter 5
Inter-site engagement
Chapter 6
Metrics that account for
traffic between sites
Metrics that account for user
multitasking behaviour
(III+IV)Applications(II)Fund.
Wikipedia
Chapter 8
Site engagement
and multitasking
How users read
articles
in Wikipedia?
Does the activity
of editors align with the
engagement of readers?
How can readers be
valuable for editors?
Yahoo
Chapter 9
Inter-site engagement
How users engage
with a provider
network of sites?
Does the hyperlink
structure affect site and
inter-site engagement?
Online news
Chapter 10+11
Inter-site engagement
How users read
stories across
news providers?
Do hyperlinks to
related content influence
provider engagement?
How can we automatically
detect related content?
Characterising user engagement Comparing site characteristics and user engagement Applications to impact user engagement
65. Site
engagement
65
PaPerns
of
Site
Engagement
Engagement
depends
on
the
site
at
hand.
Games
Not
many
users,
but
they
stay
long
Search
Users
come
frequently,
but
do
not
stay
long
Social
media
Users
come
frequently
and
stay
long
Shopping
Users
do
not
come
frequently,
but
stay
long
News
Users
come
frequently
and
stay
long
Service
Users
do
not
come
frequently,
but
stay
long
67. Online
mulLtasking
67
MoCvaCon
Users
switch
between
sites,
to
perform
related
or
totally
unrelated
tasks.
Switching
between
tasks
(sites)
“…within-‐session
page
revisits
represent
the
most
common
form
of
revisitaLon,
covering
73,54%
of
all
revisits.”
[Herder]
Performing
tasks
(sites)
in
parallel
using
browser
tabs
“Most
of
our
parLcipants
switched
tabs
more
oken
than
they
used
the
back
bu;on.”
[Dubroy]
[Herder]
E.
Herder.
CharacterizaHons
of
user
web
revisit
behavior.
WWW
Workshop
ABIS,
2005.
[Dubroy]
P.
Dubroy,
R.
Balakrishnan.
A
study
of
tabbed
browsing
among
mozilla
firefox
users.
SIGCHI,
2010.
68. Online
mulLtasking
68
Data
Dataset
and
site
categories.
Cat. Subcat. %Sites Description
news
22.1%
news 5.79%
news (soc.) 5.13% society
news (sport) 2.63%
news (enter.) 2.24% music, movies, tv, etc.
news 1.97%
news (life) 1.58% health, housing, etc.
news (tech) 1.58% technology
news (weather) 1.18%
service
15.5%
service 7.63% translators, banks, etc.
mail 3.95%
maps 3.03%
organisation 0.92% bookmarks, calendar, etc.
search
15.3%
search 12.63%
search (special) 1.58% search for lyrics, jobs, etc.
directory 1.05%
sharing
9.6%
blogging 3.55%
knowledge 3.55% collaborative creation and collection of content
sharing 2.50% sharing of videos, etc.
navi
9.3%
front page 6.58%
front page (p.) 1.84% personalised front pages
sitemap 0.92%
leisure
8.7%
adult 2.76%
games 1.97%
social media 1.97%
dating 1.05%
entertainment 0.92% sites with music, tv, etc.support
8.7%
support 1.58% sites that provide products and support for them
download 7.11% downloading software
shopping
7.9%
shopping 4.34%
auctions 2.11%
comparison 1.45% sites to compare prices of products
settings
2.9%
login 1.71%
site settings 1.18% pr e setting, site personalisation
InteracCon
data
• July
2012
• 2.5M
users
• 785M
page
views
NavigaCon
model
• We
defined
a
new
navigaLon
model
(see
paper
for
details)
Site
categories
• 760
sites
from
70
countries/
regions
• 11
categories
• 33
subcategories
69. Online
mulLtasking
69
MulCtasking
Metrics
CumAct
accounts
for
the
acLvity
between
site
visits.
CumulaCve
acCvity
The
metric
is
defined
as
follows:
InterpretaCon
High
CumAct
à
High
engagement
If
users
return
aker
short
Lme,
they
return
to
conLnue
with
same
task.
If
users
return
aker
longer
Lme,
they
return
to
perform
a
new
task
–
a
sign
of
loyalty.
CumActk = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
Browsing
acLvity
during
the
ith
visit
Browsing
acLvity
between
the
(i-‐1)th
and
ith
visit
Rescaling
factor
for
ivi
k = 3
vi
ivi
1
4
3
10
3
CumAct
= log10 (3+13
•4+103
•3)
= 3.48
Site
visit
70. Online
mulLtasking
70
MulCtasking
Metrics
AWRange
and
AWShik
describe
changes
between
the
visits.
APenCon
shie
and
range
The
metrics
is
defined
as
follows:
InterpretaCon
AWShik
models
the
shik
of
a;enLon,
and
AWRange
models
the
fluctuaLons
in
the
browsing
acLvity.
AttShiftn =
invn − minInvn
| maxInvn |− | minInvn |
AttRangen =
σ (Vn )
µ(Vn )
Variance
in
the
visit
acLvity
Average
of
the
visit
acLvity
Number
of
visits
in
session
ModificaLon
of
the
“Inversion
number”
n = 4
σ
µi
Inv
0
>0
-‐1
constant
decreasing
0
constant
complex
+1
constant
increasing
AWenHon
range
AWenHon
shik
71. 0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.05
('-' insignificant correlations)
Online
mulLtasking
71
EvaluaCon
CorrelaLons
between
mulLtasking
and
acLvity
metrics.
[MT]SessVisits
[MT]SessSites
[MT]CumAct
[MT]AttShift
[MT]AttRange
[ACT]DwellTimeS
SessSites [MT] 0.42
CumAct [MT] 0.41 -
AttShift [MT] 0.09 - -
AttRange [MT] - - -0.38 0.27
DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08
DwellTimeV [ACT] -0.40 - - 0.14 - 0.50
No
or
only
weak
correlaCons
between
the
metrics.
All
metrics
convey
different
aspects
about
users’
online
behaviour.
72. Online
mulLtasking
72
MulCtasking
PaPerns
Cluster
centers,
site
categories
and
acLvity
pa;erns.
CategoriesMultitasking
DwellTimeV CumAct SessVisitsDwellTimeS
sitemap
site settings
news (wheather)
download
+75%
+73%
+69%
+67%
PD
139 sites
Quick task Continuous
multitasking
SessSitesBars from left to right:
111 sites
auctions
shopping
adult
dating
+79%
+71%
+71%
+62%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
PD - Probability difference
Activity
Activity pattern: De - Decreasing In - Increasing Cn - Constant Cm - Complex
De In CmCn De In CmCn
60%
0%
147 sites
Recurring task
search
front page (p.)
front page
organisation
+77%
+62%
+57%
+24%
PD
-1.0
1.0
0.0
De In CmCn
0.6
0.0
137 sites
Focused task
news (tech)
news (life)
support
mail
+66%
+66%
+65%
+64%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
De In CmCn
60%
0%
60%
0%
142 sites
Rapid
multitasking
news (enter.)
knowledge
comparison
service
+64%
+63%
+62%
+59%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
De In CmCn
60%
0%
Single-task-oriented browsing Multitask-oriented browsing
74. Inter-‐site
engagement
74
Data
Dataset,
network
and
site
categories.
InteracCon
data
• August
2013
to
July
2014
• 53M
sessions
Provider
network
G=(N,
E,
λ)
N:
155
Yahoo
sites
from
five
countries
E:
User
traffic
λ(e):
Traffic
volume
(#Clicks)
Site
categories
• 155
sites
from
5
countries
• 5
categories
Cat. %Sites Description
35%
19%
13%
23%
10%
news
service
leisure
provider
front page
mail, calendar, etc.
social media, games, etc.
account settings, help, etc.
front pages, site maps
servicefront page news providerleisure
75. Inter-‐site
engagement
75
Inter-‐site
Engagement
Metrics
Flow
accounts
for
the
extent
users
navigate
between
sites.
Traffic
Flow
The
metric
is
defined
as
follows:
InterpretaCon
High
Flow
à
High
inter-‐site
engagement
Users
navigate
oken
between
the
sites
of
the
network.
Flow =
wi, ji, j∑
vii∑
#Clicks
between
node
i
and
j
#Visits
on
node
i
wi, j
vi
Flow = 30/60 = 0.5
10
5
20
20
20
10 5
1
1
20
20
20
1 1
Flow = 4/60 = 0.07
76. Inter-‐site
engagement
76
Inter-‐site
Engagement
Metrics
Density
describes
the
connecLvity
of
the
network.
Density
We
use
the
density
measure
of
[Wasserman]:
InterpretaCon
High
Density
à
High
inter-‐site
engagement
Users
navigate
between
many
different
sites
(inter-‐site
engagement
is
highly
diverse).
[Wasserman]
S.
Wasserman.
Social
network
analysis:
Methods
and
applicaHons,
1994.
Density =
# Edges
# Possible_ Edges
Density = 4/6 = 0.7
Flow = 2/6 = 0.3
77. Inter-‐site
engagement
77
Inter-‐site
Engagement
Metrics
Reciprocity
measures
the
homogeneity
of
traffic
between
two
sites.
Reciprocity
We
use
the
reciprocity
measure
of
[SquarLni]:
InterpretaCon
High
Reciprocity
à
High
inter-‐site
engagement
Users
navigate
between
two
sites
in
both
direcLons
(inter-‐site
engagement
is
highly
homogenious).
[SquarHni]
T.
SquarHni,
F.
Picciolo,
F.
RuzzenenH,
and
D.
Garlaschelli.
Reciprocity
of
weighted
networks.
Nature:
ScienHfic
reports,
2013.
#Clicks
between
node
i
and
j
wi, j
RP =
min[wi, j,wj,i ]
i<j∑
wi, ji≠j∑
1
10 5
20
1
Reciprocity = 15/50 = 0.3
Reciprocity = 2/37 = 0.05
10
10 5
20
5
78. Inter-‐site
engagement
78
Inter-‐site
Engagement
Metrics
Entry/ExitDisp
measures
how
the
traffic
to/from
the
network
is
distributed
over
the
sites.
Entry
disparity
and
exit
disparity
We
use
the
group
degree
measure
of
[Freeman]
and
adapt
it
as
follows:
InterpretaCon
High
Entry/ExitDisp
à
Low
inter-‐site
engagement
The
network
is
more
vulnerable
to
outages,
because
only
few
sites
are
used
to
enter
(leave)
the
network.
EntryDisp =
(gin
max − gin
i )
i∑
| N |• gin
ii∑
[Freeman]
L.
C
Freeman.
Centrality
in
social
networks
conceptual
clarificaHon.
Social
networks,
1979.
Number
of
visits
that
started
at
node
ni
(user
entered
the
network)
Maximum
value
of
gin
Number
of
nodes
| N |
gi
in
gin
max
EntryDisp = 20/3 40 = 0.17
20
10
10
40
5
5
EntryDisp = 70/3 50 = 0.47
79. Inter-‐site
engagement
79
EvaluaCon:
Network-‐level
CorrelaLons
between
inter-‐site
and
network
engagement
metrics.
[IS]Density
[IS]Reciprocity
[IS]EntryDisparity
[IS]ExitDisparity
[POP]#Sessions
[ACT]DwellTimeS
[ACT]#Sites
Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65
Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25
Reciprocity [IS] -0.38 -0.32 0.42 - 0.25
EntryDisparity [IS] 0.84 -0.54 0.33 -
ExitDisparity [IS] -0.55 0.38 0.20
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
Density
and
#Sessions
The
more
users
are
visiCng
the
network,
the
more
diverse
is
the
inter-‐
site
engagement.
Entry-‐
and
ExitDisparity
Volume
of
in-‐
and
out-‐
going
traffic
of
the
nodes
depend
on
each
other.
Flow
and
#Sites
The
more
sites
are
visited
during
a
session,
the
higher
the
flow
of
traffic.
80. Inter-‐site
engagement
80
EvaluaCon:
Node-‐level
CorrelaLons
between
inter-‐site
and
site
engagement
metrics.
[IS]Downstream
[IS]EntryProb
[IS]ExitProb
[POP]#Sessions
[ACT]DwellTimeS
[MT]#Visits
[MT]CumAct
PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31
Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02
EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35
ExitProb [IS] 0.08 -0.18 0.18 0.32
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
PageRank
and
#Sessions
Popular
sites
in
the
provider
network,
are
also
visited
frequently
when
browsing
through
the
network.
Entry-‐
and
ExitProb
Nodes
that
are
used
to
enter
the
network
are
also
frequently
used
to
exit
the
network.
84. NaLve
AdverLsing
84
Effect
on
User
Engagement
0%
200%
400%
600%
short ad clicks long ad clicks
adclickdifference
short ad clicks long ad clicks
clicksperdaydifference
PosiLve
experience
has
a
strong
effect
on
users
clicking
on
ads
again,
and
a
small
effect
on
user
engagement
with
the
stream.
85. NaLve
AdverLsing
85
Mobile
vs.
Desktop
Ad
post-‐click
experience
between
mobile
and
desktop
differs.
For
dwell
Lme
we
obtain
rho
=
0.50;
this
value
is
even
smaller
for
bounce
rate
with
rho
=
0.23.
0.00
0.05
0.10
0.15
dwell time difference
p(dwelltimedifference)
higher on mobilehigher on desktop
0.00
0.05
0.10
0.15
bounce rate difference
p(bounceratedifference)
higher on mobilehigher on desktop
86. NaLve
AdverLsing
86
Mobile
OpCmised
Landing
Pages
Dwell
Cme:
The
distribuLon
is
very
similar
for
both
groups.
Bounce
rate:
Decreases
by
6.9%
(median
decreases
by
30.4%)
for
Opt
landing
pages
but
increases
by
13.4%
(median
decreases
by
11.5%)
for
Npt
landing
pages.
not mobile optimized mobile optimized
0.0
0.1
0.2
0.3
dwell time difference
p(dwelltimedifference)
higher on mobilehigher on desktop
mobile opt.
not mobile opt.
0.0
0.1
0.2
bounce rate difference
p(bounceratedifference)
higher on mobilehigher on desktop
mobile opt.
not mobile opt.
88. Wikipedia
88
Wikipedia
Research
Literature
review
by
Okoli
et
al.:
The
people’s
encyclopedia
under
the
gaze
of
the
sages:
A
systemaLc
review
of
scholarly
research
on
wikipedia.
89. Wikipedia
89
Reading
Preferences
Popularitylow high
ArticleLengthshortlong
borderline casesII I
III IV
Jeanne Tsai
Douglas Adams
Luis
Palomino
Anne Stears
Peter
Ehrlich
Alec
Mango
Stephen D.
Lovejoy
1st
Dalai Lama
Dexter Jackson
(safety)
Katie Green
Brittany
Borman
Anthony Anenih
Ronnie Bird
Jan Anderson
(scientist)
Fitch
Robertson
Sean Bennett
For 4.2% (group IV) of the articles
editing activity is low, but reading activity is high.!
91. Wikipedia
91
Reading
PaPerns
over
Time
Stability
• 30%
of
the
arLcles
are
popular
in
1
month
• 10%
are
popular
over
the
whole
13-‐months
• Almost
all
arLcles
have
one
reading
pa;ern
half
of
their
life
Lme
TransiCons
• TransiLons
are
temporary
–
arLcles
move
temporarily
to
another
cluster
• High
reciprocity
–
similar
number
of
transiLons
in
both
direcLons
• “Focus”
cluster
is
isolated
-‐
ArLcles
in
that
cluster
are
the
most
stable
ones
• Strong
connecLon
between
the
“Passing”,
“ExploraLon”,
and
“Trending”
clusters
–
many
arLcles
adopt
all
three
pa;erns
93. 93
Upstream
Traffic
TeleportaCon
Social
media
/
News
Search
/
Ext-‐Yahoo
Users
engage
(quickly)
to
many
sites.
Users
conHnue
with
same
acHvity
inside
the
provider
network.
Users
visit
site
they
are
interested
in,
perform
a
quick
task,
and
leave.
Network
DwellTime
-‐-‐
++
-‐-‐
Traffic
DistribuCon
++
-‐-‐
-‐-‐
Entry
Disparity
-‐-‐
Yahoo
Users
engage
differently
depending
on
where
they
are
coming
from.
94. 94
Network
Effect
PaPern
Yahoo
Sites
change
their
popularity
in
the
same
way.
Ac>vity
(dwell
>me)
on
a
site
depends
more
on
the
site
itself,
but
there
are
some
nega>ve
dependencies.
Pattern
examples
41 patterns
Simple star-like
6 patterns
Complex star-like
1 pattern
Cluster-like
3.00 [3.00,4.00]
0.67 [0.00,0.89]
0 [0,0]
8.00 [7.00,18.00]
0.76 [0.56,0.84]
0 [0,0]
52
0.91
0.51
N
Recip
Trans
N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure
(4) (5) (6)(1) (2) (3)
98. Online
news
98
Hyperlink
Performance
Number
of
Inline
Links
• <10
links
may
be
wasLng
an
opportunity
• 10-‐29
links
does
not
result
in
more
clicks
• >29
links
may
harm
the
user
experience
PosiCon
of
Inline
Links
• 30%
at
the
end,
16%
at
the
beginning,
46%
are
distributed
within
the
arLcle
text.
• Performance
of
links
located
at
the
beginning
of
the
text
is
very
low
(-‐28%)
• Best
performance
is
achieved
with
links
at
the
end
of
the
arLcle
text
(+35%)
Link popularity● Link performance
Position in article text
Linkpopularity [0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0]
10%
20%
30%
-0.2
0.0
0.2
Linkperformance
●●
●●
●●
●● ●● ●● ●● ●●
●●
●●
●●
●●
●● ●● ●● ●● ●● ●●
●●
●●
Number of inline links in article
Clicksperlink
0.0
0.2
0.4
0.6
[0,2] [9,11] [18,20] [27,29] [36,38]
Number of inline links in article
Numberofclicks
[0,2] [9,11] [18,20] [27,29] [36,38]
2.5
5.0
7.5
99. Online
news
99
Effect
on
User
Engagement
Internal
Focused
Short-‐term:
Only
3
(out
of
50)
providers
have
their
corresponding
average
dwell
Lme
lower
for
the
story-‐focused
provider
sessions.
The
average
increase
in
dwell
Lme
from
non-‐story-‐focused
to
story-‐focused
provider
sessions
is
50%.
Long-‐term:
For
78%
of
the
providers,
we
find
that
there
are
more
users
that
return
earlier
aker
they
have
a
story-‐focused
provider
session.
Internal
News provider
Dwelltimepersession
Non-focused Focused Ext-focused
News provider
p(absence12h)
Non-focused Focused Ext-focused
100. Online
news
100
Effect
on
User
Engagement
External
Focused
Short-‐term:
We
do
not
observe
an
effect
on
the
dwell
Lme
(neither
posiLve
nor
negaLve).
The
average
increase
is
only
5.5%,
and
based
on
the
K-‐S
test
we
cannot
confirm
that
the
distribuLons
are
different
(p-‐
value=0.36).
Long-‐term:
For
70%
of
these
news
sites,
the
probability
that
users
return
within
the
following
12
hours
increases
(the
average
increase
is
76%).
External
News provider
Dwelltimepersession
Non-focused Focused Ext-focused
News provider
p(absence12h)
Non-focused Focused Ext-focused
101. Online
news
101
Discovering
Story-‐related
Content
in
TwiPer