SlideShare a Scribd company logo
1 of 101
Download to read offline
photo	
  credit	
  donsolo,	
  CC	
  BY-­‐NC-­‐SA	
  2.0	
  
From	
  Site	
  to	
  Inter-­‐site	
  User	
  Engagement	
  
Jane;e	
  Lehmann	
  
Barcelona,	
  February	
  26,	
  2015	
  
	
  
Advisors:	
  Ricardo	
  Baeza-­‐Yates	
  
Co-­‐Advisor:	
  Mounia	
  Lalmas	
  
•  User	
  engagement	
  is	
  a	
  quality	
  of	
  the	
  user	
  experience	
  that	
  emphasizes	
  the	
  
posiLve	
  aspects	
  of	
  interacLon	
  with	
  a	
  website	
  –	
  in	
  parLcular	
  the	
  fact	
  of	
  being	
  
capLvated	
  by	
  the	
  website.	
  
•  In-­‐the-­‐moment	
  engagement	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
Users	
  stay	
  on	
  a	
  website	
  over	
  a	
  long	
  Lme.	
  
•  Long-­‐term	
  engagement	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
Users	
  come	
  back	
  frequently	
  and	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
over	
  a	
  long-­‐term.	
  
IntroducLon	
   2	
  
User	
  Engagement	
  
DefiniLon	
  
Successful	
  websites	
  are	
  not	
  just	
  used,	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
they	
  are	
  engaged	
  with.	
  
User	
  Engagement	
  
Measuring	
  
3	
  IntroducLon	
  
Before	
  we	
  can	
  design	
  engaging	
  websites,	
  
it	
  is	
  crucial	
  that	
  we	
  are	
  able	
  to	
  measure	
  engagement.	
  
“If	
  you	
  can	
  measure	
  it,	
  you	
  can	
  improve	
  it.”	
  
Sir	
  William	
  Thomson	
  
Analysis/Planning	
  
Design	
  Changes	
  Measuring	
  
Main	
  Research	
  Goals	
  
4	
  IntroducLon	
  
Primary	
  goal 	
   	
  Can	
  we	
  define	
  new	
  engagement	
  metrics	
  that	
  	
  
Measuring 	
   	
   	
  enhance	
  our	
  understanding	
  of	
  engagement?	
  	
  
	
   	
   	
   	
   	
  	
  
Secondary	
  goal	
   	
  Can	
  we	
  idenLfy	
  ways	
  to	
  influence	
  engagement?
Analysis/Planning 	
   	
  	
  
Analysis/Planning	
  
Design	
  Changes	
  Measuring	
  
IntroducLon	
   5	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  Online	
  mulLtasking	
  
Inter-­‐site	
  
engagement	
  
Site	
  engagement	
  
Effect	
  of	
  providing	
  
off-­‐site	
  content	
  
Effect	
  of	
  hyperlinks	
  
IntroducLon	
   6	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  
Site	
  engagement	
  
Measuring	
  Engagement	
  
InteracLon	
  data	
  
7	
  Site	
  engagement	
  
Data	
  
Browsing	
  events	
  provided	
  by	
  Yahoo	
  toolbar	
  (client-­‐side).	
  
Engagement	
  
Analysing	
  the	
  data	
  using	
  online	
  behaviour	
  metrics.	
  
	
  
	
  
Online	
  session:	
  
Visit	
  on	
  Yahoo	
  News	
  
Site	
  engagement	
   8	
  
Measuring	
  Engagement	
  
Online	
  behaviour	
  metrics	
  
K.	
  Rodden,	
  H.	
  Hutchinson,	
  X.	
  Fu.	
  Measuring	
  the	
  user	
  experience	
  on	
  a	
  large	
  scale:	
  User-­‐centered	
  metrics	
  for	
  web	
  applicaHons.	
  CHI,	
  2010.	
  
E.	
  Peterson,	
  J.	
  Carrabis.	
  Measuring	
  the	
  immeasurable:	
  Visitor	
  engagement.	
  Web	
  AnalyHcs	
  DemysHfied,	
  2008.	
  
B.	
  Haven,	
  S.	
  ViWal.	
  Measuring	
  engagement.	
  Forrester	
  Research,	
  2008.	
  
B.	
  Weischedel	
  and	
  E.	
  Huizingh.	
  Website	
  opHmizaHon	
  with	
  web	
  metrics:	
  A	
  case	
  study.	
  Conference	
  on	
  Electronic	
  commerce,	
  2006.	
  
Site	
  engagement	
   9	
  
Measuring	
  Engagement	
  
Online	
  behaviour	
  metrics	
  
Popularity	
  
#Users	
   Number	
  of	
  users.	
  
#Visits	
   Number	
  of	
  visits.	
  
#Clicks	
   Number	
  of	
  clicks.	
  
AcCvity	
  (within	
  a	
  visit)	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  In-­‐the-­‐moment	
  engagement	
  
PageViews	
   Avg.	
  number	
  of	
  page	
  views	
  per	
  visit.	
  
DwellTime	
   Avg.	
  Lme	
  on	
  site	
  per	
  visit.	
  
Loyalty	
  (across	
  visits)	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  Long-­‐term	
  engagement	
  
ReturnRate	
   Number	
  of	
  Lmes	
  a	
  user	
  visited	
  the	
  site.	
  
AcLveDays	
   Number	
  of	
  days	
  a	
  user	
  visited	
  the	
  site.	
  
Site	
  engagement	
   10	
  
Measuring	
  Engagement	
  
Differences	
  in	
  engagement	
  
ComScore,	
  Alexa,	
  
GoogleAnalyHcs,…	
  
Shopping	
  
Users	
  do	
  not	
  come	
  
frequently,	
  but	
  
stay	
  long	
  
Games	
  
Not	
  many	
  users,	
  
but	
  they	
  stay	
  long	
  
News	
  
Users	
  come	
  
frequently	
  and	
  	
  
stay	
  long	
  
Measuring	
  Engagement	
  
Problem	
  
11	
  Site	
  engagement	
  
Isolated	
  view:	
  The	
  metrics	
  focus	
  
on	
  engagement	
  with	
  a	
  single	
  site.	
  
RelaLonships	
  to	
  other	
  sites	
  are	
  
not	
  considered.	
  
IntroducLon	
   12	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  Online	
  mulLtasking	
  
Site	
  engagement	
  
Online	
  mulLtasking	
   13	
  
MoCvaCon	
  
In-­‐the-­‐moment	
  engagement	
  
ComScore,	
  Alexa,	
  
GoogleAnalyHcs,…	
  
What	
  web	
  analyCcs	
  think	
  we	
  do…	
  
1	
  visit	
  with	
  4	
  page	
  views.	
  
Online	
  mulLtasking	
   14	
  
MoCvaCon	
  
In-­‐the-­‐moment	
  engagement	
  
ComScore,	
  Alexa,	
  
GoogleAnalyHcs,…	
  
…	
  and	
  what	
  we	
  really	
  do:	
  
3	
  visit	
  with	
  on	
  average	
  1.3	
  page	
  views.	
  
Online	
  mulLtasking	
   15	
  
MoCvaCon	
  
Online	
  mulLtasking.	
  
Problem	
  
	
  
•  Engagement	
  metrics	
  do	
  not	
  capture	
  such	
  behaviour.	
  
	
  
•  Measuring	
  acLvity	
  on	
  a	
  site	
  can	
  lead	
  to	
  incorrect	
  conclusions.	
  
	
  
Online	
  mulCtasking	
  
Users	
  visit	
  several	
  sites	
  and	
  switch	
  between	
  them	
  
during	
  an	
  online	
  session,	
  to	
  perform	
  related	
  or	
  totally	
  
unrelated	
  tasks.	
  
Research	
  QuesCon	
  
16	
  Online	
  mulLtasking	
  
How	
  can	
  we	
  measure	
  engagement	
  by	
  
accounLng	
  for	
  user	
  mulLtasking	
  behaviour?	
  
Analysis/Planning	
  
Design	
  Changes	
  Measuring	
  
Extent	
  of	
  mulCtasking	
  
•  10.2	
  disLnct	
  sites,	
  2	
  visits	
  per	
  site.	
  
	
  
Absence	
  Cme	
  
•  50%	
  of	
  sites	
  are	
  revisited	
  aker	
  <	
  1min.	
  
	
  InterrupHon	
  of	
  a	
  task	
  
	
  
•  There	
  are	
  revisits	
  aker	
  long	
  breaks.	
  
	
  Performing	
  a	
  new	
  task	
  
Online	
  mulLtasking	
   17	
  
Online	
  MulCtasking	
  
CharacterisLcs	
  
0.00
0.25
0.50
0.75
1.00
10
2
10
1
10
0
10
1
10
2
Cumulativeprobability Absence time [min]
news (finance)
news (tech)
social media
mail
2.09
1.76
2.28
2.09
#Visits Absence
time [min]
3.85
3.95
4.47
6.86
Absence time: Time between two visits
AcCvity	
  paPerns	
  	
  
•  Four	
  types:	
  Decreasing,	
  increasing,	
  constant,	
  complex.	
  
•  Successive	
  visits	
  can	
  belong	
  together	
  (i.e.	
  to	
  the	
  same	
  task).	
  
•  Complex	
  cases	
  refer	
  to	
  no	
  specific	
  pa;ern	
  or	
  repeated	
  pa;ern.	
  
	
  
Online	
  mulLtasking	
   18	
  
Online	
  MulCtasking	
  
CharacterisLcs	
  
1 2 3 4
ith
visit on site
1 2 3 4
ith
visit on site
1 2 3 4
ith
visit on site
1 2 3 4
ith
visit on site
Proportionoftotal
dwelltimeonsite
0.23
0.28
0.33 p-value = 0.09
m = -0.01
p-value = 0.07
m = -0.02
p-value = 0.79
m = 0.00
news (finance) sitesmail sites social media sites news (tech) sites
decreasing attention increasing attention constant attention complex attention
Online	
  mulLtasking	
   19	
  
Measuring	
  Engagement	
  
Online	
  mulLtasking	
  metrics	
  
Extent	
  of	
  mulCtasking	
  
SessSites	
   Total	
  number	
  of	
  sites	
  accessed	
  (#tasks).	
  
SessVisits	
   Number	
  of	
  visits	
  to	
  site	
  (site	
  switching).	
  
Absence	
  Cme	
  
CumAct	
  
Aggregates	
  the	
  dwell	
  Lmes	
  of	
  the	
  visits	
  with	
  
accounLng	
  for	
  the	
  Lme	
  between	
  the	
  visits.	
  
AcCvity	
  paPern	
  
A;Shik	
  
A;Range	
  
Describe	
  the	
  four	
  cases	
  of	
  a;enLon	
  shiks.	
  
20	
  
CASE	
  STUDY:	
  
MulCtasking	
  PaPerns	
  
	
  
•  ObjecCve:	
  Analyse	
  mulLtasking	
  acLvity	
  on	
  sites;	
  
idenLfy	
  mulLtasking	
  pa;erns	
  (clustering).	
  
•  Metrics:	
  Site	
  DwellTime,	
  MulLtasking	
  metrics.	
  
•  Data:	
  July	
  2012,	
  2.5M	
  users,	
  760	
  sites	
  (shopping,	
  
news,	
  search,	
  etc.).	
  
21	
  
Case	
  Study:	
  MulCtasking	
  PaPerns	
  
Results	
  
No	
  mulCtasking	
   MulCtasking	
  
Quick	
   Focused	
   Rapid	
   ConCnuous	
   Recurring	
  
Checking	
  
weather	
  
Reading	
  
mails	
  
Following	
  link	
  to	
  
off-­‐site	
  content	
  
Purchasing	
  
an	
  item	
  
Performing	
  
search	
  
Site	
  
DwellTime	
  
-­‐-­‐	
   ++	
   ++	
   ++	
   -­‐-­‐	
  
Extent	
  of	
  
mulCtasking	
  
-­‐-­‐	
   -­‐-­‐	
   ++	
   ++	
   ++	
  
Absence	
  
Cme	
  
-­‐-­‐	
   ++	
   ++	
  
ImplicaCons	
  
Provide	
  
interesHng	
  off-­‐
site	
  content	
  
Shopping	
  
takes	
  more	
  than	
  
one	
  visit	
  
Support	
  user	
  
by	
  finishing	
  
tasks	
  quickly	
  
Online	
  mulLtasking	
  
-- low value ++ high value
22	
  
Case	
  Study:	
  MulCtasking	
  PaPerns	
  
Results	
  
No	
  mulCtasking	
   MulCtasking	
  
Quick	
   Focused	
   Rapid	
   ConCnuous	
   Recurring	
  
Checking	
  
weather	
  
Reading	
  
mails	
  
Following	
  link	
  to	
  
off-­‐site	
  content	
  
Purchasing	
  
an	
  item	
  
Performing	
  
search	
  
Site	
  
DwellTime	
  
-­‐-­‐	
   ++	
   ++	
   ++	
   -­‐-­‐	
  
Extent	
  of	
  
mulCtasking	
  
-­‐-­‐	
   -­‐-­‐	
   ++	
   ++	
   ++	
  
Absence	
  
Cme	
  
-­‐-­‐	
   ++	
   ++	
  
AcCvity	
  
paPern	
  
Online	
  mulLtasking	
  
De In CmCn
60%
0%
De In CmCn
60%
0%
De In CmCn
60%
0%
Activity pattern: De – Decreasing In – Increasing Cn – Constant Cm - Complex
-- low value ++ high value
23	
  
CASE	
  STUDY:	
  
Wikipedia	
  (on-­‐site	
  mulCtasking)	
  
	
  
•  ObjecCve:	
  Analyse	
  reading	
  acLvity	
  on	
  Wikipedia	
  
arLcles;	
  idenLfy	
  reading	
  pa;erns	
  (clustering).	
  
•  Metrics:	
  ArLcle	
  DwellTime,	
  #ArLcles	
  in	
  session,	
  
#Views	
  to	
  focal	
  arLcle.	
  
•  Data:	
  Sep	
  2011	
  –	
  Sep	
  2012,	
  500K	
  users,	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
10K	
  biography	
  arLcles.	
  
24	
  
Case	
  Study:	
  Wikipedia	
  
Approach	
  
Online	
  mulLtasking	
  
Users’	
  reading	
  behaviour	
  on	
  an	
  Wikipedia	
  arCcle	
  
ArLcle	
  DwellTime	
   	
   	
   	
  How	
  much	
  Lme	
  do	
  users	
  spend	
  on	
  an	
  arLcle?	
  
#ArLcles	
  in	
  session 	
   	
   	
  Do	
  users	
  view	
  also	
  other	
  arLcles	
  during	
  an	
  	
  
	
   	
   	
   	
   	
   	
   	
  online	
  session?	
  
#Views	
  on	
  focal	
  arLcle 	
   	
  How	
  oken	
  do	
  users	
  view	
  the	
  arLcle?	
  
25	
  
Case	
  Study:	
  Wikipedia	
  
Results	
  
No	
  mulCtasking	
   MulCtasking	
  
Focus	
   ExploraCon	
   Passing	
  
Focus	
  is	
  on	
  
focal	
  arHcle	
  
Exploring	
  topic	
  
around	
  the	
  focal	
  arHcle	
  
Exploring	
  topic	
  and	
  pass	
  
through	
  the	
  focal	
  arHcle	
  
ArCcle	
  
DwellTime	
  
++	
   -­‐-­‐	
  
#ArCcles	
  in	
  
session	
  
-­‐-­‐	
   ++	
   ++	
  
#Views	
  to	
  	
  	
  	
  	
  	
  
focal	
  arCcle	
  
++	
   -­‐-­‐	
  
ImplicaCons	
  
Content	
  quality	
  
is	
  important	
  
Links	
  to	
  
addiHonal	
  content	
  
are	
  important	
  
ArHcles	
  might	
  
need	
  to	
  be	
  extended	
  
Online	
  mulLtasking	
  
 
On-­‐site	
  mulCtasking	
  
	
  
•  MulLtasking	
  between	
  news	
  arLcles	
  of	
  a	
  provider.	
  
•  MulLtasking	
  between	
  different	
  tasks	
  on	
  a	
  social	
  media	
  
site	
  (e.g.	
  sharing,	
  chapng,	
  updaLng	
  profile).	
  
•  …	
  
	
  
Inter-­‐site	
  mulCtasking	
  
	
  
•  MulLtasking	
  when	
  purchasing	
  items	
  online	
  (comparing	
  
offers,	
  product	
  reviews,	
  search,	
  etc.)	
  
•  …	
  
	
  
Online	
  mulLtasking	
   26	
  
Further	
  Use	
  Cases	
  
Take	
  Aways	
  
•  AccounLng	
  for	
  mulLtasking	
  leads	
  to	
  a	
  
be;er	
  understanding	
  on	
  how	
  users	
  
engage	
  with	
  sites.	
  
	
  
•  Leaving	
  a	
  site	
  does	
  not	
  necessarily	
  
entail	
  less	
  engagement,	
  as	
  users	
  oken	
  
return	
  to	
  the	
  site	
  later	
  on.	
  
	
  
	
  
Publications
J. Lehmann, M. Lalmas, G.
Dupret, and R. Baeza-Yates.
Online multitasking and user
engagement. CIKM 2013.
J. Lehmann, C. Müller-Birn, D.
Laniado, M. Lalmas, and A.
Kaltenbrunner. Reader
preferences and behavior on
Wikipedia. HT 2014, Ted
Nelson Newcomer Paper
Award.
J. Lehmann, C. Müller-Birn, D.
Laniado, M. Lalmas, and A.
Kaltenbrunner. What and
how users read: Transforming
reading behavior into
valuable feedback for the
Wikipedia community.
Wikimania 2014.
Online	
  mulLtasking	
   27	
  
IntroducLon	
   28	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  Online	
  mulLtasking	
  
Inter-­‐site	
  
engagement	
  
Site	
  engagement	
  
Inter-­‐site	
  engagement	
   29	
  
MoCvaCon	
  
Large	
  online	
  service	
  providers	
  
ComScore,	
  Alexa,	
  
GoogleAnalyHcs,…	
  
Engagement	
  
Popularity:	
  #Users,	
  #Visits,	
  …	
  
AcLvity:	
  DwellTime,	
  PageViews,	
  …	
  
Loyalty:	
  ReturnRate,	
  AcLveDays,	
  …	
  
Inter-­‐site	
  engagement	
   30	
  
MoCvaCon	
  
Large	
  online	
  service	
  providers	
  
frontpage	
  
tv	
   sports	
  
shopping	
  
autos	
  
search	
  
daLng	
  
jobs	
  
news	
  
shine	
  
groups	
  
maps	
  
local	
  
health	
  
answer	
  
weather	
  
games	
  
mail	
  
omg	
  
homes	
  
travel	
  
flickr	
  
finance	
  
Large	
  online	
  service	
  	
  
providers	
  	
  
(AOL,	
  Google,	
  Yahoo,	
  etc.)	
  	
  
have	
  not	
  only	
  one	
  site,	
  	
  
but	
  many	
  sites.	
  
tumblr	
  
Inter-­‐site	
  engagement	
   31	
  
MoCvaCon	
  
Large	
  online	
  service	
  providers	
  
frontpage	
  
tv	
   sports	
  
shopping	
  
autos	
  
search	
  
daLng	
  
jobs	
  
news	
  
shine	
  
groups	
  
maps	
  
local	
  
health	
  
answer	
  
weather	
  
games	
  
mail	
  
omg	
  
homes	
  
travel	
  
flickr	
  
finance	
  
Providers	
  want	
  
that	
  users	
  engage	
  with	
  
many	
  of	
  their	
  sites.	
  
tumblr	
  
Inter-­‐site	
  engagement	
   32	
  
MoCvaCon	
  
Online	
  mulLtasking	
  
Problem	
  
	
  
•  Engagement	
  metrics	
  do	
  not	
  measure	
  engagement	
  across	
  sites.	
  	
  
•  How	
  to	
  adapt	
  them	
  is	
  not	
  obvious.	
  
	
  
Inter-­‐site	
  engagement	
  
Users	
  visit	
  sites	
  that	
  belong	
  to	
  the	
  
same	
  network	
  of	
  sites.	
  
Research	
  QuesCon	
  
33	
  Inter-­‐site	
  engagement	
  
How	
  can	
  we	
  measure	
  
engagement	
  by	
  also	
  considering	
  the	
  
relaLonships	
  between	
  sites?	
  
Analysis/Planning	
  
Design	
  Changes	
  Measuring	
  
Inter-­‐site	
  engagement	
   34	
  
Traffic	
  Networks	
  
Modelling	
  
We	
  model	
  sites	
  (nodes)	
  and	
  user	
  traffic	
  
(edges)	
  between	
  them	
  as	
  a	
  network.	
  
	
  
Provider	
  network	
  G=(N,	
  E,	
  λ)	
  
	
  N: 	
   	
  Sites	
  
	
  E: 	
   	
  User	
  traffic	
  
	
  λ(e): 	
  Traffic	
  volume	
  (#Clicks)	
  
	
  
	
  
4	
  clicks	
  
2	
  clicks	
  
50	
  clicks	
  10	
  clicks	
  
Inter-­‐site	
  engagement	
   35	
  
Measuring	
  Engagement	
  
Inter-­‐site	
  engagement	
  metrics:	
  Network-­‐level	
  
Traffic	
  distribuCon	
  
Flow	
  
Extent	
  to	
  which	
  users	
  navigate	
  
between	
  sites.	
  
Density1	
   Diversity	
  of	
  inter-­‐site	
  engagement.	
  	
  
Reciprocity2	
   Homogeneity	
  of	
  traffic	
  between	
  sites.	
  	
  
External	
  traffic	
  
EntryDisparity	
  
Variability	
  of	
  in-­‐going	
  traffic	
  to	
  the	
  
network.	
  	
  
ExitDisparity	
  
Variability	
  of	
  out-­‐going	
  traffic	
  from	
  
the	
  network.	
  	
  
[1]	
  S.	
  Wasserman.	
  Social	
  network	
  analysis:	
  Methods	
  and	
  applicaHons,	
  1994.	
  
[2]	
  T.	
  SquarHni,	
  F.	
  Picciolo,	
  F.	
  RuzzenenH,	
  and	
  D.	
  Garlaschelli.	
  Reciprocity	
  of	
  weighted	
  networks.	
  Nature:	
  ScienHfic	
  reports,	
  2013.	
  
Inter-­‐site	
  engagement	
   36	
  
Measuring	
  Engagement	
  
Inter-­‐site	
  engagement	
  metrics:	
  Node-­‐level	
  
Traffic	
  distribuCon	
  
PageRank1	
  
Probability	
  that	
  a	
  user	
  will	
  visit	
  the	
  
site.	
  
Downstream	
  
Probability	
  that	
  a	
  user	
  will	
  conLnue	
  
browsing	
  to	
  other	
  sites.	
  
External	
  traffic	
  
EntryProb	
  
Probability	
  that	
  a	
  user	
  enters	
  the	
  
network	
  in	
  this	
  site.	
  
ExitProb	
  
Probability	
  that	
  a	
  user	
  leaves	
  the	
  
network	
  in	
  this	
  site.	
  
	
  
[1]	
  L.	
  Page,	
  S.	
  Brin,	
  R.	
  Motwani,	
  T.	
  Winograd.	
  The	
  pagerank	
  citaHon	
  ranking:	
  Bringing	
  order	
  to	
  the	
  web.	
  Technical	
  report,	
  Stanford	
  InfoLab,	
  1999.	
  
37	
  
CASE	
  STUDY:	
  
Yahoo	
  Provider	
  Networks	
  
	
  
•  ObjecCve:	
  Compare	
  networks;	
  characterise	
  the	
  sites	
  
in	
  a	
  network.	
  
•  Metrics:	
  Network	
  DwellTime,	
  Site	
  DwellTime,	
  Inter-­‐
site	
  engagement	
  metrics.	
  
•  Data:	
  February	
  2014,	
  3.2M	
  clicks/network,	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
4	
  country-­‐based	
  networks,	
  31	
  sites	
  per	
  network.	
  
38	
  
Case	
  Study:	
  Yahoo	
  
Comparing	
  provider	
  networks	
  
Network	
  1	
   Network	
  2	
   Network	
  3	
   Network	
  4	
  
High	
  
engaging	
  
Users	
  engage	
  quickly	
  
with	
  many	
  sites	
  
Users	
  engage	
  to	
  a	
  
subset	
  of	
  sites	
  
Low	
  
engaging	
  
Network	
  
DwellTime	
  
++	
   -­‐-­‐	
   ++	
   -­‐-­‐	
  
Traffic	
  
DistribuCon	
  
++	
   ++	
  
Flow	
  ++	
  
Density	
  -­‐-­‐	
  
-­‐-­‐	
  
Entry	
  
Disparity	
  
++	
   -­‐-­‐	
   ++	
  
ImplicaCons	
  
The	
  network	
  
is	
  performing	
  
well.	
  
This	
  should	
  
be	
  looked	
  into.	
  
MoHvate	
  
users	
  to	
  visit	
  
other	
  sites.	
  
This	
  should	
  
be	
  looked	
  into.	
  
Inter-­‐site	
  engagement	
  
-- low value ++ high value
39	
  
Case	
  Study:	
  Yahoo	
  
Sites	
  within	
  a	
  provider	
  network	
  
Traffic	
  Hub	
   Supporter	
  
Focused	
  
Engagement	
  
Shared	
  
Engagement	
  
Search,	
  front	
  pages	
   Support,	
  services	
   Leisure,	
  support	
   News,	
  leisure	
  
Site	
  
DwellTime	
  
-­‐-­‐	
   -­‐-­‐	
   ++	
   ++	
  
Traffic	
  
DistribuCon	
  
++	
   -­‐-­‐	
   -­‐-­‐	
   ++	
  
Entry	
  
Probability	
  
++	
   -­‐-­‐	
   ++	
   -­‐-­‐	
  
ImplicaCons	
  
The	
  sites	
  
forward	
  traffic	
  to	
  
other	
  sites.	
  
Users	
  visit	
  sites	
  
for	
  specific	
  needs	
  
and	
  support.	
  
MoHvate	
  
users	
  to	
  visit	
  
other	
  sites.	
  
The	
  sites	
  
are	
  performing	
  
well.	
  
Inter-­‐site	
  engagement	
  
-- low value ++ high value
 
Comparing	
  networks	
  
	
  
•  Device,	
  Lme,	
  upstream	
  traffic,	
  user.	
  
•  SimulaLons	
  (effect	
  of	
  adding/removing	
  sites).	
  
•  …	
  
	
  
Network	
  types	
  
	
  
•  Network	
  of	
  pages	
  (e.g.	
  compare	
  language-­‐based	
  
Wikipedia	
  networks)	
  
•  Network	
  of	
  sites	
  from	
  different	
  providers	
  (e.g.	
  shopping	
  
sites,	
  news	
  providers)	
  
•  …	
  
	
  
Inter-­‐site	
  engagement	
   40	
  
Further	
  Use	
  Cases	
  
Take	
  Aways	
  
•  Inter-­‐site	
  engagement	
  allows	
  for	
  a	
  
more	
  comprehensive	
  look	
  at	
  user	
  
engagement	
  by	
  also	
  considering	
  the	
  
relaLonships	
  between	
  sites.	
  	
  
	
  
•  Deeply	
  engaged	
  users	
  do	
  not	
  only	
  
engage	
  with	
  one	
  site,	
  but	
  with	
  many	
  
sites	
  in	
  a	
  network.	
  
	
  
	
  
Publications
J. Lehmann, M. Lalmas, and R. Baeza-
Yates. Measuring Inter-Site Engagement.
Handbook of Statistics, Elsevier, 2015. To
appear.
J. Lehmann, M. Lalmas, R. Baeza-Yates,
and E. Yom-Tov. Networked User
Engagement. ACM Workshop on User
engagement optimization at CIKM, 2013.
J. Lehmann, M. Lalmas, and R. Baeza-
Yates. Temporal Variations in Networked
User Engagement. TNETS Satellite at ECCS,
2013.
Some of the metrics were employed to
characterise online news reading across
news sites:
J. Lehmann, C. Castillo, M. Lalmas, and R.
Baeza-Yates. Story-Focused Reading in
Online News. Submitted for publication.
Inter-­‐site	
  engagement	
   41	
  
IntroducLon	
   42	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  Online	
  mulLtasking	
  
Inter-­‐site	
  
engagement	
  
Site	
  engagement	
  
Effect	
  of	
  providing	
  
off-­‐site	
  content	
  
43	
  
CASE	
  STUDY:	
  
Online	
  News	
  
	
  
•  Hypothesis:	
  It	
  may	
  be	
  beneficial	
  (long-­‐term)	
  to	
  
enLce	
  users	
  to	
  leave	
  a	
  site	
  by	
  offering	
  interesLng	
  
off-­‐site	
  content.	
  
•  Data:	
  October	
  2013,	
  57K	
  users,	
  50	
  news	
  sites,	
  
26K	
  news	
  arLcles.	
  
Types	
  of	
  reading	
  sessions	
  
	
  
No	
  click	
   	
  	
  	
  Did	
  not	
  follow	
  a	
  related	
  	
  	
  	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  link.	
  
	
  
Off-­‐site	
  click	
  	
  	
  Followed	
  a	
  related	
  link	
  to	
  	
  
	
   	
   	
  	
  	
  content	
  on	
  another	
  site.	
  
	
  
Effect	
  on	
  engagement	
  
	
  
Short-­‐term 	
  Dwell	
  Lme	
  per	
  reading	
  	
  
	
   	
   	
  session.	
  
	
  
Long-­‐term 	
  Probability	
  that	
  user	
  starts	
  	
  
	
   	
   	
  next	
  reading	
  session	
  within	
  	
  
	
   	
   	
  the	
  next	
  12h.	
  
	
  
	
   44	
  
Case	
  Study:	
  Online	
  News	
  
Related	
  off-­‐site	
  content	
  
Approach	
  
Effect	
  of	
  providing	
  off-­‐site	
  content	
  
Providing	
  links	
  to	
  related	
  off-­‐site	
  content	
  has	
  a	
  
no	
  short-­‐term	
  effect,	
  but	
  a	
  posiCve	
  long-­‐term	
  effect.	
  
	
  
45	
  
Case	
  Study:	
  Online	
  News	
  
Results	
  
Effect	
  of	
  providing	
  off-­‐site	
  content	
  
News provider
Dwelltimepersession
News provider
p(absence12h)
No Click Off-site click
IntroducLon	
   46	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  Online	
  mulLtasking	
  
Inter-­‐site	
  
engagement	
  
Site	
  engagement	
  
Effect	
  of	
  providing	
  
off-­‐site	
  content	
  
Effect	
  of	
  hyperlinks	
  
47	
  
CASE	
  STUDY:	
  
Yahoo	
  Provider	
  Network	
  
	
  
•  Hypothesis:	
  We	
  can	
  use	
  hyperlinks	
  to	
  influence	
  
inter-­‐site	
  engagement	
  in	
  a	
  provider	
  network.	
  
•  Data:	
  February	
  2014,	
  235M	
  clicks,	
  Yahoo	
  US	
  
network,	
  73	
  sites.	
  
 
Hyperlink	
  vs.	
  traffic	
  network	
  
	
  
On-­‐site 	
   	
  Links/Traffic	
  to	
  pages	
  	
  
	
   	
   	
  within	
  the	
  same	
  site.	
  
	
  
Inter-­‐site 	
  Links/Traffic	
  to	
  pages	
  to	
  	
  
	
   	
   	
  other	
  sites	
  in	
  the	
  	
  
	
   	
   	
  network.	
  
	
  
External	
   	
  Links/Traffic	
  to	
  	
  
	
   	
   	
  somewhere	
  	
  
	
   	
   	
  else	
  on	
  the	
  Web.	
  
48	
  
Case	
  Study:	
  Yahoo	
  
Approach	
  
frontpage	
  
sports	
  
search	
  
news	
  
shine	
  
groups	
  
answer	
  
weather	
  
mail	
  
omg	
  
homes	
  
flickr	
  
Effect	
  of	
  hyperlinks	
  
Hyperlinks	
  can	
  be	
  used	
  to	
  influence	
  site	
  
and	
  inter-­‐site	
  engagement	
  in	
  a	
  provider	
  network.	
  
However,	
  both	
  types	
  of	
  engagement	
  influence	
  each	
  other.	
  	
  
49	
  
Case	
  Study:	
  Yahoo	
  
Results	
  
Effect	
  of	
  hyperlinks	
  
Traffic
On-site Inter-site External
Hyperlinks
On-site
Inter-site
External
0.54
-0.40
-
-0.45
0.50
-
-0.38
-
0.39
IntroducLon	
   50	
  
Analysis/
Planning	
  
Design	
  Changes	
  Measuring	
  Online	
  mulLtasking	
  
Inter-­‐site	
  
engagement	
  
Site	
  engagement	
  
Effect	
  of	
  providing	
  
off-­‐site	
  content	
  
Effect	
  of	
  hyperlinks	
  
Two	
  new	
  perspecHves	
  for	
  measuring	
  
engagement	
  which	
  consider	
  the	
  
relaLonships	
  between	
  sites.	
  
	
  
Online	
  mulCtasking	
  
Accounts	
  for	
  user	
  mulLtasking	
  
behaviour.	
  
	
  
Inter-­‐site	
  engagement	
  
Accounts	
  for	
  the	
  traffic	
  between	
  sites.	
  
ContribuLons	
  and	
  future	
  work	
   51	
  
Main	
  ContribuCons	
  
Measuring	
  engagement	
  
Analysis/
Planning	
  
Design	
  
Changes	
  
Measuring	
  
AccounLng	
  for	
  the	
  new	
  perspecLves	
  
when	
  influencing	
  engagement.	
  
	
  
Online	
  news	
  
Providing	
  related	
  off-­‐site	
  content	
  
influences	
  long-­‐term	
  engagement.	
  
	
  
Provider	
  network	
  
Hyperlinks	
  affect	
  site	
  and	
  inter-­‐site	
  
engagement,	
  but	
  both	
  influence	
  
each	
  other.	
  
	
  
ContribuLons	
  and	
  future	
  work	
   52	
  
Main	
  ContribuCons	
  
Analysis/Planning	
  
Analysis/
Planning	
  
Design	
  
Changes	
  
Measuring	
  
Wikipedia	
  
Providing	
  informaLon	
  about	
  readers’	
  
engagement	
  to	
  the	
  editor	
  community.	
  
Yahoo	
  
Using	
  inter-­‐site	
  engagement	
  metrics	
  to	
  
make	
  informed	
  decisions	
  about	
  design	
  
changes	
  (hyperlinks).	
  
Spiegel	
  Online	
  
Measuring	
  and	
  improving	
  engagement	
  
by	
  providing	
  interesLng	
  off-­‐site	
  
content.	
  
ContribuLons	
  and	
  future	
  work	
   53	
  
What	
  next?	
  
Ongoing	
  and	
  future	
  work	
  
Analysis/
Planning	
  
Design	
  
Changes	
  
Measuring	
  
photo	
  credit	
  donsolo,	
  CC	
  BY-­‐NC-­‐SA	
  2.0	
  
Thank	
  you!	
  
	
  
Jane;e	
  Lehmann	
  
	
  
Barcelona,	
  February	
  26,	
  2015	
  
lehmannj@acm.org	
  
Acknowledgements	
  
Ricardo	
  Baeza-­‐Yates	
  
Mounia	
  Lalmas	
  
Claudia	
  Müller-­‐Birn	
  
Carlos	
  CasLllo	
  
David	
  Laniado	
  
Andreas	
  Kaltenbrunner	
  	
  
Elad	
  Yom-­‐Tov	
  
Georges	
  Dupret	
  
Guy	
  Shaked	
  
Fabrizio	
  Silvestri	
  
Gabriele	
  Tolomei	
  	
  
Ethan	
  Zuckerman	
  	
  
John	
  Agapiou	
  
Andy	
  Haines	
  
Diego	
  Sáez-­‐Trumper	
  
Hemant	
  Purohit	
  
Noora	
  Al	
  Emadi	
  
Mohammed	
  El-­‐Haddad	
  
Nasir	
  Khan	
  
	
  
•  Mounia Lalmas and Janette Lehmann. “Models of User Engagement”. In H. L. O’Brien and M. Lalmas (Eds.), Why Engagement
Matters: Cross-disciplinary Perspectives and Innovations on User Engagement with Digital Media. Springer, 2015, in progress.
•  Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. “Models of user engagement.” International
Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), pp. 164-175, Montreal, Canada, July, 2012.
•  Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. “Online multitasking and user engagement.”
ACM International Conference on Information and Knowledge Management (CIKM 2013), pp. 519-528, San Francisco, United
States, October, 2013.
•  Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Measuring Inter-Site Engagement.”. In V. Govindaraju, V. V.
Raghavan, and C. R. Rao (Eds.), Handbook of Statistics, Elsevier, 2015.
•  Janette Lehmann, Mounia Lalmas, Ricardo Baeza-Yates, and Elad Yom-Tov. “Networked User Engagement.”, ACM Workshop
on User engagement optimization at CIKM, pp. 7-10, San Francisco, United States, October, 2013.
•  Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Temporal Variations in Networked User Engagement.”, TNETS
Satellite at European Conference on Complex Systems (ECCS), Barcelona, Spain, September, 2013.
•  Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. “Measuring Post-click User Experience
with Mobile Native Advertising on Streams.”, submitted for publication.
•  Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “Reader preferences and
behavior on Wikipedia.”, ACM International Conference on Hypertext and Social Media (HT 2014), pp. 88-97, Santiago, Chile,
September, 2014, Ted Nelson Newcomer Paper Award.
•  Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “What and how users
read: Transforming reading behavior into valuable feedback for the Wikipedia community.”, Presentation at Wikimania,
London, UK, August, 2014.
•  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. “Story-Focused Reading in Online News.”,
submitted for publication.
•  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Transient News Crowds in Social Media.”
International AAAI Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA, July, 2013.
•  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Finding News Curators in Twitter.” ACM
International Conference on World Wide Web Companion (WWW 2013 Companion), 863-870, Rio de Janeiro, Brazil, May,
2013.
55	
  
PublicaCons	
  
User engagement
•  Mounia Lalmas, Heather L O’Brien, and Elad Yom-Tov. Measuring user engagement. Synthesis Lectures on Sample Series #1.
Morgan and cLaypool publishers, 2014.
•  Heather L O’Brien and Elaine G Toms. What is user engagement? a conceptual framework for defining user engagement
with technology. American Society for Information Science and Technology (ASIS&T), 59(6):938–955, 2008.
•  Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. Towards a science of user engagement (position
paper). In Proc. Workshop on User Modelling for Web Applications, WSDM, 2011.
•  Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web
applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.
Online behaviour metrics
•  Brian Haven and Suresh Vittal. Measuring engagement. Forrester Research, 2008.
•  Eric T Peterson and Joseph Carrabis. Measuring the immeasurable: Visitor engagement. Web Analytics Demystified, 2008.
•  Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web
applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010.
•  Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. In Proc.
Conference on Web Search and Data Mining, WSDM, pages 173–182. ACM, 2013.
•  Randolph E Bucklin and Catarina Sismeiro. A model of web site browsing behavior estimated on clickstream data. Journal of
Marketing Research, 40(3):249–267, 2003.
•  Birgit Weischedel and Eelko KRE Huizingh. Website optimization with web metrics: a case study. In Proc. Conference on
Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to
conducting successful business on the internet, pages 463–470. ACM, 2006.
•  Peifeng Yin, Ping Luo, Wang-Chien Lee, and Min Wang. Silence is also evidence: interpreting dwell time for recommendation
from psychological perspective. In Proc. Conference on Knowledge Discovery and Data Mining, SIGKDD, pages 989–997.
ACM, 2013.
56	
  
Selected	
  References	
  
Online multitasking
•  Qing Wang and Huiyou Chang. Multitasking bar: prototype and evaluation of introducing the task concept into a browser. In
Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 103–112. ACM, 2010.
•  Hartmut Obendorf, Harald Weinreich, Eelco Herder, and Matthias Mayer. Web page revisitation revisited: implications of a
long-term click-stream study of browser usage. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages
597–606. ACM, 2007.
•  Jeff Huang and Ryen W White. Parallel browsing behavior on the web. In Proc. Conference on Hypertext and Hypermedia,
HT, pages 13–18. ACM, 2010.
•  Patrick Dubroy and Ravin Balakrishnan. A study of tabbed browsing among mozilla firefox users. In Proc. Conference on
Human Factors in Computing Systems, SIGCHI, pages 673–682. ACM, 2010.
Inter-site engagement
•  Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167–256, 2003. 76, 77, 165
•  Anna Chmiel, Kamila Kowalska, and Janusz A Hołyst. Scaling of human behavior during portal browsing.
•  Mark R Meiss, Filippo Menczer, Santo Fortunato, Alessandro Flammini, and Alessandro Vespignani. Ranking web sites with real
user traffic. In Proc. Conference on Web Search and Data Mining, WSDM, pages 65–76. ACM, 2008.
•  Young-Hoon Park and Peter S Fader. Modeling browsing behavior at multiple websites. Marketing Science, 23(3):280–303,
2004.
•  Qiqi Jiang, Chuan-Hoo Tan, and Kwok-Kee Wei. Cross-website navigation behavior and purchase commitment: A pluralistic
field research. In Proc. Pacific Asia Conference on Information Systems, PACIS, 2012.
•  Kevin Koidl, Owen Conlan, and Vincent Wade. Cross-site personalization: assisting users in addressing information needs that
span independently hosted websites. In Proc. Conference on Hypertext and Hypermedia, HT, pages 66–76. ACM, 2014.
•  The PEW Research Center. Understanding the participatory news consumer. http://www.pewinternet.org/~/media/Files/
Reports/ 2010/PIP_Understanding_the_Participatory_News_Consumer. pdf, 2010.
•  Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.
57	
  
Selected	
  References	
  
Link economy
•  Joseph Turow and Lokman Tsui. The hyperlinked society. The University of Michigan Press, 2008.
•  Juliette De Maeyer. Hyperlinks and journalism: where do they connect? In Proc. Future of Journalism Conference, 2011.
•  Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink
formation in content networks. Management Science, 59(10):2360–2379, 2013.
•  Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012.
•  Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink
formation in content networks. Management Science, 59(10):2360–2379, 2013.
•  Hakan Ceylan, Ioannis Arapakis, Pinar Donmez, and Mounia Lalmas. Automatically embedding newsworthy links to articles. In
Proc. Conference on Information and Knowledge Management, CIKM, pages 1502–1506. ACM, 2012.
Recommendation
•  Richard McCreadie, Craig Macdonald, and Iadh Ounis. News vertical search: when and what to display to users. In Proc.
Conference on Research and Development in Information Retrieval, SIGIR, pages 253–262. ACM, 2013.
•  Samuel Ieong, Mohammad Mahdian, and Sergei Vassilvitskii. Advertising in a stream. In Proc. Conference on World Wide
Web, WWW, pages 29–38. ACM, 2014.
•  Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. A predictive model for advertiser value-per-click in sponsored search. In
Proc. Conference on Information and Knowledge Management, CIKM, pages 1179–1190. ACM, 2013.
•  Narongsak Thongpapanl and Abdul Rehman Ashraf. Enhancing online performance through website content and
personalization. Journal of Computer Information Systems, 52(1):3, 2011.
•  Jian Wang and Yi Zhang. Utilizing marginal net utility for recommendation in e-commerce. In Proc. Conference on Research
and Development in Information Retrieval, SIGIR, pages 1003–1012. ACM, 2011.
•  Joshua Porter. Designing for the social web. Peachpit Press, 2010.
58	
  
Selected	
  References	
  
ATTACHMENT:	
  
IntroducCon	
  
•  “In	
  a	
  world	
  full	
  of	
  choices	
  where	
  the	
  fleeCng	
  aPenCon	
  of	
  
the	
  user	
  becomes	
  a	
  prime	
  resource,	
  it	
  is	
  essenLal	
  that	
  [...]	
  
providers	
  do	
  not	
  just	
  design	
  [websites]	
  but	
  that	
  they	
  
design	
  engaging	
  experiences.”	
  [A}ield].	
  
•  In	
  addiLon	
  to	
  uLlitarian	
  factors,	
  such	
  as	
  usability	
  and	
  
usefulness,	
  we	
  must	
  consider	
  other	
  factors	
  of	
  interacLng	
  
with	
  websites,	
  such	
  as	
  fun,	
  fulfillment,	
  play,	
  and	
  user	
  
engagement.	
  
Successful	
  websites	
  are	
  not	
  just	
  used,	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  
they	
  are	
  engaged	
  with.	
  
•  In	
  order	
  to	
  design	
  engaging	
  websites,	
  it	
  is	
  crucial	
  to	
  
understand	
  what	
  user	
  engagement	
  is	
  and	
  how	
  to	
  
measure	
  it.	
  
IntroducLon	
   60	
  
MoCvaCon	
  
Why	
  is	
  it	
  important	
  to	
  engage	
  users?	
  
Methodology	
  
InteracLon	
  data,	
  online	
  sessions	
  and	
  site	
  visits.	
  
61	
  IntroducLon	
  
t0
t1
t2
t3
t4
t5
t6
t7
session
end
session
start
time
Online session
Browsing activity on Wikipedia
https://ie-mg42.mail.yahoo.com
http://en.wikipedia.org/wiki/Freddie˙Mercury
http://www.bbc.com/news/uk-29149115
http://www.bbc.com/news/uk-england-nottinghamshire-29643802
http://en.wikipedia.org/wiki/Star Wars
http://en.wikipedia.org/wiki/Yoda
http://en.wikipedia.org/wiki/Albert˙Einstein
https://www.facebook.com/janette.lehmann.5
t0
t1
t2
t3
t4
t5
t6
t7
bc0
bc0
bc0
bc0
bc0
bc0
bc0
bc0
BCookie Timestamp URL
-
-
-
http://www.bbc.com/news/uk-29149115
-
http://en.wikipedia.org/wiki/Star Wars
http://en.wikipedia.org/wiki/Yoda
-
ReferrerURL
Interaction data
Page view on Wikipedia Page view on other site
IntroducLon	
   62	
  
Thesis	
  structure	
  
Metrics that account for site
popularity, activity and loyalty
Advertising
Chapter 7
Site engagement
How users experience
ads on desktop and
mobile devices?
Does ad quality
affect the engagement
with the publisher?
How can we identify
high quality ads?
Site engagement
Chapter 4
Multitasking
Chapter 5
Inter-site engagement
Chapter 6
Metrics that account for
traffic between sites
Metrics that account for user
multitasking behaviour
(III+IV)Applications(II)Fund.
Wikipedia
Chapter 8
Site engagement
and multitasking
How users read
articles
in Wikipedia?
Does the activity
of editors align with the
engagement of readers?
How can readers be
valuable for editors?
Yahoo
Chapter 9
Inter-site engagement
How users engage
with a provider
network of sites?
Does the hyperlink
structure affect site and
inter-site engagement?
Online news
Chapter 10+11
Inter-site engagement
How users read
stories across
news providers?
Do hyperlinks to
related content influence
provider engagement?
How can we automatically
detect related content?
Characterising user engagement Comparing site characteristics and user engagement Applications to impact user engagement
ATTACHMENT:	
  
Site	
  engagement	
  
0-1 1-0.5 0.5
Kendall’s tau with p-value < 0.05
('-' insignificant correlations)
Site	
  engagement	
   64	
  
EvaluaCon	
  
CorrelaLons	
  between	
  engagement	
  metrics.	
  
High	
  correlaCons	
  	
  	
  	
  	
  	
  	
  
within	
  metric	
  groups.	
  	
  
	
  
Low	
  correlaCons	
  	
  
between	
  metric	
  groups.	
  
[POP]#Users
[POP]#Visits
[POP]#Clicks
[ACT]PageViewsV
[ACT]DwellTimeV
[LOY]ActiveDays
[LOY]ReturnRate
#Users [POP] 0.82 0.75 - - 0.43 0.34
#Visits [POP] 0.82 0.85 - - 0.60 0.52
#Clicks [POP] 0.75 0.85 0.16 0.18 0.59 0.51
PageViewsV [ACT] - - 0.16 0.33 - -
DwellTimeV [ACT] - - 0.18 0.33 - -
ActiveDays [LOY] 0.43 0.60 0.59 - - 0.79
ReturnRate [LOY] 0.34 0.52 0.51 - - 0.79
0.69
Site	
  engagement	
   65	
  
PaPerns	
  of	
  Site	
  Engagement	
  
Engagement	
  depends	
  on	
  the	
  site	
  at	
  hand.	
  
Games	
  
Not	
  many	
  users,	
  
but	
  they	
  stay	
  long	
  
Search	
  
Users	
  come	
  
frequently,	
  but	
  do	
  
not	
  stay	
  long	
  
Social	
  media	
  
Users	
  come	
  
frequently	
  and	
  
stay	
  long	
  
Shopping	
  
Users	
  do	
  not	
  come	
  
frequently,	
  but	
  
stay	
  long	
  
News	
  
Users	
  come	
  
frequently	
  and	
  	
  
stay	
  long	
  
Service	
  
Users	
  do	
  not	
  come	
  
frequently,	
  but	
  
stay	
  long	
  
ATTACHMENT:	
  
MulCtasking	
  
Online	
  mulLtasking	
   67	
  
MoCvaCon	
  
Users	
  switch	
  between	
  sites,	
  to	
  perform	
  related	
  or	
  totally	
  unrelated	
  tasks.	
  
	
  
	
  
	
  
	
  
	
  
	
  
Switching	
  between	
  tasks	
  (sites)	
  
“…within-­‐session	
  page	
  revisits	
  represent	
  the	
  most	
  common	
  form	
  of	
  revisitaLon,	
  
covering	
  73,54%	
  of	
  all	
  revisits.”	
  [Herder]	
  
	
  
Performing	
  tasks	
  (sites)	
  in	
  parallel	
  using	
  browser	
  tabs	
  
“Most	
  of	
  our	
  parLcipants	
  switched	
  tabs	
  more	
  oken	
  than	
  they	
  used	
  the	
  back	
  
bu;on.”	
  [Dubroy]	
  
[Herder]	
  E.	
  Herder.	
  CharacterizaHons	
  of	
  user	
  web	
  revisit	
  behavior.	
  WWW	
  Workshop	
  ABIS,	
  2005.	
  
[Dubroy]	
  P.	
  Dubroy,	
  R.	
  Balakrishnan.	
  A	
  study	
  of	
  tabbed	
  browsing	
  among	
  mozilla	
  firefox	
  users.	
  SIGCHI,	
  2010.	
  
Online	
  mulLtasking	
   68	
  
Data	
  
Dataset	
  and	
  site	
  categories.	
  
Cat. Subcat. %Sites Description
news
22.1%
news 5.79%
news (soc.) 5.13% society
news (sport) 2.63%
news (enter.) 2.24% music, movies, tv, etc.
news 1.97%
news (life) 1.58% health, housing, etc.
news (tech) 1.58% technology
news (weather) 1.18%
service
15.5%
service 7.63% translators, banks, etc.
mail 3.95%
maps 3.03%
organisation 0.92% bookmarks, calendar, etc.
search
15.3%
search 12.63%
search (special) 1.58% search for lyrics, jobs, etc.
directory 1.05%
sharing
9.6%
blogging 3.55%
knowledge 3.55% collaborative creation and collection of content
sharing 2.50% sharing of videos, etc.
navi
9.3%
front page 6.58%
front page (p.) 1.84% personalised front pages
sitemap 0.92%
leisure
8.7%
adult 2.76%
games 1.97%
social media 1.97%
dating 1.05%
entertainment 0.92% sites with music, tv, etc.support
8.7%
support 1.58% sites that provide products and support for them
download 7.11% downloading software
shopping
7.9%
shopping 4.34%
auctions 2.11%
comparison 1.45% sites to compare prices of products
settings
2.9%
login 1.71%
site settings 1.18% pr e setting, site personalisation
InteracCon	
  data	
  
•  July	
  2012	
  
•  2.5M	
  users	
  
•  785M	
  page	
  views	
  
NavigaCon	
  model	
  
•  We	
  defined	
  a	
  new	
  navigaLon	
  
model	
  (see	
  paper	
  for	
  details)	
  
	
  
Site	
  categories	
  
•  760	
  sites	
  from	
  70	
  countries/
regions	
  
•  11	
  categories	
  
•  33	
  subcategories	
  
Online	
  mulLtasking	
   69	
  
MulCtasking	
  Metrics	
  
CumAct	
  accounts	
  for	
  the	
  acLvity	
  between	
  site	
  visits.	
  
CumulaCve	
  acCvity	
  
The	
  metric	
  is	
  defined	
  as	
  follows:	
  
	
  
	
  
	
  
	
  
	
  
InterpretaCon	
  
High	
  CumAct	
  à	
  High	
  engagement	
  
If	
  users	
  return	
  aker	
  short	
  Lme,	
  they	
  return	
  to	
  
conLnue	
  with	
  same	
  task.	
  
If	
  users	
  return	
  aker	
  longer	
  Lme,	
  they	
  return	
  to	
  
perform	
  a	
  new	
  task	
  –	
  a	
  sign	
  of	
  loyalty.	
  
CumActk = log10 (v1 + ivi
k
•vi
i=2
n
∑ )
Browsing	
  acLvity	
  during	
  the	
  ith	
  visit	
  
Browsing	
  acLvity	
  between	
  the	
  (i-­‐1)th	
  and	
  ith	
  visit	
  
Rescaling	
  factor	
  for	
  ivi	
  
	
  
	
  
k = 3
vi
ivi
1	
   4	
  3	
   10	
   3	
  
CumAct
= log10 (3+13
•4+103
•3)
= 3.48
Site	
  visit	
  
Online	
  mulLtasking	
   70	
  
MulCtasking	
  Metrics	
  
AWRange	
  and	
  AWShik	
  describe	
  changes	
  between	
  the	
  visits.	
  
APenCon	
  shie	
  and	
  range	
  
The	
  metrics	
  is	
  defined	
  as	
  follows:	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
InterpretaCon	
  
AWShik	
  models	
  the	
  shik	
  of	
  a;enLon,	
  and	
  
AWRange	
  models	
  the	
  fluctuaLons	
  
in	
  the	
  browsing	
  acLvity.	
  
AttShiftn =
invn − minInvn
| maxInvn |− | minInvn |
AttRangen =
σ (Vn )
µ(Vn )
Variance	
  in	
  the	
  visit	
  acLvity	
  
Average	
  of	
  the	
  visit	
  acLvity	
  
Number	
  of	
  visits	
  in	
  session	
  	
  
ModificaLon	
  of	
  the	
  “Inversion	
  number”	
  	
  
	
  
n = 4
σ
µi
Inv
0	
   >0	
  
-­‐1	
  
constant	
   decreasing	
  
0	
  
constant	
   complex	
  
+1	
  
constant	
   increasing	
  
AWenHon	
  range	
  
AWenHon	
  shik	
  
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.05
('-' insignificant correlations)
Online	
  mulLtasking	
   71	
  
EvaluaCon	
  
CorrelaLons	
  between	
  mulLtasking	
  and	
  acLvity	
  metrics.	
  
[MT]SessVisits
[MT]SessSites
[MT]CumAct
[MT]AttShift
[MT]AttRange
[ACT]DwellTimeS
SessSites [MT] 0.42
CumAct [MT] 0.41 -
AttShift [MT] 0.09 - -
AttRange [MT] - - -0.38 0.27
DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08
DwellTimeV [ACT] -0.40 - - 0.14 - 0.50
No	
  or	
  only	
  weak	
  
correlaCons	
  between	
  	
  	
  	
  	
  	
  	
  	
  	
  
the	
  metrics.	
  
	
  
All	
  metrics	
  convey	
  
different	
  aspects	
  about	
  
users’	
  online	
  behaviour.	
  
Online	
  mulLtasking	
   72	
  
MulCtasking	
  PaPerns	
  
Cluster	
  centers,	
  site	
  categories	
  and	
  acLvity	
  pa;erns.	
  
CategoriesMultitasking
DwellTimeV CumAct SessVisitsDwellTimeS
sitemap
site settings
news (wheather)
download
+75%
+73%
+69%
+67%
PD
139 sites
Quick task Continuous
multitasking
SessSitesBars from left to right:
111 sites
auctions
shopping
adult
dating
+79%
+71%
+71%
+62%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
PD - Probability difference
Activity
Activity pattern: De - Decreasing In - Increasing Cn - Constant Cm - Complex
De In CmCn De In CmCn
60%
0%
147 sites
Recurring task
search
front page (p.)
front page
organisation
+77%
+62%
+57%
+24%
PD
-1.0
1.0
0.0
De In CmCn
0.6
0.0
137 sites
Focused task
news (tech)
news (life)
support
mail
+66%
+66%
+65%
+64%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
De In CmCn
60%
0%
60%
0%
142 sites
Rapid
multitasking
news (enter.)
knowledge
comparison
service
+64%
+63%
+62%
+59%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
De In CmCn
60%
0%
Single-task-oriented browsing Multitask-oriented browsing
ATTACHMENT:	
  
Inter-­‐site	
  engagement	
  
Inter-­‐site	
  engagement	
   74	
  
Data	
  
Dataset,	
  network	
  and	
  site	
  categories.	
  
InteracCon	
  data	
  
•  August	
  2013	
  to	
  July	
  2014	
  
•  53M	
  sessions	
  
Provider	
  network	
  G=(N,	
  E,	
  λ)	
  
	
  	
  N:	
   	
  155	
  Yahoo	
  sites	
  
	
   	
  from	
  five	
  countries	
  
	
  	
  E: 	
   	
  User	
  traffic	
  
	
  	
  λ(e): 	
  Traffic	
  volume	
  (#Clicks)	
  
	
  
	
  
	
  
	
  
Site	
  categories	
  
•  155	
  sites	
  from	
  5	
  countries	
  
•  5	
  categories	
  
Cat. %Sites Description
35%
19%
13%
23%
10%
news
service
leisure
provider
front page
mail, calendar, etc.
social media, games, etc.
account settings, help, etc.
front pages, site maps
servicefront page news providerleisure
Inter-­‐site	
  engagement	
   75	
  
Inter-­‐site	
  Engagement	
  Metrics	
  
Flow	
  accounts	
  for	
  the	
  extent	
  users	
  navigate	
  between	
  sites.	
  
Traffic	
  Flow	
  
The	
  metric	
  is	
  defined	
  as	
  follows:	
  
	
  
	
  
	
  
	
  
	
  
	
  
InterpretaCon	
  
High	
  Flow	
  à	
  High	
  inter-­‐site	
  engagement	
  
Users	
  navigate	
  oken	
  between	
  the	
  sites	
  of	
  the	
  
network.	
  
Flow =
wi, ji, j∑
vii∑
#Clicks	
  between	
  node	
  i	
  and	
  j	
  
#Visits	
  on	
  node	
  i	
  
wi, j
vi
Flow = 30/60 = 0.5
10
5
20
20
20
10 5
1
1
20
20
20
1 1
Flow = 4/60 = 0.07
Inter-­‐site	
  engagement	
   76	
  
Inter-­‐site	
  Engagement	
  Metrics	
  
Density	
  describes	
  the	
  connecLvity	
  of	
  the	
  network.	
  
Density	
  
We	
  use	
  the	
  density	
  measure	
  	
  of	
  [Wasserman]:	
  
	
  
	
  
	
  
	
  
	
  
InterpretaCon	
  
High	
  Density	
  à	
  High	
  inter-­‐site	
  engagement	
  
Users	
  navigate	
  between	
  many	
  different	
  sites	
  
(inter-­‐site	
  engagement	
  is	
  highly	
  diverse).	
  
[Wasserman]	
  S.	
  Wasserman.	
  Social	
  network	
  analysis:	
  Methods	
  and	
  applicaHons,	
  1994.	
  
Density =
# Edges
# Possible_ Edges
Density = 4/6 = 0.7
Flow = 2/6 = 0.3
Inter-­‐site	
  engagement	
   77	
  
Inter-­‐site	
  Engagement	
  Metrics	
  
Reciprocity	
  measures	
  the	
  homogeneity	
  of	
  traffic	
  between	
  two	
  sites.	
  
Reciprocity	
  
We	
  use	
  the	
  reciprocity	
  measure	
  of	
  [SquarLni]:	
  
	
  
	
  
	
  
	
  
	
  
	
  
InterpretaCon	
  
High	
  Reciprocity	
  à	
  High	
  inter-­‐site	
  engagement	
  
Users	
  navigate	
  between	
  two	
  sites	
  in	
  both	
  direcLons	
  
(inter-­‐site	
  engagement	
  is	
  highly	
  homogenious).	
  
[SquarHni]	
  T.	
  SquarHni,	
  F.	
  Picciolo,	
  F.	
  RuzzenenH,	
  and	
  D.	
  Garlaschelli.	
  Reciprocity	
  of	
  weighted	
  networks.	
  Nature:	
  ScienHfic	
  reports,	
  2013.	
  
#Clicks	
  between	
  node	
  i	
  and	
  j	
  wi, j
RP =
min[wi, j,wj,i ]
i<j∑
wi, ji≠j∑
1
10 5
20
1
Reciprocity = 15/50 = 0.3
Reciprocity = 2/37 = 0.05
10
10 5
20
5
Inter-­‐site	
  engagement	
   78	
  
Inter-­‐site	
  Engagement	
  Metrics	
  
Entry/ExitDisp	
  measures	
  how	
  the	
  traffic	
  to/from	
  the	
  network	
  is	
  distributed	
  over	
  the	
  sites.	
  
Entry	
  disparity	
  and	
  exit	
  disparity	
  
We	
  use	
  the	
  group	
  degree	
  measure	
  of	
  [Freeman]	
  and	
  
adapt	
  it	
  as	
  follows:	
  
	
  
	
  
	
  
	
  
	
  
	
  
	
  
InterpretaCon	
  
High	
  Entry/ExitDisp	
  à	
  Low	
  inter-­‐site	
  engagement	
  
The	
  network	
  is	
  more	
  vulnerable	
  to	
  outages,	
  because	
  
only	
  few	
  sites	
  are	
  used	
  to	
  enter	
  (leave)	
  the	
  network.	
  
EntryDisp =
(gin
max − gin
i )
i∑
| N |• gin
ii∑
[Freeman]	
  L.	
  C	
  Freeman.	
  Centrality	
  in	
  social	
  networks	
  conceptual	
  clarificaHon.	
  Social	
  networks,	
  1979.	
  
Number	
  of	
  visits	
  that	
  started	
  at	
  node	
  ni	
  
(user	
  entered	
  the	
  network)	
  
Maximum	
  value	
  of	
  gin	
  
Number	
  of	
  nodes	
  | N |
gi
in
gin
max
EntryDisp = 20/3 40 = 0.17
20
10
10
40
5
5
EntryDisp = 70/3 50 = 0.47
Inter-­‐site	
  engagement	
   79	
  
EvaluaCon:	
  Network-­‐level	
  
CorrelaLons	
  between	
  inter-­‐site	
  and	
  network	
  engagement	
  metrics.	
  
[IS]Density
[IS]Reciprocity
[IS]EntryDisparity
[IS]ExitDisparity
[POP]#Sessions
[ACT]DwellTimeS
[ACT]#Sites
Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65
Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25
Reciprocity [IS] -0.38 -0.32 0.42 - 0.25
EntryDisparity [IS] 0.84 -0.54 0.33 -
ExitDisparity [IS] -0.55 0.38 0.20
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
Density	
  and	
  #Sessions	
  
The	
  more	
  users	
  are	
  
visiCng	
  the	
  network,	
  the	
  
more	
  diverse	
  is	
  the	
  inter-­‐
site	
  engagement.	
  
	
  
Entry-­‐	
  and	
  ExitDisparity	
  
Volume	
  of	
  in-­‐	
  and	
  out-­‐
going	
  traffic	
  of	
  the	
  nodes	
  
depend	
  on	
  each	
  other.	
  
	
  
Flow	
  and	
  #Sites	
  
The	
  more	
  sites	
  are	
  visited	
  
during	
  a	
  session,	
  the	
  
higher	
  the	
  flow	
  of	
  traffic.	
  
Inter-­‐site	
  engagement	
   80	
  
EvaluaCon:	
  Node-­‐level	
  
CorrelaLons	
  between	
  inter-­‐site	
  and	
  site	
  engagement	
  metrics.	
  
[IS]Downstream
[IS]EntryProb
[IS]ExitProb
[POP]#Sessions
[ACT]DwellTimeS
[MT]#Visits
[MT]CumAct
PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31
Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02
EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35
ExitProb [IS] 0.08 -0.18 0.18 0.32
0-1 1-0.5 0.5
Spearman’s rho with p-value < 0.01
('-' insignificant correlations)
PageRank	
  and	
  #Sessions	
  
Popular	
  sites	
  in	
  the	
  
provider	
  network,	
  are	
  
also	
  visited	
  frequently	
  
when	
  browsing	
  through	
  
the	
  network.	
  
	
  
Entry-­‐	
  and	
  ExitProb	
  
Nodes	
  that	
  are	
  used	
  to	
  
enter	
  the	
  network	
  are	
  
also	
  frequently	
  used	
  to	
  
exit	
  the	
  network.	
  
Inter-­‐site	
  engagement	
   81	
  
Comparing	
  Provider	
  Networks	
  
Country2
Country1
Country3
Country4
Country5
Flow Reciprocity EntryDisparityDensity DwellTimeBars from left to right:
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
Inter-­‐site	
  engagement	
   82	
  
PaPerns	
  of	
  Inter-­‐site	
  Engagement	
  CategoriesEngagement
PageRank EntryProb DwellTimeDownstream PD - Probability difference
46 sites
Focused eng.
front page
service
provider
leisure
news
+80%
+63%
-100%
-100%
-100%
PD
23 sites
Traffic hub
46 sites
Supporter
40 sites
Shared eng.
CumActBars from left to right:
-1.0
1.0
0.0
provider
service
news
front page
leisure
+31%
+19%
-2%
-10%
-100%
PD
leisure
provider
service
news
front page
+67%
+48%
-21%
-94%
-100%
PD
news
leisure
provider
front page
service
+63%
-61%
-100%
-100%
-100%
PD
-1.0
1.0
0.0
-1.0
1.0
0.0
-1.0
1.0
0.0
ATTACHMENT:	
  
NaCve	
  AdverCsing	
  
NaLve	
  AdverLsing	
   84	
  
Effect	
  on	
  User	
  Engagement	
  
0%
200%
400%
600%
short ad clicks long ad clicks
adclickdifference
short ad clicks long ad clicks
clicksperdaydifference
PosiLve	
  experience	
  has	
  a	
  strong	
  effect	
  on	
  
users	
  clicking	
  on	
  ads	
  again,	
  and	
  a	
  small	
  
effect	
  on	
  user	
  engagement	
  with	
  the	
  
stream.	
  	
  
	
  
NaLve	
  AdverLsing	
   85	
  
Mobile	
  vs.	
  Desktop	
  
Ad	
  post-­‐click	
  experience	
  between	
  mobile	
  
and	
  desktop	
  differs.	
  
For	
  dwell	
  Lme	
  we	
  obtain	
  rho	
  =	
  0.50;	
  this	
  
value	
  is	
  even	
  smaller	
  for	
  bounce	
  rate	
  with	
  
rho	
  =	
  0.23.	
  	
  
0.00
0.05
0.10
0.15
dwell time difference
p(dwelltimedifference)
higher on mobilehigher on desktop
0.00
0.05
0.10
0.15
bounce rate difference
p(bounceratedifference)
higher on mobilehigher on desktop
NaLve	
  AdverLsing	
   86	
  
Mobile	
  OpCmised	
  Landing	
  Pages	
  
Dwell	
  Cme:	
  The	
  distribuLon	
  is	
  very	
  similar	
  for	
  
both	
  groups.	
  
	
  
Bounce	
  rate:	
  Decreases	
  by	
  6.9%	
  (median	
  
decreases	
  by	
  30.4%)	
  for	
  Opt	
  landing	
  pages	
  but	
  
increases	
  by	
  13.4%	
  (median	
  decreases	
  by	
  11.5%)	
  
for	
  Npt	
  landing	
  pages.	
   not mobile optimized mobile optimized
0.0
0.1
0.2
0.3
dwell time difference
p(dwelltimedifference)
higher on mobilehigher on desktop
mobile opt.
not mobile opt.
0.0
0.1
0.2
bounce rate difference
p(bounceratedifference)
higher on mobilehigher on desktop
mobile opt.
not mobile opt.
ATTACHMENT:	
  
Wikipedia	
  
Wikipedia	
   88	
  
Wikipedia	
  Research	
  
Literature	
  review	
  by	
  Okoli	
  et	
  al.:	
  The	
  people’s	
  encyclopedia	
  under	
  the	
  gaze	
  of	
  the	
  sages:	
  A	
  
systemaLc	
  review	
  of	
  scholarly	
  research	
  on	
  wikipedia.	
  	
  
	
  
Wikipedia	
   89	
  
Reading	
  Preferences	
  
Popularitylow high
ArticleLengthshortlong
borderline casesII I
III IV
Jeanne Tsai
Douglas Adams
Luis
Palomino
Anne Stears
Peter
Ehrlich
Alec
Mango
Stephen D.
Lovejoy
1st
Dalai Lama
Dexter Jackson
(safety)
Katie Green
Brittany
Borman
Anthony Anenih
Ronnie Bird
Jan Anderson
(scientist)
Fitch
Robertson
Sean Bennett
For 4.2% (group IV) of the articles
editing activity is low, but reading activity is high.!
Wikipedia	
   90	
  
Reading	
  PaPerns	
  Article
topic
Reading
behavior
ArticleViewsa
SessionArticlesa
Popularitya
ReadingTimea
CA - Percentage in topic
4,826 articles
11,579 behavior vectors
sportsperson
musician
media pers.
28%
26%
23%
CA
Exploration
artist/writer
historical fig.
polit./businessp.
43%
41%
37%
CA
5,278 articles
10,605 behavior vectors
Focus
3,876 articles
14,267 behavior vectors
historical fig.
criminal/victim
musican
42%
38%
38%
CA
Trending
5,684 articles
13,470 behavior vectors
media pers.
sportsperson
musician
27%
27%
19%
CA
Passing
28K [16K,51K]
11 [5,23]
7.7%
38K [21K,69K]
20 [9,41]
16.9%
26K [15K,45K]
10 [5,21]
10.5%
16K [10K,27K]
8 [3,18]
5.1%
ArtLen
#Edits
%HQA
#Edits - Number of edits
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
-1.0
0.5
-0.5
0.0
1.0
%HQA - Percentage of high quality articlesArtLen - Article length
Wikipedia	
   91	
  
Reading	
  PaPerns	
  over	
  Time	
  
Stability	
  
•  30%	
  of	
  the	
  arLcles	
  are	
  popular	
  in	
  1	
  month	
  
•  10%	
  are	
  popular	
  over	
  the	
  whole	
  13-­‐months	
  
•  Almost	
  all	
  arLcles	
  have	
  one	
  reading	
  pa;ern	
  
half	
  of	
  their	
  life	
  Lme	
  
TransiCons	
  
•  TransiLons	
  are	
  temporary	
  –	
  arLcles	
  move	
  
temporarily	
  to	
  another	
  cluster	
  
•  High	
  reciprocity	
  –	
  similar	
  number	
  of	
  
transiLons	
  in	
  both	
  direcLons	
  
•  “Focus”	
  cluster	
  is	
  isolated	
  -­‐	
  ArLcles	
  in	
  that	
  
cluster	
  are	
  the	
  most	
  stable	
  ones	
  
•  Strong	
  connecLon	
  between	
  the	
  “Passing”,	
  
“ExploraLon”,	
  and	
  “Trending”	
  clusters	
  –	
  
many	
  arLcles	
  adopt	
  all	
  three	
  pa;erns	
  
ATTACHMENT:	
  
Yahoo	
  
93	
  
Upstream	
  Traffic	
  
TeleportaCon	
   Social	
  media	
  /	
  News	
   Search	
  /	
  Ext-­‐Yahoo	
  
Users	
  engage	
  (quickly)	
  
to	
  many	
  sites.	
  
Users	
  conHnue	
  with	
  
same	
  acHvity	
  inside	
  
the	
  provider	
  network.	
  
Users	
  visit	
  site	
  they	
  are	
  
interested	
  in,	
  perform	
  a	
  
quick	
  task,	
  and	
  leave.	
  
Network	
  
DwellTime	
  
-­‐-­‐	
   ++	
   -­‐-­‐	
  
Traffic	
  
DistribuCon	
  
++	
   -­‐-­‐	
   -­‐-­‐	
  
Entry	
  
Disparity	
  
-­‐-­‐	
  
Yahoo	
  
Users	
  engage	
  differently	
  depending	
  on	
  
where	
  they	
  are	
  coming	
  from.	
  
94	
  
Network	
  Effect	
  PaPern	
  
Yahoo	
  
Sites	
  change	
  their	
  popularity	
  in	
  the	
  same	
  way.	
  
Ac>vity	
  (dwell	
  >me)	
  on	
  a	
  site	
  depends	
  more	
  on	
  the	
  site	
  itself,	
  
but	
  there	
  are	
  some	
  nega>ve	
  dependencies.	
  
Pattern
examples
41 patterns
Simple star-like
6 patterns
Complex star-like
1 pattern
Cluster-like
3.00 [3.00,4.00]
0.67 [0.00,0.89]
0 [0,0]
8.00 [7.00,18.00]
0.76 [0.56,0.84]
0 [0,0]
52
0.91
0.51
N
Recip
Trans
N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure
(4) (5) (6)(1) (2) (3)
95	
  
Hyperlink	
  Performance	
  
Yahoo	
  
0%
25%
50%
75%
100%
Onsitelinks
front page providerservice news leisure
Intersitelinks
front page providerservice news leisure
0%
20%
40%
60%
80%
Externallinks
front page providerservice news leisure
20%
40%
60%
(a) PageRank and downstream.
Traffic
PageRank Downstream
Hyperlinks
PageRank 0.54 -
Downstream - -
(b) On-site, inter-site, and external.
Traffic
On-site Inter-site External
Hyperlinks
On-site 0.54 -0.45 -0.38
Inter-site -0.40 0.50 -
External - - 0.39
ATTACHMENT:	
  
Online	
  News	
  
Online	
  news	
   97	
  
Focused	
  versus	
  Non-­‐focused	
  Sessions	
  
Internal	
  
Non-focused sessionsFocused sessions●
(b) Duration
(d) p(focused session)
(a) %Sessions
(f) Flow
25
15
5
60%
20%
0.6
0.2
0.2
0.1
2 3 4 5 6 7 7 2 3 4 5 6 7 7 2 3 4 5 6 7 7
#Articles #Articles #Articles
●
●
●
●
●
●
●
(c) #Providers
2.5
2.0
1.5
●
●
●
●
●
●
●
●●
●●
●●
●● ●●
●●
●●
(e) EntryDisparity
0.5
0.3
0.1 ●
●
●
●
●
●
●
When	
  users	
  focus	
  on	
  a	
  news	
  story,	
  they	
  spend	
  more	
  >me	
  reading	
  the	
  
ar>cles	
  and	
  the	
  inter-­‐site	
  engagement	
  between	
  providers	
  is	
  higher.	
  
Online	
  news	
   98	
  
Hyperlink	
  Performance	
  
	
  
Number	
  of	
  Inline	
  Links	
  
•  <10	
  links	
  may	
  be	
  wasLng	
  an	
  opportunity	
  
•  10-­‐29	
  links	
  does	
  not	
  result	
  in	
  more	
  clicks	
  
•  >29	
  links	
  may	
  harm	
  the	
  user	
  experience	
  
	
  
	
  
PosiCon	
  of	
  Inline	
  Links	
  
•  30%	
  at	
  the	
  end,	
  16%	
  at	
  the	
  beginning,	
  46%	
  
are	
  distributed	
  within	
  the	
  arLcle	
  text.	
  
•  Performance	
  of	
  links	
  located	
  at	
  the	
  
beginning	
  of	
  the	
  text	
  is	
  very	
  low	
  (-­‐28%)	
  
•  Best	
  performance	
  is	
  achieved	
  with	
  links	
  at	
  
the	
  end	
  of	
  the	
  arLcle	
  text	
  (+35%)	
  
Link popularity● Link performance
Position in article text
Linkpopularity [0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0]
10%
20%
30%
-0.2
0.0
0.2
Linkperformance
●●
●●
●●
●● ●● ●● ●● ●●
●●
●●
●●
●●
●● ●● ●● ●● ●● ●●
●●
●●
Number of inline links in article
Clicksperlink
0.0
0.2
0.4
0.6
[0,2] [9,11] [18,20] [27,29] [36,38]
Number of inline links in article
Numberofclicks
[0,2] [9,11] [18,20] [27,29] [36,38]
2.5
5.0
7.5
Online	
  news	
   99	
  
Effect	
  on	
  User	
  Engagement	
  
Internal	
  Focused	
  
	
  
Short-­‐term:	
  Only	
  3	
  (out	
  of	
  50)	
  providers	
  
have	
  their	
  corresponding	
  average	
  dwell	
  Lme	
  
lower	
  for	
  the	
  story-­‐focused	
  provider	
  
sessions.	
  The	
  average	
  increase	
  in	
  dwell	
  Lme	
  
from	
  non-­‐story-­‐focused	
  to	
  story-­‐focused	
  
provider	
  sessions	
  is	
  50%.	
  	
  
	
  
Long-­‐term:	
  For	
  78%	
  of	
  the	
  providers,	
  we	
  
find	
  that	
  there	
  are	
  more	
  users	
  that	
  return	
  
earlier	
  aker	
  they	
  have	
  a	
  story-­‐focused	
  
provider	
  session.	
  	
  
	
  
Internal	
  
News provider
Dwelltimepersession
Non-focused Focused Ext-focused
News provider
p(absence12h)
Non-focused Focused Ext-focused
Online	
  news	
   100	
  
Effect	
  on	
  User	
  Engagement	
  
External	
  Focused	
  
	
  
Short-­‐term:	
  We	
  do	
  not	
  observe	
  an	
  effect	
  on	
  
the	
  dwell	
  Lme	
  (neither	
  posiLve	
  nor	
  
negaLve).	
  The	
  average	
  increase	
  is	
  only	
  5.5%,	
  
and	
  based	
  on	
  the	
  K-­‐S	
  test	
  we	
  cannot	
  confirm	
  
that	
  the	
  distribuLons	
  are	
  different	
  (p-­‐
value=0.36).	
  	
  
	
  
Long-­‐term:	
  For	
  70%	
  of	
  these	
  news	
  sites,	
  the	
  
probability	
  that	
  users	
  return	
  within	
  the	
  
following	
  12	
  hours	
  increases	
  (the	
  average	
  
increase	
  is	
  76%).	
  	
  
	
  
	
  
External	
  
News provider
Dwelltimepersession
Non-focused Focused Ext-focused
News provider
p(absence12h)
Non-focused Focused Ext-focused
Online	
  news	
   101	
  
Discovering	
  Story-­‐related	
  Content	
  in	
  TwiPer	
  

More Related Content

Similar to From Site to Inter-site User Engagement

Tutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerceTutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerceMounia Lalmas-Roelleke
 
Know and Delight Your Users: UX Analytics
Know and Delight Your Users: UX AnalyticsKnow and Delight Your Users: UX Analytics
Know and Delight Your Users: UX AnalyticsCemal Buyukgokcesu
 
Google Analytics Beginners: How to Measure Your Homepage Design
Google Analytics Beginners: How to Measure Your Homepage DesignGoogle Analytics Beginners: How to Measure Your Homepage Design
Google Analytics Beginners: How to Measure Your Homepage DesignBop Design
 
NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...
NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...
NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...NYCUXPA
 
Networked user engagement
Networked user engagementNetworked user engagement
Networked user engagementJanette Lehmann
 
An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...Mounia Lalmas-Roelleke
 
How Big Data is Changing User Engagement
How Big Data is Changing User EngagementHow Big Data is Changing User Engagement
How Big Data is Changing User EngagementMounia Lalmas-Roelleke
 
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptxPUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptxKevinKregelCampollo1
 
Qualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca Grivet
Qualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca GrivetQualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca Grivet
Qualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca Grivethypergriso
 
Peak Usability Seminar - ROI of Usability
Peak Usability Seminar - ROI of UsabilityPeak Usability Seminar - ROI of Usability
Peak Usability Seminar - ROI of UsabilityPeak Usability
 
Digital Marketing Course Week 4: Digital Analytics
Digital Marketing Course Week 4: Digital AnalyticsDigital Marketing Course Week 4: Digital Analytics
Digital Marketing Course Week 4: Digital AnalyticsAyca Turhan
 
Continuous Improvement
Continuous ImprovementContinuous Improvement
Continuous ImprovementReading Room
 
UX Analytics Cemal Büyükgökçesu
UX Analytics Cemal BüyükgökçesuUX Analytics Cemal Büyükgökçesu
UX Analytics Cemal BüyükgökçesuUserspots
 
Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12Tim Frick
 
User Engagement: from Sites to a Network of Sites or The Network Effect Matters!
User Engagement: from Sites to a Network of Sites or The Network Effect Matters!User Engagement: from Sites to a Network of Sites or The Network Effect Matters!
User Engagement: from Sites to a Network of Sites or The Network Effect Matters!Mounia Lalmas-Roelleke
 
How to Run a Data Driven Product Dev Organization by Skedulo CPM
How to Run a Data Driven Product Dev Organization by Skedulo CPMHow to Run a Data Driven Product Dev Organization by Skedulo CPM
How to Run a Data Driven Product Dev Organization by Skedulo CPMProduct School
 
Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12Mightybytes
 

Similar to From Site to Inter-site User Engagement (20)

analytics_def
analytics_defanalytics_def
analytics_def
 
An engaging click
An engaging clickAn engaging click
An engaging click
 
Tutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerceTutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerce
 
Know and Delight Your Users: UX Analytics
Know and Delight Your Users: UX AnalyticsKnow and Delight Your Users: UX Analytics
Know and Delight Your Users: UX Analytics
 
Google Analytics Beginners: How to Measure Your Homepage Design
Google Analytics Beginners: How to Measure Your Homepage DesignGoogle Analytics Beginners: How to Measure Your Homepage Design
Google Analytics Beginners: How to Measure Your Homepage Design
 
NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...
NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...
NYC UXPA: 2014 - Bringing Together User Experience and Web Analytics (Michael...
 
Networked user engagement
Networked user engagementNetworked user engagement
Networked user engagement
 
An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...
 
How Big Data is Changing User Engagement
How Big Data is Changing User EngagementHow Big Data is Changing User Engagement
How Big Data is Changing User Engagement
 
The Role of UX in SEO
The Role of UX in SEOThe Role of UX in SEO
The Role of UX in SEO
 
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptxPUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
PUBLISHING-AND-SUSTAINING-ICT-PROGECTS-FOR-SOCIAL-CHANGE-2.pptx
 
Qualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca Grivet
Qualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca GrivetQualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca Grivet
Qualche spunto sul misurare i social network - Barcamp Torino 2008 - Luca Grivet
 
Peak Usability Seminar - ROI of Usability
Peak Usability Seminar - ROI of UsabilityPeak Usability Seminar - ROI of Usability
Peak Usability Seminar - ROI of Usability
 
Digital Marketing Course Week 4: Digital Analytics
Digital Marketing Course Week 4: Digital AnalyticsDigital Marketing Course Week 4: Digital Analytics
Digital Marketing Course Week 4: Digital Analytics
 
Continuous Improvement
Continuous ImprovementContinuous Improvement
Continuous Improvement
 
UX Analytics Cemal Büyükgökçesu
UX Analytics Cemal BüyükgökçesuUX Analytics Cemal Büyükgökçesu
UX Analytics Cemal Büyükgökçesu
 
Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12
 
User Engagement: from Sites to a Network of Sites or The Network Effect Matters!
User Engagement: from Sites to a Network of Sites or The Network Effect Matters!User Engagement: from Sites to a Network of Sites or The Network Effect Matters!
User Engagement: from Sites to a Network of Sites or The Network Effect Matters!
 
How to Run a Data Driven Product Dev Organization by Skedulo CPM
How to Run a Data Driven Product Dev Organization by Skedulo CPMHow to Run a Data Driven Product Dev Organization by Skedulo CPM
How to Run a Data Driven Product Dev Organization by Skedulo CPM
 
Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12Content Jam Measure What Matters 06/14/12
Content Jam Measure What Matters 06/14/12
 

More from Janette Lehmann

Dynamical Classes of Collective Attention in Twitter
Dynamical Classes of Collective Attention in TwitterDynamical Classes of Collective Attention in Twitter
Dynamical Classes of Collective Attention in TwitterJanette Lehmann
 
User Engagement - A scientific Challenge
User Engagement - A scientific ChallengeUser Engagement - A scientific Challenge
User Engagement - A scientific ChallengeJanette Lehmann
 
Reading Preference and Behavior on Wikipedia
Reading Preference and Behavior on WikipediaReading Preference and Behavior on Wikipedia
Reading Preference and Behavior on WikipediaJanette Lehmann
 
Temporal Variations in Networked User Engagement
Temporal Variations in Networked User EngagementTemporal Variations in Networked User Engagement
Temporal Variations in Networked User EngagementJanette Lehmann
 
Finding News Curators in Twitter
Finding News Curators in TwitterFinding News Curators in Twitter
Finding News Curators in TwitterJanette Lehmann
 
Models of user engagement
Models of user engagementModels of user engagement
Models of user engagementJanette Lehmann
 
From site to networked engagement (Keynote)
From site to networked engagement (Keynote)From site to networked engagement (Keynote)
From site to networked engagement (Keynote)Janette Lehmann
 

More from Janette Lehmann (7)

Dynamical Classes of Collective Attention in Twitter
Dynamical Classes of Collective Attention in TwitterDynamical Classes of Collective Attention in Twitter
Dynamical Classes of Collective Attention in Twitter
 
User Engagement - A scientific Challenge
User Engagement - A scientific ChallengeUser Engagement - A scientific Challenge
User Engagement - A scientific Challenge
 
Reading Preference and Behavior on Wikipedia
Reading Preference and Behavior on WikipediaReading Preference and Behavior on Wikipedia
Reading Preference and Behavior on Wikipedia
 
Temporal Variations in Networked User Engagement
Temporal Variations in Networked User EngagementTemporal Variations in Networked User Engagement
Temporal Variations in Networked User Engagement
 
Finding News Curators in Twitter
Finding News Curators in TwitterFinding News Curators in Twitter
Finding News Curators in Twitter
 
Models of user engagement
Models of user engagementModels of user engagement
Models of user engagement
 
From site to networked engagement (Keynote)
From site to networked engagement (Keynote)From site to networked engagement (Keynote)
From site to networked engagement (Keynote)
 

Recently uploaded

TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxFinatron037
 
Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsThinkInnovation
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxdhiyaneswaranv1
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 

Recently uploaded (16)

TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
Rock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptxRock Songs common codes and conventions.pptx
Rock Songs common codes and conventions.pptx
 
Optimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in LogisticsOptimal Decision Making - Cost Reduction in Logistics
Optimal Decision Making - Cost Reduction in Logistics
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptxCCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
CCS336-Cloud-Services-Management-Lecture-Notes-1.pptx
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 

From Site to Inter-site User Engagement

  • 1. photo  credit  donsolo,  CC  BY-­‐NC-­‐SA  2.0   From  Site  to  Inter-­‐site  User  Engagement   Jane;e  Lehmann   Barcelona,  February  26,  2015     Advisors:  Ricardo  Baeza-­‐Yates   Co-­‐Advisor:  Mounia  Lalmas  
  • 2. •  User  engagement  is  a  quality  of  the  user  experience  that  emphasizes  the   posiLve  aspects  of  interacLon  with  a  website  –  in  parLcular  the  fact  of  being   capLvated  by  the  website.   •  In-­‐the-­‐moment  engagement                                                                                                                                                                                                     Users  stay  on  a  website  over  a  long  Lme.   •  Long-­‐term  engagement                                                                                                                                                                                                                   Users  come  back  frequently  and                                                                                                                                                                                             over  a  long-­‐term.   IntroducLon   2   User  Engagement   DefiniLon   Successful  websites  are  not  just  used,                                                                           they  are  engaged  with.  
  • 3. User  Engagement   Measuring   3  IntroducLon   Before  we  can  design  engaging  websites,   it  is  crucial  that  we  are  able  to  measure  engagement.   “If  you  can  measure  it,  you  can  improve  it.”   Sir  William  Thomson   Analysis/Planning   Design  Changes  Measuring  
  • 4. Main  Research  Goals   4  IntroducLon   Primary  goal    Can  we  define  new  engagement  metrics  that     Measuring      enhance  our  understanding  of  engagement?                 Secondary  goal    Can  we  idenLfy  ways  to  influence  engagement? Analysis/Planning       Analysis/Planning   Design  Changes  Measuring  
  • 5. IntroducLon   5   Analysis/ Planning   Design  Changes  Measuring  Online  mulLtasking   Inter-­‐site   engagement   Site  engagement   Effect  of  providing   off-­‐site  content   Effect  of  hyperlinks  
  • 6. IntroducLon   6   Analysis/ Planning   Design  Changes  Measuring   Site  engagement  
  • 7. Measuring  Engagement   InteracLon  data   7  Site  engagement   Data   Browsing  events  provided  by  Yahoo  toolbar  (client-­‐side).   Engagement   Analysing  the  data  using  online  behaviour  metrics.       Online  session:   Visit  on  Yahoo  News  
  • 8. Site  engagement   8   Measuring  Engagement   Online  behaviour  metrics   K.  Rodden,  H.  Hutchinson,  X.  Fu.  Measuring  the  user  experience  on  a  large  scale:  User-­‐centered  metrics  for  web  applicaHons.  CHI,  2010.   E.  Peterson,  J.  Carrabis.  Measuring  the  immeasurable:  Visitor  engagement.  Web  AnalyHcs  DemysHfied,  2008.   B.  Haven,  S.  ViWal.  Measuring  engagement.  Forrester  Research,  2008.   B.  Weischedel  and  E.  Huizingh.  Website  opHmizaHon  with  web  metrics:  A  case  study.  Conference  on  Electronic  commerce,  2006.  
  • 9. Site  engagement   9   Measuring  Engagement   Online  behaviour  metrics   Popularity   #Users   Number  of  users.   #Visits   Number  of  visits.   #Clicks   Number  of  clicks.   AcCvity  (within  a  visit)                                                                                In-­‐the-­‐moment  engagement   PageViews   Avg.  number  of  page  views  per  visit.   DwellTime   Avg.  Lme  on  site  per  visit.   Loyalty  (across  visits)                                                                                                        Long-­‐term  engagement   ReturnRate   Number  of  Lmes  a  user  visited  the  site.   AcLveDays   Number  of  days  a  user  visited  the  site.  
  • 10. Site  engagement   10   Measuring  Engagement   Differences  in  engagement   ComScore,  Alexa,   GoogleAnalyHcs,…   Shopping   Users  do  not  come   frequently,  but   stay  long   Games   Not  many  users,   but  they  stay  long   News   Users  come   frequently  and     stay  long  
  • 11. Measuring  Engagement   Problem   11  Site  engagement   Isolated  view:  The  metrics  focus   on  engagement  with  a  single  site.   RelaLonships  to  other  sites  are   not  considered.  
  • 12. IntroducLon   12   Analysis/ Planning   Design  Changes  Measuring  Online  mulLtasking   Site  engagement  
  • 13. Online  mulLtasking   13   MoCvaCon   In-­‐the-­‐moment  engagement   ComScore,  Alexa,   GoogleAnalyHcs,…   What  web  analyCcs  think  we  do…   1  visit  with  4  page  views.  
  • 14. Online  mulLtasking   14   MoCvaCon   In-­‐the-­‐moment  engagement   ComScore,  Alexa,   GoogleAnalyHcs,…   …  and  what  we  really  do:   3  visit  with  on  average  1.3  page  views.  
  • 15. Online  mulLtasking   15   MoCvaCon   Online  mulLtasking.   Problem     •  Engagement  metrics  do  not  capture  such  behaviour.     •  Measuring  acLvity  on  a  site  can  lead  to  incorrect  conclusions.     Online  mulCtasking   Users  visit  several  sites  and  switch  between  them   during  an  online  session,  to  perform  related  or  totally   unrelated  tasks.  
  • 16. Research  QuesCon   16  Online  mulLtasking   How  can  we  measure  engagement  by   accounLng  for  user  mulLtasking  behaviour?   Analysis/Planning   Design  Changes  Measuring  
  • 17. Extent  of  mulCtasking   •  10.2  disLnct  sites,  2  visits  per  site.     Absence  Cme   •  50%  of  sites  are  revisited  aker  <  1min.    InterrupHon  of  a  task     •  There  are  revisits  aker  long  breaks.    Performing  a  new  task   Online  mulLtasking   17   Online  MulCtasking   CharacterisLcs   0.00 0.25 0.50 0.75 1.00 10 2 10 1 10 0 10 1 10 2 Cumulativeprobability Absence time [min] news (finance) news (tech) social media mail 2.09 1.76 2.28 2.09 #Visits Absence time [min] 3.85 3.95 4.47 6.86 Absence time: Time between two visits
  • 18. AcCvity  paPerns     •  Four  types:  Decreasing,  increasing,  constant,  complex.   •  Successive  visits  can  belong  together  (i.e.  to  the  same  task).   •  Complex  cases  refer  to  no  specific  pa;ern  or  repeated  pa;ern.     Online  mulLtasking   18   Online  MulCtasking   CharacterisLcs   1 2 3 4 ith visit on site 1 2 3 4 ith visit on site 1 2 3 4 ith visit on site 1 2 3 4 ith visit on site Proportionoftotal dwelltimeonsite 0.23 0.28 0.33 p-value = 0.09 m = -0.01 p-value = 0.07 m = -0.02 p-value = 0.79 m = 0.00 news (finance) sitesmail sites social media sites news (tech) sites decreasing attention increasing attention constant attention complex attention
  • 19. Online  mulLtasking   19   Measuring  Engagement   Online  mulLtasking  metrics   Extent  of  mulCtasking   SessSites   Total  number  of  sites  accessed  (#tasks).   SessVisits   Number  of  visits  to  site  (site  switching).   Absence  Cme   CumAct   Aggregates  the  dwell  Lmes  of  the  visits  with   accounLng  for  the  Lme  between  the  visits.   AcCvity  paPern   A;Shik   A;Range   Describe  the  four  cases  of  a;enLon  shiks.  
  • 20. 20   CASE  STUDY:   MulCtasking  PaPerns     •  ObjecCve:  Analyse  mulLtasking  acLvity  on  sites;   idenLfy  mulLtasking  pa;erns  (clustering).   •  Metrics:  Site  DwellTime,  MulLtasking  metrics.   •  Data:  July  2012,  2.5M  users,  760  sites  (shopping,   news,  search,  etc.).  
  • 21. 21   Case  Study:  MulCtasking  PaPerns   Results   No  mulCtasking   MulCtasking   Quick   Focused   Rapid   ConCnuous   Recurring   Checking   weather   Reading   mails   Following  link  to   off-­‐site  content   Purchasing   an  item   Performing   search   Site   DwellTime   -­‐-­‐   ++   ++   ++   -­‐-­‐   Extent  of   mulCtasking   -­‐-­‐   -­‐-­‐   ++   ++   ++   Absence   Cme   -­‐-­‐   ++   ++   ImplicaCons   Provide   interesHng  off-­‐ site  content   Shopping   takes  more  than   one  visit   Support  user   by  finishing   tasks  quickly   Online  mulLtasking   -- low value ++ high value
  • 22. 22   Case  Study:  MulCtasking  PaPerns   Results   No  mulCtasking   MulCtasking   Quick   Focused   Rapid   ConCnuous   Recurring   Checking   weather   Reading   mails   Following  link  to   off-­‐site  content   Purchasing   an  item   Performing   search   Site   DwellTime   -­‐-­‐   ++   ++   ++   -­‐-­‐   Extent  of   mulCtasking   -­‐-­‐   -­‐-­‐   ++   ++   ++   Absence   Cme   -­‐-­‐   ++   ++   AcCvity   paPern   Online  mulLtasking   De In CmCn 60% 0% De In CmCn 60% 0% De In CmCn 60% 0% Activity pattern: De – Decreasing In – Increasing Cn – Constant Cm - Complex -- low value ++ high value
  • 23. 23   CASE  STUDY:   Wikipedia  (on-­‐site  mulCtasking)     •  ObjecCve:  Analyse  reading  acLvity  on  Wikipedia   arLcles;  idenLfy  reading  pa;erns  (clustering).   •  Metrics:  ArLcle  DwellTime,  #ArLcles  in  session,   #Views  to  focal  arLcle.   •  Data:  Sep  2011  –  Sep  2012,  500K  users,                             10K  biography  arLcles.  
  • 24. 24   Case  Study:  Wikipedia   Approach   Online  mulLtasking   Users’  reading  behaviour  on  an  Wikipedia  arCcle   ArLcle  DwellTime        How  much  Lme  do  users  spend  on  an  arLcle?   #ArLcles  in  session      Do  users  view  also  other  arLcles  during  an                  online  session?   #Views  on  focal  arLcle    How  oken  do  users  view  the  arLcle?  
  • 25. 25   Case  Study:  Wikipedia   Results   No  mulCtasking   MulCtasking   Focus   ExploraCon   Passing   Focus  is  on   focal  arHcle   Exploring  topic   around  the  focal  arHcle   Exploring  topic  and  pass   through  the  focal  arHcle   ArCcle   DwellTime   ++   -­‐-­‐   #ArCcles  in   session   -­‐-­‐   ++   ++   #Views  to             focal  arCcle   ++   -­‐-­‐   ImplicaCons   Content  quality   is  important   Links  to   addiHonal  content   are  important   ArHcles  might   need  to  be  extended   Online  mulLtasking  
  • 26.   On-­‐site  mulCtasking     •  MulLtasking  between  news  arLcles  of  a  provider.   •  MulLtasking  between  different  tasks  on  a  social  media   site  (e.g.  sharing,  chapng,  updaLng  profile).   •  …     Inter-­‐site  mulCtasking     •  MulLtasking  when  purchasing  items  online  (comparing   offers,  product  reviews,  search,  etc.)   •  …     Online  mulLtasking   26   Further  Use  Cases  
  • 27. Take  Aways   •  AccounLng  for  mulLtasking  leads  to  a   be;er  understanding  on  how  users   engage  with  sites.     •  Leaving  a  site  does  not  necessarily   entail  less  engagement,  as  users  oken   return  to  the  site  later  on.       Publications J. Lehmann, M. Lalmas, G. Dupret, and R. Baeza-Yates. Online multitasking and user engagement. CIKM 2013. J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. Reader preferences and behavior on Wikipedia. HT 2014, Ted Nelson Newcomer Paper Award. J. Lehmann, C. Müller-Birn, D. Laniado, M. Lalmas, and A. Kaltenbrunner. What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community. Wikimania 2014. Online  mulLtasking   27  
  • 28. IntroducLon   28   Analysis/ Planning   Design  Changes  Measuring  Online  mulLtasking   Inter-­‐site   engagement   Site  engagement  
  • 29. Inter-­‐site  engagement   29   MoCvaCon   Large  online  service  providers   ComScore,  Alexa,   GoogleAnalyHcs,…   Engagement   Popularity:  #Users,  #Visits,  …   AcLvity:  DwellTime,  PageViews,  …   Loyalty:  ReturnRate,  AcLveDays,  …  
  • 30. Inter-­‐site  engagement   30   MoCvaCon   Large  online  service  providers   frontpage   tv   sports   shopping   autos   search   daLng   jobs   news   shine   groups   maps   local   health   answer   weather   games   mail   omg   homes   travel   flickr   finance   Large  online  service     providers     (AOL,  Google,  Yahoo,  etc.)     have  not  only  one  site,     but  many  sites.   tumblr  
  • 31. Inter-­‐site  engagement   31   MoCvaCon   Large  online  service  providers   frontpage   tv   sports   shopping   autos   search   daLng   jobs   news   shine   groups   maps   local   health   answer   weather   games   mail   omg   homes   travel   flickr   finance   Providers  want   that  users  engage  with   many  of  their  sites.   tumblr  
  • 32. Inter-­‐site  engagement   32   MoCvaCon   Online  mulLtasking   Problem     •  Engagement  metrics  do  not  measure  engagement  across  sites.     •  How  to  adapt  them  is  not  obvious.     Inter-­‐site  engagement   Users  visit  sites  that  belong  to  the   same  network  of  sites.  
  • 33. Research  QuesCon   33  Inter-­‐site  engagement   How  can  we  measure   engagement  by  also  considering  the   relaLonships  between  sites?   Analysis/Planning   Design  Changes  Measuring  
  • 34. Inter-­‐site  engagement   34   Traffic  Networks   Modelling   We  model  sites  (nodes)  and  user  traffic   (edges)  between  them  as  a  network.     Provider  network  G=(N,  E,  λ)    N:    Sites    E:    User  traffic    λ(e):  Traffic  volume  (#Clicks)       4  clicks   2  clicks   50  clicks  10  clicks  
  • 35. Inter-­‐site  engagement   35   Measuring  Engagement   Inter-­‐site  engagement  metrics:  Network-­‐level   Traffic  distribuCon   Flow   Extent  to  which  users  navigate   between  sites.   Density1   Diversity  of  inter-­‐site  engagement.     Reciprocity2   Homogeneity  of  traffic  between  sites.     External  traffic   EntryDisparity   Variability  of  in-­‐going  traffic  to  the   network.     ExitDisparity   Variability  of  out-­‐going  traffic  from   the  network.     [1]  S.  Wasserman.  Social  network  analysis:  Methods  and  applicaHons,  1994.   [2]  T.  SquarHni,  F.  Picciolo,  F.  RuzzenenH,  and  D.  Garlaschelli.  Reciprocity  of  weighted  networks.  Nature:  ScienHfic  reports,  2013.  
  • 36. Inter-­‐site  engagement   36   Measuring  Engagement   Inter-­‐site  engagement  metrics:  Node-­‐level   Traffic  distribuCon   PageRank1   Probability  that  a  user  will  visit  the   site.   Downstream   Probability  that  a  user  will  conLnue   browsing  to  other  sites.   External  traffic   EntryProb   Probability  that  a  user  enters  the   network  in  this  site.   ExitProb   Probability  that  a  user  leaves  the   network  in  this  site.     [1]  L.  Page,  S.  Brin,  R.  Motwani,  T.  Winograd.  The  pagerank  citaHon  ranking:  Bringing  order  to  the  web.  Technical  report,  Stanford  InfoLab,  1999.  
  • 37. 37   CASE  STUDY:   Yahoo  Provider  Networks     •  ObjecCve:  Compare  networks;  characterise  the  sites   in  a  network.   •  Metrics:  Network  DwellTime,  Site  DwellTime,  Inter-­‐ site  engagement  metrics.   •  Data:  February  2014,  3.2M  clicks/network,                                   4  country-­‐based  networks,  31  sites  per  network.  
  • 38. 38   Case  Study:  Yahoo   Comparing  provider  networks   Network  1   Network  2   Network  3   Network  4   High   engaging   Users  engage  quickly   with  many  sites   Users  engage  to  a   subset  of  sites   Low   engaging   Network   DwellTime   ++   -­‐-­‐   ++   -­‐-­‐   Traffic   DistribuCon   ++   ++   Flow  ++   Density  -­‐-­‐   -­‐-­‐   Entry   Disparity   ++   -­‐-­‐   ++   ImplicaCons   The  network   is  performing   well.   This  should   be  looked  into.   MoHvate   users  to  visit   other  sites.   This  should   be  looked  into.   Inter-­‐site  engagement   -- low value ++ high value
  • 39. 39   Case  Study:  Yahoo   Sites  within  a  provider  network   Traffic  Hub   Supporter   Focused   Engagement   Shared   Engagement   Search,  front  pages   Support,  services   Leisure,  support   News,  leisure   Site   DwellTime   -­‐-­‐   -­‐-­‐   ++   ++   Traffic   DistribuCon   ++   -­‐-­‐   -­‐-­‐   ++   Entry   Probability   ++   -­‐-­‐   ++   -­‐-­‐   ImplicaCons   The  sites   forward  traffic  to   other  sites.   Users  visit  sites   for  specific  needs   and  support.   MoHvate   users  to  visit   other  sites.   The  sites   are  performing   well.   Inter-­‐site  engagement   -- low value ++ high value
  • 40.   Comparing  networks     •  Device,  Lme,  upstream  traffic,  user.   •  SimulaLons  (effect  of  adding/removing  sites).   •  …     Network  types     •  Network  of  pages  (e.g.  compare  language-­‐based   Wikipedia  networks)   •  Network  of  sites  from  different  providers  (e.g.  shopping   sites,  news  providers)   •  …     Inter-­‐site  engagement   40   Further  Use  Cases  
  • 41. Take  Aways   •  Inter-­‐site  engagement  allows  for  a   more  comprehensive  look  at  user   engagement  by  also  considering  the   relaLonships  between  sites.       •  Deeply  engaged  users  do  not  only   engage  with  one  site,  but  with  many   sites  in  a  network.       Publications J. Lehmann, M. Lalmas, and R. Baeza- Yates. Measuring Inter-Site Engagement. Handbook of Statistics, Elsevier, 2015. To appear. J. Lehmann, M. Lalmas, R. Baeza-Yates, and E. Yom-Tov. Networked User Engagement. ACM Workshop on User engagement optimization at CIKM, 2013. J. Lehmann, M. Lalmas, and R. Baeza- Yates. Temporal Variations in Networked User Engagement. TNETS Satellite at ECCS, 2013. Some of the metrics were employed to characterise online news reading across news sites: J. Lehmann, C. Castillo, M. Lalmas, and R. Baeza-Yates. Story-Focused Reading in Online News. Submitted for publication. Inter-­‐site  engagement   41  
  • 42. IntroducLon   42   Analysis/ Planning   Design  Changes  Measuring  Online  mulLtasking   Inter-­‐site   engagement   Site  engagement   Effect  of  providing   off-­‐site  content  
  • 43. 43   CASE  STUDY:   Online  News     •  Hypothesis:  It  may  be  beneficial  (long-­‐term)  to   enLce  users  to  leave  a  site  by  offering  interesLng   off-­‐site  content.   •  Data:  October  2013,  57K  users,  50  news  sites,   26K  news  arLcles.  
  • 44. Types  of  reading  sessions     No  click        Did  not  follow  a  related                                                              link.     Off-­‐site  click      Followed  a  related  link  to              content  on  another  site.     Effect  on  engagement     Short-­‐term  Dwell  Lme  per  reading          session.     Long-­‐term  Probability  that  user  starts          next  reading  session  within          the  next  12h.       44   Case  Study:  Online  News   Related  off-­‐site  content   Approach   Effect  of  providing  off-­‐site  content  
  • 45. Providing  links  to  related  off-­‐site  content  has  a   no  short-­‐term  effect,  but  a  posiCve  long-­‐term  effect.     45   Case  Study:  Online  News   Results   Effect  of  providing  off-­‐site  content   News provider Dwelltimepersession News provider p(absence12h) No Click Off-site click
  • 46. IntroducLon   46   Analysis/ Planning   Design  Changes  Measuring  Online  mulLtasking   Inter-­‐site   engagement   Site  engagement   Effect  of  providing   off-­‐site  content   Effect  of  hyperlinks  
  • 47. 47   CASE  STUDY:   Yahoo  Provider  Network     •  Hypothesis:  We  can  use  hyperlinks  to  influence   inter-­‐site  engagement  in  a  provider  network.   •  Data:  February  2014,  235M  clicks,  Yahoo  US   network,  73  sites.  
  • 48.   Hyperlink  vs.  traffic  network     On-­‐site    Links/Traffic  to  pages          within  the  same  site.     Inter-­‐site  Links/Traffic  to  pages  to          other  sites  in  the          network.     External    Links/Traffic  to          somewhere          else  on  the  Web.   48   Case  Study:  Yahoo   Approach   frontpage   sports   search   news   shine   groups   answer   weather   mail   omg   homes   flickr   Effect  of  hyperlinks  
  • 49. Hyperlinks  can  be  used  to  influence  site   and  inter-­‐site  engagement  in  a  provider  network.   However,  both  types  of  engagement  influence  each  other.     49   Case  Study:  Yahoo   Results   Effect  of  hyperlinks   Traffic On-site Inter-site External Hyperlinks On-site Inter-site External 0.54 -0.40 - -0.45 0.50 - -0.38 - 0.39
  • 50. IntroducLon   50   Analysis/ Planning   Design  Changes  Measuring  Online  mulLtasking   Inter-­‐site   engagement   Site  engagement   Effect  of  providing   off-­‐site  content   Effect  of  hyperlinks  
  • 51. Two  new  perspecHves  for  measuring   engagement  which  consider  the   relaLonships  between  sites.     Online  mulCtasking   Accounts  for  user  mulLtasking   behaviour.     Inter-­‐site  engagement   Accounts  for  the  traffic  between  sites.   ContribuLons  and  future  work   51   Main  ContribuCons   Measuring  engagement   Analysis/ Planning   Design   Changes   Measuring  
  • 52. AccounLng  for  the  new  perspecLves   when  influencing  engagement.     Online  news   Providing  related  off-­‐site  content   influences  long-­‐term  engagement.     Provider  network   Hyperlinks  affect  site  and  inter-­‐site   engagement,  but  both  influence   each  other.     ContribuLons  and  future  work   52   Main  ContribuCons   Analysis/Planning   Analysis/ Planning   Design   Changes   Measuring  
  • 53. Wikipedia   Providing  informaLon  about  readers’   engagement  to  the  editor  community.   Yahoo   Using  inter-­‐site  engagement  metrics  to   make  informed  decisions  about  design   changes  (hyperlinks).   Spiegel  Online   Measuring  and  improving  engagement   by  providing  interesLng  off-­‐site   content.   ContribuLons  and  future  work   53   What  next?   Ongoing  and  future  work   Analysis/ Planning   Design   Changes   Measuring  
  • 54. photo  credit  donsolo,  CC  BY-­‐NC-­‐SA  2.0   Thank  you!     Jane;e  Lehmann     Barcelona,  February  26,  2015   lehmannj@acm.org   Acknowledgements   Ricardo  Baeza-­‐Yates   Mounia  Lalmas   Claudia  Müller-­‐Birn   Carlos  CasLllo   David  Laniado   Andreas  Kaltenbrunner     Elad  Yom-­‐Tov   Georges  Dupret   Guy  Shaked   Fabrizio  Silvestri   Gabriele  Tolomei     Ethan  Zuckerman     John  Agapiou   Andy  Haines   Diego  Sáez-­‐Trumper   Hemant  Purohit   Noora  Al  Emadi   Mohammed  El-­‐Haddad   Nasir  Khan    
  • 55. •  Mounia Lalmas and Janette Lehmann. “Models of User Engagement”. In H. L. O’Brien and M. Lalmas (Eds.), Why Engagement Matters: Cross-disciplinary Perspectives and Innovations on User Engagement with Digital Media. Springer, 2015, in progress. •  Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. “Models of user engagement.” International Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), pp. 164-175, Montreal, Canada, July, 2012. •  Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. “Online multitasking and user engagement.” ACM International Conference on Information and Knowledge Management (CIKM 2013), pp. 519-528, San Francisco, United States, October, 2013. •  Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Measuring Inter-Site Engagement.”. In V. Govindaraju, V. V. Raghavan, and C. R. Rao (Eds.), Handbook of Statistics, Elsevier, 2015. •  Janette Lehmann, Mounia Lalmas, Ricardo Baeza-Yates, and Elad Yom-Tov. “Networked User Engagement.”, ACM Workshop on User engagement optimization at CIKM, pp. 7-10, San Francisco, United States, October, 2013. •  Janette Lehmann, Mounia Lalmas, and Ricardo Baeza-Yates. “Temporal Variations in Networked User Engagement.”, TNETS Satellite at European Conference on Complex Systems (ECCS), Barcelona, Spain, September, 2013. •  Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri, and Gabriele Tolomei. “Measuring Post-click User Experience with Mobile Native Advertising on Streams.”, submitted for publication. •  Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “Reader preferences and behavior on Wikipedia.”, ACM International Conference on Hypertext and Social Media (HT 2014), pp. 88-97, Santiago, Chile, September, 2014, Ted Nelson Newcomer Paper Award. •  Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. “What and how users read: Transforming reading behavior into valuable feedback for the Wikipedia community.”, Presentation at Wikimania, London, UK, August, 2014. •  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ricardo Baeza-Yates. “Story-Focused Reading in Online News.”, submitted for publication. •  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Transient News Crowds in Social Media.” International AAAI Conference on Weblogs and Social Media (ICWSM 2013), Boston, USA, July, 2013. •  Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman. “Finding News Curators in Twitter.” ACM International Conference on World Wide Web Companion (WWW 2013 Companion), 863-870, Rio de Janeiro, Brazil, May, 2013. 55   PublicaCons  
  • 56. User engagement •  Mounia Lalmas, Heather L O’Brien, and Elad Yom-Tov. Measuring user engagement. Synthesis Lectures on Sample Series #1. Morgan and cLaypool publishers, 2014. •  Heather L O’Brien and Elaine G Toms. What is user engagement? a conceptual framework for defining user engagement with technology. American Society for Information Science and Technology (ASIS&T), 59(6):938–955, 2008. •  Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. Towards a science of user engagement (position paper). In Proc. Workshop on User Modelling for Web Applications, WSDM, 2011. •  Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010. Online behaviour metrics •  Brian Haven and Suresh Vittal. Measuring engagement. Forrester Research, 2008. •  Eric T Peterson and Joseph Carrabis. Measuring the immeasurable: Visitor engagement. Web Analytics Demystified, 2008. •  Kerry Rodden, Hilary Hutchinson, and Xin Fu. Measuring the user experience on a large scale: user-centered metrics for web applications. In Proc. Conference on Human Factors in Computing Systems, CHI, pages 2395–2398. ACM, 2010. •  Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. In Proc. Conference on Web Search and Data Mining, WSDM, pages 173–182. ACM, 2013. •  Randolph E Bucklin and Catarina Sismeiro. A model of web site browsing behavior estimated on clickstream data. Journal of Marketing Research, 40(3):249–267, 2003. •  Birgit Weischedel and Eelko KRE Huizingh. Website optimization with web metrics: a case study. In Proc. Conference on Electronic commerce: The new e-commerce: innovations for conquering current barriers, obstacles and limitations to conducting successful business on the internet, pages 463–470. ACM, 2006. •  Peifeng Yin, Ping Luo, Wang-Chien Lee, and Min Wang. Silence is also evidence: interpreting dwell time for recommendation from psychological perspective. In Proc. Conference on Knowledge Discovery and Data Mining, SIGKDD, pages 989–997. ACM, 2013. 56   Selected  References  
  • 57. Online multitasking •  Qing Wang and Huiyou Chang. Multitasking bar: prototype and evaluation of introducing the task concept into a browser. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 103–112. ACM, 2010. •  Hartmut Obendorf, Harald Weinreich, Eelco Herder, and Matthias Mayer. Web page revisitation revisited: implications of a long-term click-stream study of browser usage. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 597–606. ACM, 2007. •  Jeff Huang and Ryen W White. Parallel browsing behavior on the web. In Proc. Conference on Hypertext and Hypermedia, HT, pages 13–18. ACM, 2010. •  Patrick Dubroy and Ravin Balakrishnan. A study of tabbed browsing among mozilla firefox users. In Proc. Conference on Human Factors in Computing Systems, SIGCHI, pages 673–682. ACM, 2010. Inter-site engagement •  Mark EJ Newman. The structure and function of complex networks. SIAM review, 45(2):167–256, 2003. 76, 77, 165 •  Anna Chmiel, Kamila Kowalska, and Janusz A Hołyst. Scaling of human behavior during portal browsing. •  Mark R Meiss, Filippo Menczer, Santo Fortunato, Alessandro Flammini, and Alessandro Vespignani. Ranking web sites with real user traffic. In Proc. Conference on Web Search and Data Mining, WSDM, pages 65–76. ACM, 2008. •  Young-Hoon Park and Peter S Fader. Modeling browsing behavior at multiple websites. Marketing Science, 23(3):280–303, 2004. •  Qiqi Jiang, Chuan-Hoo Tan, and Kwok-Kee Wei. Cross-website navigation behavior and purchase commitment: A pluralistic field research. In Proc. Pacific Asia Conference on Information Systems, PACIS, 2012. •  Kevin Koidl, Owen Conlan, and Vincent Wade. Cross-site personalization: assisting users in addressing information needs that span independently hosted websites. In Proc. Conference on Hypertext and Hypermedia, HT, pages 66–76. ACM, 2014. •  The PEW Research Center. Understanding the participatory news consumer. http://www.pewinternet.org/~/media/Files/ Reports/ 2010/PIP_Understanding_the_Participatory_News_Consumer. pdf, 2010. •  Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012. 57   Selected  References  
  • 58. Link economy •  Joseph Turow and Lokman Tsui. The hyperlinked society. The University of Michigan Press, 2008. •  Juliette De Maeyer. Hyperlinks and journalism: where do they connect? In Proc. Future of Journalism Conference, 2011. •  Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink formation in content networks. Management Science, 59(10):2360–2379, 2013. •  Jason MT Roos. Hyper-Media Search and Consumption. PhD thesis, Duke University, 2012. •  Chrysanthos Dellarocas, Zsolt Katona, and William Rand. Media, aggregators, and the link economy: Strategic hyperlink formation in content networks. Management Science, 59(10):2360–2379, 2013. •  Hakan Ceylan, Ioannis Arapakis, Pinar Donmez, and Mounia Lalmas. Automatically embedding newsworthy links to articles. In Proc. Conference on Information and Knowledge Management, CIKM, pages 1502–1506. ACM, 2012. Recommendation •  Richard McCreadie, Craig Macdonald, and Iadh Ounis. News vertical search: when and what to display to users. In Proc. Conference on Research and Development in Information Retrieval, SIGIR, pages 253–262. ACM, 2013. •  Samuel Ieong, Mohammad Mahdian, and Sergei Vassilvitskii. Advertising in a stream. In Proc. Conference on World Wide Web, WWW, pages 29–38. ACM, 2014. •  Eric Sodomka, Sébastien Lahaie, and Dustin Hillard. A predictive model for advertiser value-per-click in sponsored search. In Proc. Conference on Information and Knowledge Management, CIKM, pages 1179–1190. ACM, 2013. •  Narongsak Thongpapanl and Abdul Rehman Ashraf. Enhancing online performance through website content and personalization. Journal of Computer Information Systems, 52(1):3, 2011. •  Jian Wang and Yi Zhang. Utilizing marginal net utility for recommendation in e-commerce. In Proc. Conference on Research and Development in Information Retrieval, SIGIR, pages 1003–1012. ACM, 2011. •  Joshua Porter. Designing for the social web. Peachpit Press, 2010. 58   Selected  References  
  • 60. •  “In  a  world  full  of  choices  where  the  fleeCng  aPenCon  of   the  user  becomes  a  prime  resource,  it  is  essenLal  that  [...]   providers  do  not  just  design  [websites]  but  that  they   design  engaging  experiences.”  [A}ield].   •  In  addiLon  to  uLlitarian  factors,  such  as  usability  and   usefulness,  we  must  consider  other  factors  of  interacLng   with  websites,  such  as  fun,  fulfillment,  play,  and  user   engagement.   Successful  websites  are  not  just  used,                                                                         they  are  engaged  with.   •  In  order  to  design  engaging  websites,  it  is  crucial  to   understand  what  user  engagement  is  and  how  to   measure  it.   IntroducLon   60   MoCvaCon   Why  is  it  important  to  engage  users?  
  • 61. Methodology   InteracLon  data,  online  sessions  and  site  visits.   61  IntroducLon   t0 t1 t2 t3 t4 t5 t6 t7 session end session start time Online session Browsing activity on Wikipedia https://ie-mg42.mail.yahoo.com http://en.wikipedia.org/wiki/Freddie˙Mercury http://www.bbc.com/news/uk-29149115 http://www.bbc.com/news/uk-england-nottinghamshire-29643802 http://en.wikipedia.org/wiki/Star Wars http://en.wikipedia.org/wiki/Yoda http://en.wikipedia.org/wiki/Albert˙Einstein https://www.facebook.com/janette.lehmann.5 t0 t1 t2 t3 t4 t5 t6 t7 bc0 bc0 bc0 bc0 bc0 bc0 bc0 bc0 BCookie Timestamp URL - - - http://www.bbc.com/news/uk-29149115 - http://en.wikipedia.org/wiki/Star Wars http://en.wikipedia.org/wiki/Yoda - ReferrerURL Interaction data Page view on Wikipedia Page view on other site
  • 62. IntroducLon   62   Thesis  structure   Metrics that account for site popularity, activity and loyalty Advertising Chapter 7 Site engagement How users experience ads on desktop and mobile devices? Does ad quality affect the engagement with the publisher? How can we identify high quality ads? Site engagement Chapter 4 Multitasking Chapter 5 Inter-site engagement Chapter 6 Metrics that account for traffic between sites Metrics that account for user multitasking behaviour (III+IV)Applications(II)Fund. Wikipedia Chapter 8 Site engagement and multitasking How users read articles in Wikipedia? Does the activity of editors align with the engagement of readers? How can readers be valuable for editors? Yahoo Chapter 9 Inter-site engagement How users engage with a provider network of sites? Does the hyperlink structure affect site and inter-site engagement? Online news Chapter 10+11 Inter-site engagement How users read stories across news providers? Do hyperlinks to related content influence provider engagement? How can we automatically detect related content? Characterising user engagement Comparing site characteristics and user engagement Applications to impact user engagement
  • 64. 0-1 1-0.5 0.5 Kendall’s tau with p-value < 0.05 ('-' insignificant correlations) Site  engagement   64   EvaluaCon   CorrelaLons  between  engagement  metrics.   High  correlaCons               within  metric  groups.       Low  correlaCons     between  metric  groups.   [POP]#Users [POP]#Visits [POP]#Clicks [ACT]PageViewsV [ACT]DwellTimeV [LOY]ActiveDays [LOY]ReturnRate #Users [POP] 0.82 0.75 - - 0.43 0.34 #Visits [POP] 0.82 0.85 - - 0.60 0.52 #Clicks [POP] 0.75 0.85 0.16 0.18 0.59 0.51 PageViewsV [ACT] - - 0.16 0.33 - - DwellTimeV [ACT] - - 0.18 0.33 - - ActiveDays [LOY] 0.43 0.60 0.59 - - 0.79 ReturnRate [LOY] 0.34 0.52 0.51 - - 0.79 0.69
  • 65. Site  engagement   65   PaPerns  of  Site  Engagement   Engagement  depends  on  the  site  at  hand.   Games   Not  many  users,   but  they  stay  long   Search   Users  come   frequently,  but  do   not  stay  long   Social  media   Users  come   frequently  and   stay  long   Shopping   Users  do  not  come   frequently,  but   stay  long   News   Users  come   frequently  and     stay  long   Service   Users  do  not  come   frequently,  but   stay  long  
  • 67. Online  mulLtasking   67   MoCvaCon   Users  switch  between  sites,  to  perform  related  or  totally  unrelated  tasks.               Switching  between  tasks  (sites)   “…within-­‐session  page  revisits  represent  the  most  common  form  of  revisitaLon,   covering  73,54%  of  all  revisits.”  [Herder]     Performing  tasks  (sites)  in  parallel  using  browser  tabs   “Most  of  our  parLcipants  switched  tabs  more  oken  than  they  used  the  back   bu;on.”  [Dubroy]   [Herder]  E.  Herder.  CharacterizaHons  of  user  web  revisit  behavior.  WWW  Workshop  ABIS,  2005.   [Dubroy]  P.  Dubroy,  R.  Balakrishnan.  A  study  of  tabbed  browsing  among  mozilla  firefox  users.  SIGCHI,  2010.  
  • 68. Online  mulLtasking   68   Data   Dataset  and  site  categories.   Cat. Subcat. %Sites Description news 22.1% news 5.79% news (soc.) 5.13% society news (sport) 2.63% news (enter.) 2.24% music, movies, tv, etc. news 1.97% news (life) 1.58% health, housing, etc. news (tech) 1.58% technology news (weather) 1.18% service 15.5% service 7.63% translators, banks, etc. mail 3.95% maps 3.03% organisation 0.92% bookmarks, calendar, etc. search 15.3% search 12.63% search (special) 1.58% search for lyrics, jobs, etc. directory 1.05% sharing 9.6% blogging 3.55% knowledge 3.55% collaborative creation and collection of content sharing 2.50% sharing of videos, etc. navi 9.3% front page 6.58% front page (p.) 1.84% personalised front pages sitemap 0.92% leisure 8.7% adult 2.76% games 1.97% social media 1.97% dating 1.05% entertainment 0.92% sites with music, tv, etc.support 8.7% support 1.58% sites that provide products and support for them download 7.11% downloading software shopping 7.9% shopping 4.34% auctions 2.11% comparison 1.45% sites to compare prices of products settings 2.9% login 1.71% site settings 1.18% pr e setting, site personalisation InteracCon  data   •  July  2012   •  2.5M  users   •  785M  page  views   NavigaCon  model   •  We  defined  a  new  navigaLon   model  (see  paper  for  details)     Site  categories   •  760  sites  from  70  countries/ regions   •  11  categories   •  33  subcategories  
  • 69. Online  mulLtasking   69   MulCtasking  Metrics   CumAct  accounts  for  the  acLvity  between  site  visits.   CumulaCve  acCvity   The  metric  is  defined  as  follows:             InterpretaCon   High  CumAct  à  High  engagement   If  users  return  aker  short  Lme,  they  return  to   conLnue  with  same  task.   If  users  return  aker  longer  Lme,  they  return  to   perform  a  new  task  –  a  sign  of  loyalty.   CumActk = log10 (v1 + ivi k •vi i=2 n ∑ ) Browsing  acLvity  during  the  ith  visit   Browsing  acLvity  between  the  (i-­‐1)th  and  ith  visit   Rescaling  factor  for  ivi       k = 3 vi ivi 1   4  3   10   3   CumAct = log10 (3+13 •4+103 •3) = 3.48 Site  visit  
  • 70. Online  mulLtasking   70   MulCtasking  Metrics   AWRange  and  AWShik  describe  changes  between  the  visits.   APenCon  shie  and  range   The  metrics  is  defined  as  follows:                     InterpretaCon   AWShik  models  the  shik  of  a;enLon,  and   AWRange  models  the  fluctuaLons   in  the  browsing  acLvity.   AttShiftn = invn − minInvn | maxInvn |− | minInvn | AttRangen = σ (Vn ) µ(Vn ) Variance  in  the  visit  acLvity   Average  of  the  visit  acLvity   Number  of  visits  in  session     ModificaLon  of  the  “Inversion  number”       n = 4 σ µi Inv 0   >0   -­‐1   constant   decreasing   0   constant   complex   +1   constant   increasing   AWenHon  range   AWenHon  shik  
  • 71. 0-1 1-0.5 0.5 Spearman’s rho with p-value < 0.05 ('-' insignificant correlations) Online  mulLtasking   71   EvaluaCon   CorrelaLons  between  mulLtasking  and  acLvity  metrics.   [MT]SessVisits [MT]SessSites [MT]CumAct [MT]AttShift [MT]AttRange [ACT]DwellTimeS SessSites [MT] 0.42 CumAct [MT] 0.41 - AttShift [MT] 0.09 - - AttRange [MT] - - -0.38 0.27 DwellTimeS [ACT] 0.20 0.24 0.12 0.32 0.08 DwellTimeV [ACT] -0.40 - - 0.14 - 0.50 No  or  only  weak   correlaCons  between                   the  metrics.     All  metrics  convey   different  aspects  about   users’  online  behaviour.  
  • 72. Online  mulLtasking   72   MulCtasking  PaPerns   Cluster  centers,  site  categories  and  acLvity  pa;erns.   CategoriesMultitasking DwellTimeV CumAct SessVisitsDwellTimeS sitemap site settings news (wheather) download +75% +73% +69% +67% PD 139 sites Quick task Continuous multitasking SessSitesBars from left to right: 111 sites auctions shopping adult dating +79% +71% +71% +62% PD -1.0 1.0 0.0 -1.0 1.0 0.0 PD - Probability difference Activity Activity pattern: De - Decreasing In - Increasing Cn - Constant Cm - Complex De In CmCn De In CmCn 60% 0% 147 sites Recurring task search front page (p.) front page organisation +77% +62% +57% +24% PD -1.0 1.0 0.0 De In CmCn 0.6 0.0 137 sites Focused task news (tech) news (life) support mail +66% +66% +65% +64% PD -1.0 1.0 0.0 -1.0 1.0 0.0 De In CmCn 60% 0% 60% 0% 142 sites Rapid multitasking news (enter.) knowledge comparison service +64% +63% +62% +59% PD -1.0 1.0 0.0 -1.0 1.0 0.0 De In CmCn 60% 0% Single-task-oriented browsing Multitask-oriented browsing
  • 74. Inter-­‐site  engagement   74   Data   Dataset,  network  and  site  categories.   InteracCon  data   •  August  2013  to  July  2014   •  53M  sessions   Provider  network  G=(N,  E,  λ)      N:    155  Yahoo  sites      from  five  countries      E:    User  traffic      λ(e):  Traffic  volume  (#Clicks)           Site  categories   •  155  sites  from  5  countries   •  5  categories   Cat. %Sites Description 35% 19% 13% 23% 10% news service leisure provider front page mail, calendar, etc. social media, games, etc. account settings, help, etc. front pages, site maps servicefront page news providerleisure
  • 75. Inter-­‐site  engagement   75   Inter-­‐site  Engagement  Metrics   Flow  accounts  for  the  extent  users  navigate  between  sites.   Traffic  Flow   The  metric  is  defined  as  follows:               InterpretaCon   High  Flow  à  High  inter-­‐site  engagement   Users  navigate  oken  between  the  sites  of  the   network.   Flow = wi, ji, j∑ vii∑ #Clicks  between  node  i  and  j   #Visits  on  node  i   wi, j vi Flow = 30/60 = 0.5 10 5 20 20 20 10 5 1 1 20 20 20 1 1 Flow = 4/60 = 0.07
  • 76. Inter-­‐site  engagement   76   Inter-­‐site  Engagement  Metrics   Density  describes  the  connecLvity  of  the  network.   Density   We  use  the  density  measure    of  [Wasserman]:             InterpretaCon   High  Density  à  High  inter-­‐site  engagement   Users  navigate  between  many  different  sites   (inter-­‐site  engagement  is  highly  diverse).   [Wasserman]  S.  Wasserman.  Social  network  analysis:  Methods  and  applicaHons,  1994.   Density = # Edges # Possible_ Edges Density = 4/6 = 0.7 Flow = 2/6 = 0.3
  • 77. Inter-­‐site  engagement   77   Inter-­‐site  Engagement  Metrics   Reciprocity  measures  the  homogeneity  of  traffic  between  two  sites.   Reciprocity   We  use  the  reciprocity  measure  of  [SquarLni]:               InterpretaCon   High  Reciprocity  à  High  inter-­‐site  engagement   Users  navigate  between  two  sites  in  both  direcLons   (inter-­‐site  engagement  is  highly  homogenious).   [SquarHni]  T.  SquarHni,  F.  Picciolo,  F.  RuzzenenH,  and  D.  Garlaschelli.  Reciprocity  of  weighted  networks.  Nature:  ScienHfic  reports,  2013.   #Clicks  between  node  i  and  j  wi, j RP = min[wi, j,wj,i ] i<j∑ wi, ji≠j∑ 1 10 5 20 1 Reciprocity = 15/50 = 0.3 Reciprocity = 2/37 = 0.05 10 10 5 20 5
  • 78. Inter-­‐site  engagement   78   Inter-­‐site  Engagement  Metrics   Entry/ExitDisp  measures  how  the  traffic  to/from  the  network  is  distributed  over  the  sites.   Entry  disparity  and  exit  disparity   We  use  the  group  degree  measure  of  [Freeman]  and   adapt  it  as  follows:                 InterpretaCon   High  Entry/ExitDisp  à  Low  inter-­‐site  engagement   The  network  is  more  vulnerable  to  outages,  because   only  few  sites  are  used  to  enter  (leave)  the  network.   EntryDisp = (gin max − gin i ) i∑ | N |• gin ii∑ [Freeman]  L.  C  Freeman.  Centrality  in  social  networks  conceptual  clarificaHon.  Social  networks,  1979.   Number  of  visits  that  started  at  node  ni   (user  entered  the  network)   Maximum  value  of  gin   Number  of  nodes  | N | gi in gin max EntryDisp = 20/3 40 = 0.17 20 10 10 40 5 5 EntryDisp = 70/3 50 = 0.47
  • 79. Inter-­‐site  engagement   79   EvaluaCon:  Network-­‐level   CorrelaLons  between  inter-­‐site  and  network  engagement  metrics.   [IS]Density [IS]Reciprocity [IS]EntryDisparity [IS]ExitDisparity [POP]#Sessions [ACT]DwellTimeS [ACT]#Sites Flow [IS] - 0.15 0.23 0.30 - 0.35 0.65 Density [IS] 0.48 -0.61 -0.60 0.92 -0.45 -0.25 Reciprocity [IS] -0.38 -0.32 0.42 - 0.25 EntryDisparity [IS] 0.84 -0.54 0.33 - ExitDisparity [IS] -0.55 0.38 0.20 0-1 1-0.5 0.5 Spearman’s rho with p-value < 0.01 ('-' insignificant correlations) Density  and  #Sessions   The  more  users  are   visiCng  the  network,  the   more  diverse  is  the  inter-­‐ site  engagement.     Entry-­‐  and  ExitDisparity   Volume  of  in-­‐  and  out-­‐ going  traffic  of  the  nodes   depend  on  each  other.     Flow  and  #Sites   The  more  sites  are  visited   during  a  session,  the   higher  the  flow  of  traffic.  
  • 80. Inter-­‐site  engagement   80   EvaluaCon:  Node-­‐level   CorrelaLons  between  inter-­‐site  and  site  engagement  metrics.   [IS]Downstream [IS]EntryProb [IS]ExitProb [POP]#Sessions [ACT]DwellTimeS [MT]#Visits [MT]CumAct PageRank [IS] 0.30 -0.08 -0.10 0.85 0.06 0.08 0.31 Downstream [IS] -0.27 -0.22 0.17 0.04 0.02 -0.02 EntryProb [IS] 0.79 0.12 -0.19 0.13 0.35 ExitProb [IS] 0.08 -0.18 0.18 0.32 0-1 1-0.5 0.5 Spearman’s rho with p-value < 0.01 ('-' insignificant correlations) PageRank  and  #Sessions   Popular  sites  in  the   provider  network,  are   also  visited  frequently   when  browsing  through   the  network.     Entry-­‐  and  ExitProb   Nodes  that  are  used  to   enter  the  network  are   also  frequently  used  to   exit  the  network.  
  • 81. Inter-­‐site  engagement   81   Comparing  Provider  Networks   Country2 Country1 Country3 Country4 Country5 Flow Reciprocity EntryDisparityDensity DwellTimeBars from left to right: -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0 -1.0 0.0 1.0
  • 82. Inter-­‐site  engagement   82   PaPerns  of  Inter-­‐site  Engagement  CategoriesEngagement PageRank EntryProb DwellTimeDownstream PD - Probability difference 46 sites Focused eng. front page service provider leisure news +80% +63% -100% -100% -100% PD 23 sites Traffic hub 46 sites Supporter 40 sites Shared eng. CumActBars from left to right: -1.0 1.0 0.0 provider service news front page leisure +31% +19% -2% -10% -100% PD leisure provider service news front page +67% +48% -21% -94% -100% PD news leisure provider front page service +63% -61% -100% -100% -100% PD -1.0 1.0 0.0 -1.0 1.0 0.0 -1.0 1.0 0.0
  • 84. NaLve  AdverLsing   84   Effect  on  User  Engagement   0% 200% 400% 600% short ad clicks long ad clicks adclickdifference short ad clicks long ad clicks clicksperdaydifference PosiLve  experience  has  a  strong  effect  on   users  clicking  on  ads  again,  and  a  small   effect  on  user  engagement  with  the   stream.      
  • 85. NaLve  AdverLsing   85   Mobile  vs.  Desktop   Ad  post-­‐click  experience  between  mobile   and  desktop  differs.   For  dwell  Lme  we  obtain  rho  =  0.50;  this   value  is  even  smaller  for  bounce  rate  with   rho  =  0.23.     0.00 0.05 0.10 0.15 dwell time difference p(dwelltimedifference) higher on mobilehigher on desktop 0.00 0.05 0.10 0.15 bounce rate difference p(bounceratedifference) higher on mobilehigher on desktop
  • 86. NaLve  AdverLsing   86   Mobile  OpCmised  Landing  Pages   Dwell  Cme:  The  distribuLon  is  very  similar  for   both  groups.     Bounce  rate:  Decreases  by  6.9%  (median   decreases  by  30.4%)  for  Opt  landing  pages  but   increases  by  13.4%  (median  decreases  by  11.5%)   for  Npt  landing  pages.   not mobile optimized mobile optimized 0.0 0.1 0.2 0.3 dwell time difference p(dwelltimedifference) higher on mobilehigher on desktop mobile opt. not mobile opt. 0.0 0.1 0.2 bounce rate difference p(bounceratedifference) higher on mobilehigher on desktop mobile opt. not mobile opt.
  • 88. Wikipedia   88   Wikipedia  Research   Literature  review  by  Okoli  et  al.:  The  people’s  encyclopedia  under  the  gaze  of  the  sages:  A   systemaLc  review  of  scholarly  research  on  wikipedia.      
  • 89. Wikipedia   89   Reading  Preferences   Popularitylow high ArticleLengthshortlong borderline casesII I III IV Jeanne Tsai Douglas Adams Luis Palomino Anne Stears Peter Ehrlich Alec Mango Stephen D. Lovejoy 1st Dalai Lama Dexter Jackson (safety) Katie Green Brittany Borman Anthony Anenih Ronnie Bird Jan Anderson (scientist) Fitch Robertson Sean Bennett For 4.2% (group IV) of the articles editing activity is low, but reading activity is high.!
  • 90. Wikipedia   90   Reading  PaPerns  Article topic Reading behavior ArticleViewsa SessionArticlesa Popularitya ReadingTimea CA - Percentage in topic 4,826 articles 11,579 behavior vectors sportsperson musician media pers. 28% 26% 23% CA Exploration artist/writer historical fig. polit./businessp. 43% 41% 37% CA 5,278 articles 10,605 behavior vectors Focus 3,876 articles 14,267 behavior vectors historical fig. criminal/victim musican 42% 38% 38% CA Trending 5,684 articles 13,470 behavior vectors media pers. sportsperson musician 27% 27% 19% CA Passing 28K [16K,51K] 11 [5,23] 7.7% 38K [21K,69K] 20 [9,41] 16.9% 26K [15K,45K] 10 [5,21] 10.5% 16K [10K,27K] 8 [3,18] 5.1% ArtLen #Edits %HQA #Edits - Number of edits -1.0 0.5 -0.5 0.0 1.0 -1.0 0.5 -0.5 0.0 1.0 -1.0 0.5 -0.5 0.0 1.0 -1.0 0.5 -0.5 0.0 1.0 %HQA - Percentage of high quality articlesArtLen - Article length
  • 91. Wikipedia   91   Reading  PaPerns  over  Time   Stability   •  30%  of  the  arLcles  are  popular  in  1  month   •  10%  are  popular  over  the  whole  13-­‐months   •  Almost  all  arLcles  have  one  reading  pa;ern   half  of  their  life  Lme   TransiCons   •  TransiLons  are  temporary  –  arLcles  move   temporarily  to  another  cluster   •  High  reciprocity  –  similar  number  of   transiLons  in  both  direcLons   •  “Focus”  cluster  is  isolated  -­‐  ArLcles  in  that   cluster  are  the  most  stable  ones   •  Strong  connecLon  between  the  “Passing”,   “ExploraLon”,  and  “Trending”  clusters  –   many  arLcles  adopt  all  three  pa;erns  
  • 93. 93   Upstream  Traffic   TeleportaCon   Social  media  /  News   Search  /  Ext-­‐Yahoo   Users  engage  (quickly)   to  many  sites.   Users  conHnue  with   same  acHvity  inside   the  provider  network.   Users  visit  site  they  are   interested  in,  perform  a   quick  task,  and  leave.   Network   DwellTime   -­‐-­‐   ++   -­‐-­‐   Traffic   DistribuCon   ++   -­‐-­‐   -­‐-­‐   Entry   Disparity   -­‐-­‐   Yahoo   Users  engage  differently  depending  on   where  they  are  coming  from.  
  • 94. 94   Network  Effect  PaPern   Yahoo   Sites  change  their  popularity  in  the  same  way.   Ac>vity  (dwell  >me)  on  a  site  depends  more  on  the  site  itself,   but  there  are  some  nega>ve  dependencies.   Pattern examples 41 patterns Simple star-like 6 patterns Complex star-like 1 pattern Cluster-like 3.00 [3.00,4.00] 0.67 [0.00,0.89] 0 [0,0] 8.00 [7.00,18.00] 0.76 [0.56,0.84] 0 [0,0] 52 0.91 0.51 N Recip Trans N - Number of nodes Recip - Reciprocity Trans - Transitivityservicefront page news providerleisure (4) (5) (6)(1) (2) (3)
  • 95. 95   Hyperlink  Performance   Yahoo   0% 25% 50% 75% 100% Onsitelinks front page providerservice news leisure Intersitelinks front page providerservice news leisure 0% 20% 40% 60% 80% Externallinks front page providerservice news leisure 20% 40% 60% (a) PageRank and downstream. Traffic PageRank Downstream Hyperlinks PageRank 0.54 - Downstream - - (b) On-site, inter-site, and external. Traffic On-site Inter-site External Hyperlinks On-site 0.54 -0.45 -0.38 Inter-site -0.40 0.50 - External - - 0.39
  • 97. Online  news   97   Focused  versus  Non-­‐focused  Sessions   Internal   Non-focused sessionsFocused sessions● (b) Duration (d) p(focused session) (a) %Sessions (f) Flow 25 15 5 60% 20% 0.6 0.2 0.2 0.1 2 3 4 5 6 7 7 2 3 4 5 6 7 7 2 3 4 5 6 7 7 #Articles #Articles #Articles ● ● ● ● ● ● ● (c) #Providers 2.5 2.0 1.5 ● ● ● ● ● ● ● ●● ●● ●● ●● ●● ●● ●● (e) EntryDisparity 0.5 0.3 0.1 ● ● ● ● ● ● ● When  users  focus  on  a  news  story,  they  spend  more  >me  reading  the   ar>cles  and  the  inter-­‐site  engagement  between  providers  is  higher.  
  • 98. Online  news   98   Hyperlink  Performance     Number  of  Inline  Links   •  <10  links  may  be  wasLng  an  opportunity   •  10-­‐29  links  does  not  result  in  more  clicks   •  >29  links  may  harm  the  user  experience       PosiCon  of  Inline  Links   •  30%  at  the  end,  16%  at  the  beginning,  46%   are  distributed  within  the  arLcle  text.   •  Performance  of  links  located  at  the   beginning  of  the  text  is  very  low  (-­‐28%)   •  Best  performance  is  achieved  with  links  at   the  end  of  the  arLcle  text  (+35%)   Link popularity● Link performance Position in article text Linkpopularity [0.0,0.1[ [0.3,0.4[ [0.6,0.7[ [0.9,1.0] 10% 20% 30% -0.2 0.0 0.2 Linkperformance ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● ●● Number of inline links in article Clicksperlink 0.0 0.2 0.4 0.6 [0,2] [9,11] [18,20] [27,29] [36,38] Number of inline links in article Numberofclicks [0,2] [9,11] [18,20] [27,29] [36,38] 2.5 5.0 7.5
  • 99. Online  news   99   Effect  on  User  Engagement   Internal  Focused     Short-­‐term:  Only  3  (out  of  50)  providers   have  their  corresponding  average  dwell  Lme   lower  for  the  story-­‐focused  provider   sessions.  The  average  increase  in  dwell  Lme   from  non-­‐story-­‐focused  to  story-­‐focused   provider  sessions  is  50%.       Long-­‐term:  For  78%  of  the  providers,  we   find  that  there  are  more  users  that  return   earlier  aker  they  have  a  story-­‐focused   provider  session.       Internal   News provider Dwelltimepersession Non-focused Focused Ext-focused News provider p(absence12h) Non-focused Focused Ext-focused
  • 100. Online  news   100   Effect  on  User  Engagement   External  Focused     Short-­‐term:  We  do  not  observe  an  effect  on   the  dwell  Lme  (neither  posiLve  nor   negaLve).  The  average  increase  is  only  5.5%,   and  based  on  the  K-­‐S  test  we  cannot  confirm   that  the  distribuLons  are  different  (p-­‐ value=0.36).       Long-­‐term:  For  70%  of  these  news  sites,  the   probability  that  users  return  within  the   following  12  hours  increases  (the  average   increase  is  76%).         External   News provider Dwelltimepersession Non-focused Focused Ext-focused News provider p(absence12h) Non-focused Focused Ext-focused
  • 101. Online  news   101   Discovering  Story-­‐related  Content  in  TwiPer