Does more information elicit users’ compliance and engagement, or the other way around? This paper explores the relationship between content strategy and user experience (UX). Specifically, we examine how the amount of information provided on marketing web pages, often called “landing pages,” impact users’ willingness to provide their e-mail address (a behavior called “conversion” in marketing terms). We describe the results of two large-scale online experiments (n= 535 and n= 27,900) conducted in real-world commercial settings. The observed results indicate a negative correlation between the amount of information on a web page and users’ decision-making and engagement. Presented by Nim Dvir

1. Less is more: an
empirical investigation
of the relationship
between the amount
of digital content and
user engagement
Nim Dvir
ndvir@Albany.edu
@nimdvir
University at
Albany
State
University of
New York
(SUNY)
1

2. How to be
interesting?
A content-based
approach for user
engagement

3. INTRODUCTION
3
Experience
Nim Dvir
Ndvir@albany.edu
Personal Page |
www.albany.edu/~nd115232/
Research & Teaching Assistant
State University of New York at Albany
Operation Manager
ClaimFame.com
Project Manager
The Jewish Agency for Israel
Editor, Journalist
Pnai Plus, Ynet, Erev Tov
Education
P
h
D
Information Science
Human-Computer Interaction
SUNY Albany
M
B
A
Information Systems
Marketing
City University of New York
B
A
International Relations
Communication
New York University

4. Quick experiment #1:
Please choose one
4

5. Quick experiment #1:
Please choose one
27,083 unique visitors
42%29%
users who had less
information were more
inclined to provide their
data
5

6. IDENTIFICATION OF
ZOMBIE HOMES USING
SYSTEM DYNAMICS
END OF THE LIBRARY:
ORGANIZED
INFORMATION AND
DIGITAL UBIQUITY
DEATH OF LIBRARY:
ORGANIZED
INFORMATION AND
DIGITAL UBIQUITY
IDENTIFICATION OF
VACANT HOMES USING
SYSTEM DYNAMICS
Quick experiment #2:
Please choose one
6

7. IDENTIFICATION OF ZOMBIE
HOMES USING SYSTEM
DYNAMICS
END OF THE LIBRARY:
ORGANIZED INFORMATION
AND DIGITAL UBIQUITY
DEATH OF LIBRARY:
ORGANIZED INFORMATION
AND DIGITAL UBIQUITY
IDENTIFICATION OF VACANT
HOMES USING SYSTEM
DYNAMICS
41%
30%
65%
76%
Vacant vs. zombie homes End of the library vs. Death of library
n = 405
7

8. IDENTIFICATION OF ZOMBIE
HOMES USING SYSTEM
DYNAMICS
END OF THE LIBRARY:
ORGANIZED INFORMATION
AND DIGITAL UBIQUITY
DEATH OF LIBRARY:
ORGANIZED INFORMATION
AND DIGITAL UBIQUITY
IDENTIFICATION OF VACANT
HOMES USING SYSTEM
DYNAMICS
1.30
3.76
2.41
3.65
1.97
4.62
1.36
3.77
1.25
4.39
2.28
4.44
1.53
4.70
2.62
3.36
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
End of the library: Adaptable
organized information and
digital ubiquity
Death of library: adaptable
organized information and
digital ubiquity
Identifying vacant homes using
system dynamics
Identifying zombie homes
using system dynamics
This article is interesting I believe that reading this article is worthwhile
This article is easy to understand This article is appealing, I wish to read more
8

9. WHAT DOES THIS STUDIES HAVE IN COMMON?
Content strategy
9

10. WHAT IS CONTENT STRATEGY?
•Information → Interaction → Reaction
•Three C’s
10
Content ConsequenceContext
Independent variable effect Dependent variable

11. OBJECTIVES
•Develop a precise criteria to allow
quantitative evaluation and
operational formulation
•Suggest a unified, generalized and
overarching principles and models
(“heuristics”)
•Test the heuristics empirically across
domains and applications
•Examine the possibility to use the heuristic for
methodological, automatic or semi-automatic
creation 11

12. Introduction and Scope:
Question: How digital content influence user engagement
Aim: Develop a framework for systematic evaluation, optimization,
and creation of engaging content
Litrature review - Theoretical Foundations
● Definitions of digital content and user engagement
● Original process-based model
● Characteristics / Determinants
Field experiments
Less is more - An Empirical Investigation of the Relationship
Between Amount of Digital Content and User Engagement
Sticky words - Automation, using Lexical substitution is the task
of NLP to increase engagement identifying a substitute for a word
in the context of a clause
Developing Framework
Content Strategy, Heuristic evaluation
Theory
Methodology
Practice

13. 13
LITERATURE
REVIEW
What we know
and what we need to
know

14. RELEVANCE
14
93% of Americans report of
consuming information online,
either via a mobile device or a
computer (Stocking, 2017)
79% of U.S. adults reported
ever making an online
purchase, spending nearly
$350 billion annually
(Smith, 2017)
80% of Americans have used at
least one online government
service
(Im, Porumbescu, & Lee, 2013)
71% of Americans report
looking online for health
information
(Rock Health, 2015)
The use of ICT has increased radically

15. CHALLENGES
1. Increased competition
substantial increase in online information, published by
commercial businesses, governments, healthcare
organizations, and private citizens, lead to billions of Web
pages published
(Lucian, 2014)
2. Information overload and limited attention
Encounters with massive amounts of potentially relevant
information limit cognitive abilities. As more web content
becomes available user attention stretches even thinner
across the online space, making it possible only for a small
percentage to receive significant traffic
(Arapakis, Cambazoglu, & Lalmas, 2017)
15

16. Information interactions are
becoming highly asymmetrical
16
Only a small portion of information
receives significant attention, while the
remaining majority is barely noticed
(Szabo & Huberman, 2008)
From the information that is
noticed, less than a third is read by
users (Nielsen, 2015)

17. PROBLEM
•Current research is design-focused
•Lack of agreement or
standarization
○Definitions
○Theoretical and operational
Frameworks
○Characteristics
○Measurement metrics
○Automation
17

18. OBJECTIVES
Critical review of
1. Established theories and
prior research
2. Conceptual models of UX
3. Determinants, attributes
and outcomes of UE in
various domains
18
User
InteractionSystem

19. 19
CONTENT STRATEGY THEORY
Aims to provide a
conceptualization that is
applicable across a range
of situations
(Brodie et al., 2013)
Theory is in its infancy, therefore
Integrates knowledge and
methods from different
disciplines
(Boyle et al., 2012; O’Brien & Cairns,
2016)
Includes adoption and usage
of technology, the feeling when
using the technology and the
motivation to use it
(O’Brien et al., 2018; Perski et al.,
2017)

20. METHODOLOGY
1. Interdisciplinarity approach
Draws well-established theoretical
frameworks from a number of domains:
Information Systems (IS) - The system
and its usage
Psychology and Behavioral Sciences –
The users and their motivations
Human-Computer Interaction (HCI) –
The interaction between system and users
2. Domain research
Online news, E-commerce and marketing, Digital
games, Social media, online information search,
Digital, Public policy and Education / eLearning
20
User Interaction
System

21. DOMAINS
• Information systems
• Effective informing influences users to take
desirable actions
• Means – end theory
• Provision of information is a central construct in
the IS discipline.
• Behavioral economics
• Digital nudging – Approach to how changes to
the decision environment influence choices
• Human-Computer Interaction (HCI)
• User interface (UI) and UX design
• User engagement
IS
BE
HCI
21

22. INTERDISCIPLINARY SYNTHESIS
22
Domain Information systems Psychology HCI
Unitof
analysis
System User Interaction
Focus
Adoption and usage
Motivation and
inherent drives
Subjective experience
Theories
• Technology Acceptance Model (TAM)
• The Unified Theory of Acceptance and Use of
Technology (UTAUT)
• The information systems success model
• Means end
• Self-Determination Theory (SDT)
• Uses and gratifications theory
(UGT)
• Flow theory
• Play theory
• Aesthetic Experience theory
Determinants
• Objective, performance-based
and pragmatic
• Effectiveness, efficiency,
perceived ease of use, and
satisfaction
• Intrinsic and extrinsic
motivations
• Cognitive, social and
personal needs and
gratifications
• Utilitarian and hedonic
benefits
• Subjective, experience-
based
• Voluntary, focused
attention, intrinsic interest
and enjoyment ,rewarding
and stimulating, and
temporal dissociation

23. ENGAGEMENT
1.A state of emotional
involvement or
commitment
2.Process of drawing
favorable attention
and interest
(Merriam-Webster
dictionary) 23

24. USER ENGAGEMENT IS
EVERYWHERE
24

25. Shane Smith, co-founder and CEO
of VICE.com (“The Engagement
Project,” Google)
“Engagement is the most
important thing. Who cares
about hits? Who cares about
pageviews? You care about a
user being engaged for a
duration of time — not just
to monetize, but because it’s
a good user experience
”
25

26. ENGAGEMENT
IS
CONSIDERED A
DESIRABLE
GOAL
26
Healthy
participation in
the political
process
Civic
Positive consumer
relationships with a
company or a
brand
Consumer
Participation in
learning
experiences
Education

27. WHAT IS USER
ENGAGEMENT (UE)?
1. Users’ level of involvement or
investment in Information and
communication technologies (ICT)
2. Characterized by time, attention,
interest and emotion
3. The quality of the user experience
(UX) and the depth of the interaction
4. Associated with being captivated by
technology and so being motivated to
use it 27

28. THE USER ENGAGEMENT SCALE (UES)
28
Endurability
Perceived
usability
Focused
attention &
Felt
involvement
Novelty &
Aesthetic
appeal
Attribute-based approach for evaluating engaging
outcomes with a technology. Taps into six dimensions
of experience:
Novelty and aesthetics reflect users’ attraction to the media system or interface,
focused attention and felt involvement capture cognitive and emotional focus on
media content, and the endurability of system use represents evaluations of
success and voluntary participation to recommend the website to others.

29. UE DETERMINANTS
29
Task and
contexts
• Organizational/social setting
• Meaningfulness of the activity
• Voluntariness of use
User
• Internal state
• Predispositions
• Expectations, needs and
motivation
• Interest
System
• Complexity, purpose,
usability, functionality, etc.
• Characteristics and features.
• Interface (design and content)
Content
An important
determinant of UE
when interacting
with ICT

30. GAP ANALYSIS
• Theoretical
• Content is not well situated in
current theory
• The UE process model does not
emphasize content; Rather, it
focused primarily on the
characteristics of the system and
how these were perceived and
acted upon by users.
• O’Brien et al. ( 2016) suggested that
the user engagement framework be
broadened to incorporate content
• Empirical
• Content and its attributes are not
thoroughly explored as a
determinant of UE. (O’Brien, 2011)
• Practical
• Despite its importance, there
are no clear methodologies for
creating engaging content.
• Current research is focused on
identifying and evaluating UE,
but not on the actual
development of engaging
experiences.
• Designing for engaging
experiences is an oft-cited goal
of interactive system
development in many
disciplines, yet there are no
guidelines to create UE
30

31. FIELD
STUDIES
-
31

32. OVERVIEW
• Empirically test a functional relationship between content
variables (independent) and engagement variables
(dependent)
• Variables - Operational definitions, isolate one independent
variable, Control of extraneous variables
• Field experiments - the environment (field) is manipulated
in a controlled way
• Between-group design - two groups of subjects each
being tested simultaneously (between-subjects)

33. Experiment 1
33
Less is More: An
empirical investigation
of the relationship
between the amount
of digital content and
user engagement

34. INTRODUCTION
● Objective: Study user engagement
through the provision of information
● Specifically, an empirical investigation of
how the amount of content provided on
marketing web pages (often called
“landing pages”) impact users’
willingness to provide their e-mail
address (“conversion”)
● Pre-test (n= 535) and a large-scale field
experiment (n= 27,900) conducted
in real-world commercial settings
34

35. PROBLEM
Risk
• Uncertainty
• Trust and Credibility
• Potential value
• Relevance and usefulness
• Quality
Complexity
• Effort
• Ease of use
• Cognitive Load and
Limited Attention
• Curiosity
35
A pertinent debate among scholars and practitioners relate to the question of
information amount: Does more information elicit compliance and engagement, or
the other way around?

36. ELABORATION (MORE
INFORMATION)
• Cialdini, R. B., & Goldstein, N. J.
(2002). The science and
practice of persuasion.
• Brian J. Fogg. (2013). The Web
Credibility Project: Guidelines.
Stanford University
• Eisingerich, A. B., & Kretschmer,
T. (2008). In E-Commerce, More
Is More. Harvard Business
Review
Brevity (less
information)
• Norton, M. I., Frost, J. H., & Ariely, D.
(2007). Less is more: The lure of
ambiguity, or why familiarity
breeds contempt. Journal of
Personality and Social Psychology,
• Alter, A. L., & Oppenheimer, D. M.
(2009). Uniting the Tribes of
Fluency to Form a Metacognitive
Nation. Personality and Social
Psychology Review, 13(3), 219–235.
• Song, H., & Schwarz, N. (2010). If it’s
easy to read, it’s easy to do, pretty,
good, and true. Psychologist.
36

37. METHOD
a between-subjects design and
A/B testing.
• Created two versions of landing
pages:
•Control - Detailed information
•Treatment – no information
•Task (observed outcome) -
form prompting the users to
sign-up for a newsletter by
providing their e-mail address
(“conversion”)
Usefulness
Trust
Potential value
Quality
37

38. MESSAGE MATCHING
•Message Matching is the idea of
matching the pre-click message to the
post-click message on the landing
page.
•The goal of message matching is to
create a landing page that directs
individuals to a page on a product or
subject they are expecting.
38

39. DATA COLLECTION
●participants (n= 535 and n= 27,900)
were recruited using online
advertising (Google Adwords)
● Online randomized experiments
(also called A/B tests, split tests) -
Using Unbounce.com.
●Participants were randomly directed
to one of the variants.
● Relevant usage metrics, such as
response rate (called “conversion
rate”), location, and visit time were
recorded.
39

40. PILOT STUDY (N= 535)
Conversion results
40

41. EXPANDED EXPERIMENTS
New York
Atlanta
Miami
Los Angeles
National
27,083 unique visitors
41

42. EXPANDED EXPERIMENTS
42%29%
users who had less information were more
inclined to provide their data.
42

43. FINDINGS
Location
PAGE
VARIANT
TOTAL
UNIQUE
VISITORS
CONVERSIO
NS
CONVERSION
RATE
t
National
“A” (Long) 5497 1405 26%
-6.651**
“B” (Short) 5494 1718 31%
Atlanta
“A” (Long) 3690 1231 33%
-
14.560*
*“B” (Short) 3688 1838 50%
Miami
“A” (Long) 1235 444 36%
2.470**
“B” (Short) 1237 639 52%
Los Angeles
“A” (Long) 1945 529 27% -
13.817*
*“B” (Short) 1948 937 48%
New York
“A” (Long) 1176 324 28%
-8.946**
“B” (Short) 1173 528 45%
Total
“A” (Long) 13543 3932 29% -
22.166*
*“B” (Short) 13540 5660 42%
Both 27,083 9,593
Conversion results
demonstrate a negative
correlation between the
content amount and
consumer behavior
Significant advantage to the
pages with less content
(treatment)
43

44. Short
Long
The shorter landing
pages had
significantly higher
conversion rates
across all
locations and
times.
44

45. DISCUSSION
1. Information impact behavior
• Paradoxical relationship between the increased level of
information and online engagement
• Users that did not have any information on the service, its value, or how their
data will be used still tended to convert more.
• By contrast, users who were provided with this information were less inclined
to convert.
2. Contrary to previous work in offline contexts, not all engagement
theories are effective online
3. While not enough to establish a linear relationship, results
suggesting that “less is more” can go a long way in creating
commercial value. 45

46. Experiment 2
"Sticky" words:
Improving user engagement using
automatic lexical substitution

47. Introduction
● Objective: influence user engagement by systematically
changing just one word on the interface
● Identify engaging words (“sticky words”) based on pre-defined
attributes
● such words are not "free-standing;" When strategically placed,
they impact their surrounding context and create a more
engaging user experience (UX)
● Lexical substitution is the task of identifying a substitute for a
word in the context of a clause (using semantic analysis, natural
language processing and a rule-based approach)

48. Method
Operationalized based on three key attributes of “sticky words”.
Novelty - Out-of-context, Low frequency (distinctiveness) in
academic titles
Familiarity - Popularity, High frequency on Imdb.com
Emotional value – imdb.com, Sentiment Analysis
Example:
Identify Influential Factors of Abandoned Buildings
Identify Influential Factors of Zombie Buildings

49. METHODOLOGY
49
Derive a list of
potential Sticky
Words from
IMDB.com (the
Internet
Movie Database)
Classify based
on polarity
using sentiment
analysis
analyze
academic titles
to find and
replace
synonyms

50. METHODOLOGY
50
A baseline prototype to operationalize the
process using the Natural Language Toolkit
(NLTK) and the WordNet Interface. Derived
a list of potential Sticky Words from
IMDB.com keywords. Used Lexical
substitution to revise academic titles.
Goal: Automatic, systematic identification
and replacement

51. 51
Identification of zombie homes using system dynamics
1
2
3
4
5
End of the library: Organized information and digital ubiquity
Death of library: Organized information and digital ubiquity
Should reproduction and adoration be related in HIV research?
Should sex and love be related in HIV research?
Identification of vacant homes using system dynamics
Grappling cancer treatment using new types of treatment
Be an exemplar: Effective civic engagement through open data and digital government
Fighting cancer using new types of treatment
Be an action hero: Effective civic engagement through open data and digital government

52. DATA
COLLECTION Apilotstudy
(Qualtrics.com)
One-way Two-group design
- two variations of the same
titles (one change per title)
Online user studies –
college student (n=405)
Randomized A/B test
(between-subjects)
Measured:
Choice (awarness,
attention)
Affect (UES)
Memorability
52

53. 33%
49%
37%
64%
30%
76%
24%
70%
41%
65%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Be an exemplar: Effective civic engagement
through open data and digital government
Be an action hero: Effective civic
engagement through open data and…
Grappling cancer treatment using new
types of treatment
Fighting cancer using new types of
treatment
End of the library: Adaptable organized
information and digital ubiquity
Death of library: adaptable organized
information and digital ubiquity
Should Reproduction and adoration be
related in HIV research?
Should sex and love be related in HIV
research?
Identifying vacant homes using system
dynamics
Identifying zombie homes using system
dynamics
53

54. 1.33
3.49
2.37
3.64
1.30
3.76
1.24
4.70
2.41
3.65
1.08
4.96
2.11
4.67
1.97
4.62
1.43
4.27
1.36
3.77
1.76
3.17
1.05
3.25
1.25
4.39
1.66
3.27
2.28
4.44
1.89
4.30
1.63
4.91
1.53
4.70
1.74
4.25
2.62
3.36
0.00
1.00
2.00
3.00
4.00
5.00
6.00
Be an exemplar:
Effective civic
engagement
through open
data and digital
government
Be an action
hero: Effective
civic
engagement
through open
data and digital
government
Grappling
cancer
treatment using
new types of
treatment
Fighting cancer
using new types
of treatment
End of the
library:
Adaptable
organized
information and
digital ubiquity
Death of library:
adaptable
organized
information and
digital ubiquity
Should
Reproduction
and adoration
be related in HIV
research?
Should sex and
love be related
in HIV research?
Identifying
vacant homes
using system
dynamics
Identifying
zombie homes
using system
dynamics
This article is interesting
I believe that reading this article is worthwhile
This article is easy to understand
This article is appealing, I wish to read more
54

55. 55Information retention

56. data love sex
governm
ent
hero hiv zombie homes library cancer
Freq 8 8 10 12 12 12 12 16 17 32
8 8
10
12 12 12 12
16
17
32
0
5
10
15
20
25
30
35
56

57. SIGNIFICANCE
• Setting criteria for effective (engaging) content
strategy (message) is relevant to various domains
(Communications, Marketing, Information Science,
Public Policy, Education…)
• Methodological - Definitions of user engagement, an
original process-based model, Identify determinants /
characteristics that can be isolated and measured
• Operational - Develop a framework for
systematic evaluation, optimization, and creation of
engaging content
➊
➋
➌

58. •Introduction and Scope:
•Question: How digital content influence user engagement
•Aim: Develop a framework for systematic evaluation, optimization,
and creation of engaging content
Litrature review - Theoretical Foundations
● Definitions of digital content and user engagement
● Original process-based model
● Characteristics / Determinants
•Field experiments
● Less is more - amount of content
● Sticky words - Automation, using Lexical substitution is the task of
NLP to increase engagement identifying a substitute for a word in the
context of a clause
Developing Framework
● Content Strategy, Heuristic evaluation

59. QUESTIONS?
59
Nim Dvir
ndvir@Albany.e
du
@nimdvir
University at
Albany
State
University of
New York
(SUNY)