Presentation delivered at the Canadian Evaluation Society Conference in Toronto in June 2013. The results and process behind the development evaluation and outcome evaluation with the Break It Off social media campaign for youth smoking cessation were discussed.

1. Evaluation of Break It Off:
a Social Media-based Campaign
Targeting Young Adult Smokers
Cameron D. Norman1
N.Bruce Baskerville2 ,Nancy Korstanje3
Steve Brown2, Barbara Zupko2 ,Larry Squire2
1 CENSE Research + Design & Dalla Lana School of Public Health, University of Toronto
2Propel Centre for Population Health Impact, 3The Canadian Cancer Society
Canadian Evaluation Society Conference, Toronto, ON June 2013

2. BACKGROUND

3.  Young adults (19-29) have the highest rate of smoking in
Canada – 24.4% use Tobacco (CCHS, 2010)
 86% of young adults (18 to 34 years) in Canada are using
social media tools – e.g Facebook, Twitter, etc. (Ipsos
Reid, 2011)
 Social media tools are immensely popular, yet not well
studied in terms of population health impact
 Evidence to-date suggest wide reach, low barriers to use,
and portability → great potential for population level
impact
3

4. Any electronic, networked information resource
that derives its principal value from user
contributions, engagement & interaction

5. Developmental Evaluation

6. Developmental Evaluation
• First proposed by Michael Quinn Patton
(1994)
• Combines Utilization-Focused
Evaluation with concepts from
Complexity Science
• Social innovation and resiliency require
actionable feedback to enable programs
to adapt and thrive through change
• Strategic learning / real-time evaluation

7. Developmental Evaluation
• Assists social innovators to learn
strategically
• Supports adaptation to changing
conditions
• Develops promising interventions
• Documents innovators’ actions
• Identifies emergent processes and
outcomes; enhances resiliency
• Determines if an innovation is ready for
formative evaluation

8. Systems Thinking
• Knowledge structures can be built upon:
– Information: symbolic pattern of matter and
energy
– Networks: the information delivery channels and
attendant limits
– Systems: Bounded elements linked to facilitate
the flow of information and support networks

11. Break
It
Off
11
 Promoted January 2012 to March 2012
• Paid advertising on websites with high levels of young
adult viewership.
• Banner ads on sites such as Microsoft, Yahoo and
Facebook.
• Homepage takeovers on the Weather Network.
• Ads on dating sites such as Plenty of Fish and OK Cupid.
• Facebook, Twitter, YouTube videos and blogs.
 Links to Smokers’ Helpline

12. What are Quitlines?
 Quitlines are computer-supported, telephone-based
tobacco cessation counselling services that help tobacco
users quit involving a live, trained cessation counsellor.
 Quitlines reach and serve tobacco users, regardless of
location.
 Residents in all 50 states, the District of Columbia, each
U.S. territory, all ten Canadian provinces and two
territories have access to public Quitline services.
 Smokers’ Helpline (SHL) is an evidence-based cessation
service managed by the Canadian Cancer Society.
 SHL reach into the young adult smoking population is
estimated to be just 0.5%.
12

13. Developmental
Evalua4on
Research
Approach
13
Step
5:
Act
In
collabora.on
with
CCS,
synthesis
and
transla.on
of
knowledge
into
publica.on,
programming
and/or
future
evalua.on
research.
Step
1:
Listen
Interact
with
CCS
to
iden.fy
their
needs
and
wants
and
build
a
shared
understanding
of
the
context,
objec.ves,
challenges
and
opportuni.es
associated
with
SM.
Step
2:
Plan
Involve
the
user
community
via
focus
groups
to
incrementally
define
and
develop
the
SM
interven.on
plan.
Step
3:
Do
Flowing
from
CCS
needs
and
user
input,
implement
the
plan
and
finalize
the
evalua.on
research
framework
(data
collec.on
methods)
to
support
it.
Step
4:
Study
Collect
data
to
assess
implementa.on
progress
and
outcomes
and
regularly
report
to
CCS
on
results
to
inform
decision-­‐
making.
Development Cycle

14. A mixed-methods approach to
data collection was used to
answer the following general
questions:
Developmental
Evaluation Questions Question #1 – How did the Social media (SM) campaign
develop over time and what evolved in its implementation?
 Question #2 – What was learned in the pursuit of population
reach using an SM-based tool with young adults?
 Question #3 – How did the SM campaign change quitting
success and influence Quitline utilization?
 Question #4 – What influence did the social network have
on quitting outcomes and youth engagement?

15. Plan, Act, Evaluate
Plan, Act, Evaluate
Plan, Act, Evaluate
“ready, fire, aim” approach to innovation
Gamble, J. (2006). A Developmental Evaluation Primer
Montreal, PQ: J.W.McConnell Foundation
Developmental vs. Traditional Evaluation Approach
Overlapping sequences of developments / evaluations

16. 16
Step
2:
Plan
Involve
the
user
community
via
focus
groups
to
incrementally
define
and
develop
the
SM
interven.on
plan.
ROUND
I
focus
group
results:
Barriers
to
quiKng/triggers
 Friends
who
smoke
 Socializing
and
drinking
 Rou.nes
that
include
smoking
 Stress
Solu4ons
 Change
rou.nes
 Replace
with
healthy
habits
 Support
of
friends
Reac4ons
to
Break
it
Off
 Visually
appealing
 Liked
the
“break-­‐up”
metaphor
 Interac.ve
elements
important
 App
was
well-­‐received
“…
it’s
kind
of
a
clever
idea,
the
whole
breaking
off
a
rela9onship
thing,
and
I
think
a
lot
of
people
will
relate
to
that.”
“I
think
that
one
major
factor
in
quiBng
smoking
is
to
just
know
that
you’re
not
alone
…
a
lot
of
friends
of
mine
quit
smoking
all
at
the
same
9me.”
“I
really
like
the
tracking
system
where
you
could
see
if
you
did
slip
up,
why
you
slipped
up
and
then
be
able
to
go
back
and
look
at
your
record
of
how
you
were
progressing
or
not
progressing.”

17. Data collection:
 Minutes of meetings, SHL/SHO administrative data, and web
analytic data collected.
 11 young adult smoker focus group sessions (Ontario,
Manitoba and New Brunswick) - 7 before the launch of the
SM campaign and 4 after.
Analysis:
 Descriptive quantitative analysis of user activity.
 Grounded Theory - Constant comparison method for
qualitative focus group data.
Measures:
 Ongoing monitoring of decision-
making, tracking of activity, SM
usage and call volumes.
 Young adult input on challenges
to quitting, potential solutions,
use of social media, and campaign
feedback.

18. Target population: English and French
speaking smoking adults 19-29 years of
age
Outcome Measures:
 Quitline Reach - proportion of eligible
smokers in the target population who
register for SHL/SHO - reach into the
young adult smoking population is
currently estimated to be just 0.5%.
Data Collection and Analysis:
 SHL/SHO administrative data on
registrants and annual Canadian
Tobacco Use Monitoring survey.
 Reach seven months prior to the SM
campaign launch and seven months
calculated with sub-group
comparisons.
Reach * Effectiveness = Impact

19. Sustainability of the SM
Campaign supported by:
Transla4on
 An iterative, developmental approach,
 Partnership with CCS using continuous
knowledge exchange,
 Embedding of the program within CCS’
suite of cessation programs
19
Step
5:
Act
In
collabora.on
with
CCS,
synthesis
and
transla.on
of
knowledge
into
publica.on,
programming
and/or
future
evalua.on
research.

20. Break
it
Off
-­‐
Real-­‐4me
Results
Number of people reached through the
Break it Off web-site and level of activity
 37,325 unique visitors
 44,172 visits
 107,600 page views
 2.44 pages per visit on average
 3,937 installations of the Break It
Off smartphone app
20

21. Results
-­‐
SM
Performance
ROUND
II
focus
group
results
of
BIO
Users:
 Two
focus
groups
conducted
to
date
(June
2012)
 Site
and
app
were
easy
to
use,
visually
appealing,
and
suitable
for
audience
 App
was
used
more
than
the
website
 Using
app
to
track
progress
and
triggers
was
helpful
 Mixed
percep.ons
on
SHL
 Sharing
quit
status
on
Facebook
can
create
a
suppor.ve
environment
21
“I
posted
and
I
got
a
lot
of
good
feedback
on
it
so
that
helped
me
support-­‐wise
to
know
that
a
lot
of
people
actually
did
care
whether
I
quit
smoking.”
“The
app
helped
me
realize
what
my
triggers
were…so
then
I
could
stop
them
before
they
even
happened
and
just
kind
of
focus
on
something
else
or
keep
myself
busy
just
to
get
past
that
ini.al
craving
and
then
I
was
fine.”
“The
app
helped
me
stay
focused,
to
see
the
number
of
days
smoke-­‐
free
rise
every
day
was
key
for
me.”
Break
it
Off
-­‐
Real-­‐4me
Results

22. Did
web
traffic
increase
over
4me?
The
campaign
was
launched
on
Jan.
12,
2012
and
remained
fairly
constant
un.l
funding
for
promo.on
ran
out
at
the
end
of
March.
Spikes
in
traffic
can
be
aributed
to
promo.onal
ac.vi.es
such
as
the
home
page
takeover
of
the
Weather
Network.
• Visits
per
day
22
Break
it
Off
-­‐
Real-­‐4me
Results

23. Where
were
users
geographically
located?
The
following
ci.es
provided
the
most
visits:
• Toronto*
-­‐
8,363
• Saskatoon*
-­‐
3,764
• Regina*
-­‐
3,096
• Oawa*
-­‐
2,288
• Winnipeg
-­‐
1,400
• Hamilton*
-­‐
1,229
*Provinces
with
expanded
promo.onal
BIO
campaigns
23
Break
it
Off
-­‐
Real-­‐4me
Results

24. Were
people
being
exposed
to
the
resources?
• 2,208
new
Facebook
page
likes
(www.facebook/smokershelpline)
since
January
2012
24
9,810
5,785
4,065
3,969
3,959
0
2,000
4,000
6,000
8,000
10,000
12,000
Break-­‐up
methods
Break-­‐up
videos
Smartphone
App
Get
Over
With
It
The
Ugly
Truth
Page
views
(Jan
12
to
Mar
31,
2012)
Break
it
Off
-­‐
Real-­‐4me
Results

25. Campaign
Results
25
Quit Rates
7 day PPA at 3 months – 47%
30 day PPA at 3 months – 32%
Quit Actions
Made a quit attempt – 91%
Cut down amount smoked – 89%
Set a quit date – 66%
Satisfaction
Very or mostly satisfied with the campaign – 77%
Unintended Consequence
Only 21 visitors out of 37,000+ connected to Quitline

26. Results
-­‐
Reflec4on
Reflections on Developmental
Evaluation
Critical success factors
 Having the right people involved
 Partnership, open dialogue, relationships
 Expertise in social media
 Goal congruence
 Adaptable and agile
 Social Innovative nature was appealing
 Enthusiasm for the project
26

27. Lessons
Learned
&
Next
Steps
Lessons Learned:
 Flexibility and adaptation are key
 “It’s about relationships”
 Continuous learning
 Organizations are not set up for development,
just outcomes… development-in-action is
challenging
27

28. Conclusion
28
Implications for Practice
 Break It Off was effective in helping young adults
quit smoking.
 Compares favourably to other digital social
media interventions - “Happy Endings” 44.7%
quit rate (7 day PPA) versus “Break it off” 47%
quit rate (7 day PPA) at 3 months
 Break It Off is reaching smokers who are not
using cessation supports such as SHL
 An integrated tobacco cessation system should
consider the role of social media based
interventions and the implications of
technological change on service offerings

29. Conclusion
29
Implications for Research & Evaluation
 Formative evaluation research on social media -
e.g. What seems to be working, not working,
reasons for use, etc.?
 Implementation science - e.g. What are the key
components that account for change in
behaviour for social media?
 Paucity of evidence on effectiveness and cost-
effectiveness of social media interventions for
decision-makers.

30. Cameron D. Norman PhD
CENSE Research + Design
cdnorman@cense.ca
@cdnorman
N. Bruce Baskerville PhD
Propel Centre for Population Health
Impact, University of Waterloo
nbbasker@uwaterloo.ca