How to get better results from a survey: Meet the Survey Octopus

How to get better
results from a survey
Caroline Jarrett
@cjforms
Content Strategy Summit 2015 #CSSummit

Caroline Jarrett @cjforms (CC) BY SA-4.0
I’m a forms specialist
2
Image credit: Flickr, taxrebate.org.uk

Why do people answer questions?
3Caroline Jarrett @cjforms (CC) BY SA-4.0

A dollar bill with a mail survey works
better than $10 (guaranteed) later
4

Response relies on
effort, reward, and trust
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram from Jarrett, C, and Gaffney, G (2008)
“Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
5

6

7
The aim of a survey is to get a number
that helps you to make a decision

8
To get better results from your survey,
think about the Survey Octopus

People ask me about surveys
• “How many people do I need to ask?”
• “How many questions can I have?”
• “Please tell me whether this is a good question”
• “I think it’s best to have 5 points in my rating scale, but
my boss wants to have 7. Who is right?”
9

Agenda “How many people do I need to ask?”
“How many questions can I have?”
“What makes a good question?”
“How many points in a rating scale?”
10

To find out how many people to ask,
start at how many we need to answer
Fieldwork:
Who answers?
11

Whether they’ll answer depends
on effort
Questions:
What are you
asking about?
How many
questions?
12

And on the reward you’re offering
Goals:
Why are you asking?
Is helping you a reward in itself?
Are you offering any other
incentive?
13

Then there’s the
‘Justin Bieber North Korea’ problem
14
http://www.bbc.com/news/10506482

We don’t just want answers,
we want answers from the right people
Response
15

So it matters where
we get our sample from
Sample:
the list you
sample from
16

17
And now it’s easy to work out
how many to ask
Sample:
the number of
people to ask

We thought about a lot of topics
to work that out
Goals
Sample
Questions
Fieldwork
Response
18

To get really good results, we want
useful answers from the right people
Response
Response
19

“You're using this site from outside the
UK. Where are you answering from?
We’d like answers like these
– Anchorage Alaska U.S.A.
– Cameroon
– Dubai, United Arab Emirates
Even better, extra-interesting answers like these:
– Cote d'Ivoire / France / UK.
– Canada, originally from the UK
– Philippines, would like to work in UK with my husband if possible
But definitely not like these:
– ä¸å›½
– its not important
– nnnnnn
– none of your business 20

Insights are the numbers
that you use for decisions
Insights 21

We’ve thought about a lot of issues
Goals
Questions
Sample
Fieldwork
ResponseResponse
Insights 22

To get better results from your survey,
think about the Survey Octopus
23

“What sorts of questions are best?”
24

Let’s work backwards from insights
25
Which answers do you need to make the decisions?

You need accurate answers
to the questions
26
Will your respondents give you
good answers to all of them?

27
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
available online services (not including
email, instant messaging, and search) to
do your work using a work computer or
other device?
%

28

29
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
https://commons.wikimedia.org/wiki/File:Pho
ne_photography.jpg#/media/File:Phone_phot
ography.jpg
Modified by Caroline Jarrett

You need questions that your
respondents can answer accurately
Which
questions
will get you
the answers
that you
can use?
30

Now we get to fieldwork
31
How many
of those
questions
are people
willing to
answer?

Fieldwork used to be expensive
so a survey was a rare event.
32
Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/

1950s mindset: “Ask Everything”
33
Survey =
Big Honkin’ Survey

2015 mindset: the Light Touch survey
• Choose ONE question
• Find ONE person
• Ask the question, face-to-face
• See if you can make ONE decision
• Improve, iterate, increase
34

Tip Interview first
35

Time for new question
36
One way to
iterate,
improve,
increase

• Negotiate to get survey
down to 20 questions
• Ask 10,000 people
• Get 1,000 responses
• Take a week (or more) to
analyze the responses
• Have a big presentation
a month later
Light touch survey
• ‘Question of the day’
(one question)
• Ask 100 people
• Get 50 responses
• Analyze them same day
• Present same day
• Repeat 4 x 5 = 20 times
37

Do both!
• Big numbers are
impressive
• Can compare answers
from different segments
• Easier ‘sell’ to
stakeholders
Light touch survey
• Quick, useful results
• Rapidly get better
at doing surveys
• Wonderful way to test the
questions for the Big
Honkin’ Survey
38

39

Caroline Jarrett @cjforms
40
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
https://commons.wikimedia.org/wiki/File:Pho
ne_photography.jpg#/media/File:Phone_phot
ography.jpg
Modified by Caroline Jarrett

Tip
Always allow for ‘other’
Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk
41

42
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
available online services (not including
email, instant messaging, and search) to
do your work using a work computer or
other device?
%

Response relies on
effort, reward, and trust
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram from Jarrett, C, and Gaffney, G (2008)
“Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
43

44
A good question works in three ways
Appropriate
Obvious Interesting

45
Why did you visit our website today?
Appropriate
Obvious Interesting

Would you recommend us to
a friend or family member?
46
In a shop,
buying a
baby carriage
In a hospital,
having a
miscarriage
Obvious Yes
Interesting Yes
Appropriate Yes Cruelly
inappropriate

Tip
Test your questions by
interviewing in context
47

48

Likert had several different types
of question in his response formats
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55.
49

You can find an academic paper to
support almost any number of points
• Krosnick and Presser refer to over 80 papers
50
Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design.
Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier.
http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf

Tip
Don’t stress too much about
the number of points in your
rating scale
51
Picture credit: Flickr - Bill Soderman (BillsoPHOTO)

Well, OK, stress a little bit.
52
This scale is
downright
peculiar. Avoid.

Summary
53

54
The aim is to get the best number you
can, within the resources you have

What you want to ask about
The resources you have
The questions you ask
The answers you get
The answers you use
The number
Who you want to ask
The list that you sample from
The sample you ask
The ones who answer
The ones whose answers
you can use
55

What you want
to ask about
The resources
you have
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use
The number
56

57
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
Resources
What you want
to ask about
The resources
you have
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use

58
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.

Caroline Jarrett
Twitter @cjforms
http://www.slideshare.net/cjforms
carolinej@effortmark.co.uk
59

How to get better results from a survey: Meet the Survey Octopus

Recommended

Recommended

More Related Content

What's hot

What's hot (16)

Similar to How to get better results from a survey: Meet the Survey Octopus

Similar to How to get better results from a survey: Meet the Survey Octopus (19)

More from Caroline Jarrett

More from Caroline Jarrett (11)

Recently uploaded

Recently uploaded (20)

How to get better results from a survey: Meet the Survey Octopus

Editor's Notes