Presentation of Holger Lütters during General Online Research Conference 2015 in Cologne, Germany.
While the majority of the online market research industry is still inviting people over the email channel, some new approaches of using advertising networks to recruit people literally pop-up.
Besides the largely discussed approach of Google consumer surveys, using the Google advertising eco-system on desktop and mobile devices, a new and independent approach has been developed by a Berlin start-up company. This approach allows inviting unknown people over an advertising partner’s website to take a survey.
The market researcher’s question at this point of technology development is, whether this approach can be considered to create valid data compared to the classic panel approach. In order to find out about the opportunities and problems of this recruiting procedure the standard panel approach was used in competition with the advertising recruiting approach.
Methods & Data:
The comparison of the competing data collection approaches was supervised by independent academic researchers and supported by the commercial online panel GapFish and the advertising recruiting approach as developed by Dalia Research. Overall more than 3.000 people were contacted to create an online sample of more than 800 respondents for each approach equally covering desktop, laptop, tablet and smartphone devices allowing to compare the results.
The device agnostic questionnaire was dealing with respondent’s behaviour when answering questionnaires and their expectations in compensation for taking questionnaires in the classic and the mobile environment.
The project was one of the interactions during the HTW Berlin / HWR Berlin Project InnoRadar. InnoRadar is supported by the Berlin Institute of Applied Research (IFAF) www.ifaf-berlin.de
Communication Accommodation Theory Kaylyn Benton.pptx
Online Panel vs. Mobile Advertising Recruiting in Market Research Study
1. General Online Research Conference 2015 – GOR15
18-20 March 2015, Cologne University of Applied Sciences, Germany
Holger Lütters – Hochschule für Technik und Wirtschaft Berlin
Suggested citation: Lütters, Holger (2015): New survey recruiting strategies: Online Panel vs. Mobile Advertising. General Online Research (GOR) Conference, Cologne.
Data Report available under http://goo.gl/K6xH4t
Contact: holger.luetters@htw-berlin.de @luetters
New survey recruiting strategies:
Online Panel vs. Mobile Advertising
This work is licensed under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/)
2. 1
Study set-up and field report
Empirical study in comparison of two market
research recruiting approaches:
Online Panel vs. Mobile Ad
3. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 3
• Questionnaire development
• Online Panel
• Mobile Ad
• Questionnaire Technology
Berlin based market researchers teamed up for the research project
Berlin Market
Research Initiative Field report
• 3029 contacts
• 1732 complete answers
• 8 minutes questionnaire with
questfox® using Gyroscale®
• Random distribution of
demographics to determine
differences in sampling
• Quota on devices
• Survey in July 2014
by
4. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 4
Two recruiting approaches competing with one identical
questionnaire
GapFish
• (Classic) Online Panel
www.entscheiderclub.de
• 110.000 participants
• Germany, Switzerland, Austria
• Advertising sampling approach
• Global reach
(200 million potential users)
• All languages possible (need of
developed mobile advertising
market structures)
Invitation on third
party ad platform
Generic invitation
Real-time screening
Re-Direct to third
party survey
(Non-)Monetary
Incentive
5. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 5
Methodological question: What do participants in studies of
(mobile) market research surveys expect from research?
Methodological issues:
• Effects of different sources on
the data?
• Effects on non-reactive data
(response time, willingness to
participate, etc.)?
• Effects of different methods
used on different devices (AHP,
PSM, GyroScale)
General interest in
• Expectations of users towards
mobile surveys
• Decision-making factors of
survey participation
• Expectations of compensation
for participation in (mobile)
surveys
6. Online Panel vs. Mobile Ad Recruiting
Sociodemographics, Devices and Non-Reactive Data
2
7. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 7
Target per approach n=800 spread across four device categories
with a quota of 200 each
8. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 8
The two approaches reached a similar, structurally representative
regional population pattern without quota setting
Same questiontype for all device categories; Launch from Berlin
Online Panel GapFish (n=888) Mobile Ad Dalia Research (n=811); Data BRD DESTATIS 2012
10,7%
12,5%
7,8%
4,7%
0,5%
2,4%
7,2%
2,0%
5,2%
22,9%
3,6%
1,6%
6,2%
3,5%
5,4%
3,5%
9,9%
16,2%
7,2%
2,5%
1,0%
2,6%
5,7%
2,7%
9,7%
21,9%
4,7%
1,2%
4,9%
2,1%
3,5%
2,8%
13,1%
15,5%
4,2%
3,0%
0,8%
2,2%
7,5%
2,0%
9,7%
21,8%
5,0%
1,2%
5,0%
2,8%
3,5%
2,7%
Baden-Württemberg
Bayern
Berlin
Brandenburg
Bremen
Hamburg
Hessen
Mecklenburg-Vorpommern
Niedersachsen
Nordrhein-Westfalen
Rheinland-Pfalz
Saarland
Sachsen
Sachsen-Anhalt
Schleswig-Holstein
Thüringen
Online Panel
Mobile Ad
BRD
9. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 9
Mobile Ad approach is significantly less representative for
sociodemographics, reaching younger and more male participants
Age Distribution Gender
Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research (n=811); no quota settings
Female
Male
Female
Male
Female
Male
Mobile
Ad
Online
Panel
Average Age
Mobile
Ad
Online
Panel
10. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 10
Both approaches are not representative in terms of education.
Mobile Ad approach achieves higher educational levels
Higher Education
A-Level
vocational school
comprehensive school
secondary school
Without formal degree
No answer
Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research (n=811); no quota in question
11. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 11
“Couch surfing” as a standard pattern in mobile research
4 out of 5 participants in the study are mobile …but at home
At Home
Work / School / University
Public Place
On the road
None of these places
12. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 12
Technographics: Mobile Ad participants have a slightly more modern
equipment, use a broader variety of browsers
but do not seem to be more tech savvy in general
13. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 13
Mobile participants tend to speed through a questionnaire
Panel participants take much more time on a mobile device
Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research (n=811); Information in seconds for complete interviews
Speeder
Mobile Ad
Online Panel
Response time
in seconds
14. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 14
The two approaches show different behavioral patterns
Criteria Online Panel Mobile Ad
Contacts 1912 1032
Complete Interviews 888 811
Average Completion Rate 79,5% 82,9%
Quota reach Complete Incomplete
Duration 7 days 13 days
Response Time Completes 497 seconds 441 seconds
Response Time including drop-outs 296 seconds 383 seconds
Overall time used in seconds 565.952 395.256
Drop-out pattern Decision in the first minute
(independent from device used)
Slightly higher drop-out during
the entire interview (independent
from device used)
Very good and good consistency AHP (CRH< 0,2) 29,5% 29,2%
Inacceptable consistency AHP (CRH> 0,5) 24,0% 29,0%
15. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 15
Conclusion non-reactive data: Strong differences in response
behavior: Mobile participants respond faster (even on the desktop)
Online Panel
● Short overall project time
● Intensive use of panelists
● Slightly more drop-outs
● Exact quota reach
● Longer interview duration
● Smartphone interview much
longer
Mobile Ad
● Double overall project time
● Used less participants
● Slightly less drop-outs
● Quota not reached for classic
devices
● Speeding behavior: All
interviews are answered
quicker on all kinds of devices
17. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 17
The online panelists have much more experience with surveys of
market research compared to the mobile participants
Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research(n=811) und n=31 professionelle Marktforscher
Mittelwerte einer Skala von 0 (überhaupt keine Erfahrung bis 10 extrem viel Erfahrung)
Mean:
Online Panel 6,63
Mobile Ad 3,72
18. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 18
¾ of the respondents prefer the (classic) online survey
Participants of the mobile recruitment are less enthusiastic about
online surveys (and could live without any kind of survey)
Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research (n=811) und n=31 professionelle Marktforscher
19. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 19
Fair compensation for entertaining – but short - interviews are
demanded. The Mobile Ad group has lower or no expectations
Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research (n=811) und n=31 professionelle Marktforscher
Mittelwerte einer Skala von 0 (trifft überhaupt nicht zu bis 10 trifft voll und ganz zu)
20. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 20
Compensation and duration are the most important factors
influencing the willingness to participate in a survey
Questions asked in pairwise comparisons with Analytic Hierarchy Process
21. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 21
The majority of online panelists expect the same compensation for
a participation in mobile market research
Frage nur an n=879 Teilnehmer Online Panel GapFish und n=30 Teilnehmer MaFo Profis
Frage nach Kompensation/Kosten im Vergleich zu klassischer Online-Befragung
22. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 22
The mobile participants expect a form of compensation derived from
the context of their recruitment into the study
23. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 23
Price Sensitivity Meter to measure a fair compensation for
participating in a mobile study
Imagine the following invitation to a survey:
Topic: Shopping; Duration: 4 minutes; participation only to mobile device
(smartphone / tablet)
24. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 24
Participants express high hopes for the incentive of a 4 minute
mobile interview which is not backed-up by the willingness to pay
of Market Research Professionals
Online Panel: 3,56 €
Mobile Ad: 4,52 €
MR Professionals: 0,94 €
Optimal Price Point (VanWestendorp)
25. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 25
The identical survey is evaluated very differently by the different
approaches
Base: Online Panel GapFish (n=888); Mobile Ad Dalia Research(n=811)
Measurement Net Promoter Score 0=would not recommend at all .. 10 totally would recommend
NPS: 0-6 Detractor; 7-8 Neutral; 9-10 Promotor
26. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 26
Mobile participants are shockingly honest about lying
... who is less honest?
Basis: Online Panel GapFish (n=888); Mobile Panel Dalia Research(n=811) und n=31 professionelle
Marktforscher; Mittelwerte einer Skala von 0-10 unter Anwendung NPS: 0-6 gelogen; 7-8 neutral; 9-10
total wahrheitsgemäß)
27. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 27
Mobile research is happening anyway.
We better find ways to manage it appropriately
Mobile Ad approach works
• Dalia Research developed a new
technology which offers more
options to the researcher
• The approach can be as
appropriate as it can be wrong
for a specific topic
• The reach of fresh and
unexperienced target groups is
an interesting option
• Complicated approaches with
special methodologies might
not work (due to the screen size
and due to the user)
Things to consider when using the
Mobile Ad recruiting approach
• Mobile takes more time in
creation, testing and surveying
• Might be more expensive
(overall)
• Mobile users have different
expectations which are not
necessarily the same as before
• Researchers should not lower
their standards in order to
please the new target groups
28. Holger Lütters · New survey recruiting strategies · #GOR15 · Cologne 20th March 2015 · @luetters 28
Author:
Holger Lütters – HTW Berlin
Contact information
Prof. Dr. Holger Lütters
Hochschule für Technik und Wirtschaft Berlin
Fachbereich 3 Wirtschafts- und Rechtswissenschaften
Treskowallee 8
D 10318 Berlin -Germany
Raum TA VG 733
Holger.Luetters@htw-berlin.de
www.luetters.com
Twitter @luetters
xing.com/profile/Holger_Luetters
de.linkedin.com/in/luetters/
The project was one of the interactions during the HTW
Berlin / HWR Berlin Project InnoRadar.
InnoRadar is supported by the Berlin Institute of Applied
Research (IFAF) www.ifaf-berlin.de.
Thank you for your interest in the study
Data live report available under http://goo.gl/K6xH4t
Thanks to the supporters of the study:
www.GapFish.com
www.DaliaResearch.com
www.pangealabs.com
Gyroscale® is a development of pangea labs GmbH Switzerland and
is available in the SaaS survey tool questfox® www.questfox.com
Editor's Notes
1
Sampling over advertising platforms is a technical challengeDalia Research offers a thrilling approach (even compared to Google)
First presentation of the use of the Gyroscale: Idea and first studyIn this session the results of the first test with the Gyroscale are presented
The Mobile Ad is generally a little more mobile (but also much at home)
Mobile Ad participants are using more recent tech equipment and less Apple devices
Online Panel 6,63
Mobile Ad 3,72
während die Mehrheit der Profis keine erhöhten Kosten durch mobile Marktforschung kommen sieht