1. Pattern-Based Specification of
Crowdsourcing Applications
Alessandro Bozzon (TU Delft)
Marco Brambilla (Politecnico di Milano)
Stefano Ceri (Politecnico di Milano)
Andrea Mauri (Politecnico di Milano)
Riccardo Volonterio (Politecnico di Milano)

2. Crowdsourcing and Human Computation
It works like magic!
Endless Applications
Endless Success Stories
2008 Olympics Opening Ceremony

3. Actually…
Often a Try&HopeError process
Task Design Matters
Crowd can be unreliable
($) Incentives Matter
Quality Control Matters
Platform of Execution Matters

5. Setting Nº1
Global Annotations with
simple counting

6. Setting Nº2
Local Annotations
with Bounding Boxes

9. Setting Nº3
Local Annotations with Verified Bounding Boxes

10. Setting Nº3
Local Annotations with Verified Bounding Boxes

11. Setting Nº3
Local Annotations with Verified Bounding Boxes

12. Ok, so what?
#Workers
#Useful
Workers
#Executions Cost $
Time
(hours)
Precision
Setting
Nº1
732
44
(6%)
488 72 ~40 ~67%
Setting
Nº2
498
25
(5%)
547 48 ~169 ~67%
Setting
Nº3
1420
464
(32%)
3387 83 ~184 ~75%
Total 2152 508 4422 203 ~16 days

13. Our study

14. Our study
Our Contribution

15. GOAL
!
Simplify and systematize the design, deploy, and
monitoring of applications (including experiments)

16. Contributions
An Abstract
Model of
Crowdsourcing
Activities
A Composition
model for
Crowdsourcing
Activities
A Library of
crowdsourcing
Patterns
• A conceptual framework
• A specification paradigm
• A reactive execution control
environment

17. Models

18. DEMO VIDEO

19. Crowd Task
[T operation types]
(intra-task patterns)
Object Type
block size
min #obj
(cons)
input buffer
batch flow (on closed task)
stream flow (on closed object)
MicroTask
[MT operation types]
r
data manipulator

20. Case Study: Movie Scene Analysis
Scenario 1: Scene Positioning
Spoiler Alert!
Order Scenes
Scene in Beg/Mid/End
Scenario 2: Actor Identification
Find Actor
Validate Actor

22. Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agreement@3)
MicroTask [Like]
Scene
block 1
min 1
5
Order Scenes
[Order] (SortByLiking)
MicroTask [Like]
Scene
block 2
min 2
[Class=E]
[Class=B OR M]
Example of Scenario 1 Model

23. Patterns
Intra-Task Auxiliary Workflow

24. Intra-Task
Pre-Processing
Post-
Processing
Task
Consensus
Splitting
Assignment
Aggregation
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
• Consensus
• Join
• Sort
• Grouping
• Performer Control
• Planning
• Assignment
• Aggregation
• Quality & Performer

25. Auxiliary Intra-Task
Pre-Processing
Post-
Processing
Task
Consensus
Splitting
Assignment
Aggregation
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
microTaskmicroTaskmicroTask
• Pruning • Tie Breaking
• Operations before or
after the execution

26. Workflow
• Set of heterogeneous
tasks
Create Decide
Improve
Compare
/ Verify
Find Fix
(a)
(b)
(c)
Auxiliary
Task
Create/Decide
Improve/Compare
Find/Fix/Verify

27. Experiments
1700 Executions
39$
September 2013

28. Streaming Vs. Batch Execution
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agreement@3)
MicroTask [Like]
Scene
block 1
min 1
7
Order Scenes
[Order]
(SortByLiking)
MicroTask [Like]
Scene
block 2
min 2
[Class=E]
[Class=B OR M]
(P1)
5
3
Position Scenes
[Classify]
(Static Agreement@3)
MicroTask [Classify]
Scene
block 1
min 1
Spoiler Scenes
[Like]
(Static Agreement@3)
MicroTask [Like]
Scene
block 1
min 1
7
Order Scenes
[Order]
(SortByLiking)
MicroTask [Like]
Scene
block 2
min 2
Cons.
[Class=E]
[Class=B OR M]
(P2)
5
3
P.Beg P.Mid P.End
P1 0.5 1 0.11
P2 0.5 0.8 0.33
Spear.Beg Spear. Mid
.MidP1 0.5 0.54
P2 0.9 0.51
Position
Order
P1 P
b) Elapsed Tim
#ClosedObjects
1
10
20
30
40
50
60
70
80
5 60 120 180 240 300 360 5
Position
Order
P1
Position
Order
P2
b) Elapsed Time (Mins)
#ClosedObjects
1
10
20
30
40
50
60
70
80
5 60 120 180 240 300 360 5 60 120 180 240 300 360
Position
Order
P1 P
b) Elapsed Tim
#ClosedObjects
1
10
20
30
40
50
60
70
80
5 60 120 180 240 300 360 5

29. Intra-Task Consensus Vs. Workflow Decision
A4 A5 A6
Precision
0.99 0.95 0.89
Recall 0.90 1 0.96
F-Score 0.93 0.97 0.90
Find Actors
[Tag]
(Static Agreement@3)
MicroTask [Tag]
Scene
block 1
min 1
Validate Actors
[Like]
MicroTask [Like]
Scene+Actor
block All
min 1
5
(A4)
Find Actors
[Tag]
MicroTask [Tag]
Scene
block 1
min 1
Validate Actors
[Like]
(Majority Voting@2)
MicroTask [Like]
Scene+Actor
block All
min 1
5 3
(A5)
(A6)
Find Actors
[Tag]
(Static Agreement@3)
MicroTask [Tag]
Scene
block 1
min 1
Validate Actors
[Like]
(Majority Voting@2)
MicroTask [Like]
Scene+Actor
block All
min 1
5 3
count(Actor.Like)<=1
Actor
Validate
A3
Actor
Validate
A4 A5
a) Elapsed Time (
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5 30 60 90 120 160 5
Actor
Validate
Actor
Validate
A4
Actor
Validate
A5
a) Elapsed Time (Mins)
30 60 90 120 160 5 30 60 90 120 160 5 60 120 180 240 60
Actor
Validate
Actor
Validate
A5
a) Elapsed Time (Mins)
Actor
Validate
A6
30 60 90 120 160 5 60 120 180 240 60 300 540 780
Actor
Validate
A3 A4
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5
Actor
Validate
A3 A4
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5
Actor
Validate
A3 A4
#ClosedObjects
0
10
20
30
40
50
60
5 30 60 90 120 160 5

30. Take-Home Message
Engineering approaches should be
applied to people management too
In crowdsourcing, control and
monitoring are key for successful
outcomes

31. Questions?