The dangers of relying on quantitative data to make decisions.

1. Beyond
Measure
@mulegirl
1

2. !#? !#?
2

3. :-)
3

4. DEEP
THOUGHT,
TELL US THE
ANSWER…
OK
4

5. 7,500,000
YEARS LATER…
5

6. ???
42!
6

7. 42!
!#!?&wtf!
¯_(ツ)_/¯
7

8. IT WOULD HAVE
BEEN SIMPLER TO
KNOW THE ACTUAL
QUESTION.
8

9. ONLY WHEN YOU
KNOW THE
QUESTION, WILL YOU
KNOW WHAT THE
ANSWER MEANS.
9

10. 10

11. I am a rational biped.
All rational bipeds rule.
So, I rule.
11

12. 12

13. :-)
13

14. :-(
14

15. We have to work together
to make better decisions.
15

16. 16

17. 17

18. Quantitative
approaches feel
better, but are often
actually worse.
18

19. We are
imperfect
humans
designing things
for other
imperfect
humans.
19

20. DON’T
PANIC
20

21. 21

22. Fe
22

23. Flickr/fungleo
Kirk Siang/Flickr
60%
23

24. Fe
24

25. Flickr/eskimo_jo
[dramatization]
25

26. 26

27. [a dramatization]
The Nourishing Home / Kelly Smith
27

28. [dramatization]
28

29. 29

30. Lucky!
30

31. luckyironfish.com
31

32. oo
O
O
O
O
HO
HO
OH
OH
H2C
H2C
CH2
CH2
CH2C
C
CH2 C
C
:N
N:
32

33. 33

34. Human problems
require illogical
approaches.
34

35. Stories
have
power
35

36. Flickr/ Billy Wilson
Only 12
people?!?
36

37. 2,500,000
000,000,
000,000
bytes
37

38. 38

39. 39

40. 90%?
40

41. 011010000110010101101
100011100000010000001
10110101100101
010010010010000001100
001011011010010000001
100010011001010110100
101101110011001110010
000001110111011000010
111010001100011011010
000110010101100100
011010000110010101101
100011100000010000001
10110101100101
41

42. :-)
42

43. This is pretty
creepy when you
think about it.
Flickr/ thevlue
43

44. 44

45. I am a rational biped.
45

46. Original image: Olly Moss
46

47. We are blind to
our minds.
47

48. Ease
Clear Display
Related Experience
Primed Idea
Good Mood
Feels True
Feels Familiar
Feels Good
Feels Effortless
Source: Daniel Kahneman
48

49. Data doesn’t
change minds.
49

50. 50

51. 26%
51

52. Our access to data
has evolved,
our brains haven’t.
52

53. Data doesn’t
have meaning.
53

54. 42
54

55. Obviously,
what I can
count
should
count more.
55

56. We have a lot of data.
We make a lot of decisions.
We like stories.
We aren’t good at math.
The hard decisions aren’t
math problems, anyway,
56

57. DON’T
PANIC
57

58. 58

59. QuantitativeQualitative
59

60. 60

61. What is going on?
How are people behaving?
Why?
What does it mean?
What is the problem?
Any ideas?
Where?
When?
How many?
How much?
How often?
Which yields more?
61

62. Usability Studies
Field Studies
Interviews
Clickstream Analysis
A/B Testing
Surveys*
62

63. Are you a
statistician?
63

64. Flickr/ Billy Wilson
Tight
control.
64

65. No fishing trips.
65

66. Surveys
66

67. Surveys are the most
dangerous research tool.
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
67

68. It is too easy
to run a
survey.
68

69. Easy feels
true.
69

70. Bad surveys don’t
smell bad.
70

71. 71

72. Lies Poppycock Irrelevancies
72

73. Direct interaction with
users is prohibited by my
organization, but I have
been allowed to conduct a
simple survey by email to
identify usability issues.
73

74. 74

75. [a dramatization]
The Nourishing Home / Kelly Smith
75

76. My boss is a convert to Foresee.
She was apparently very skeptical
of it at first, but she's a very
analytical person and was
converted by its promise of being
able to quantify unquantifiable data
—like "satisfaction".
76

77. Customer satisfaction
is a lie.
77

78. 78

79. 79

80. Will the people I’m
surveying be willing and
able to provide a truthful
answer to my question?
80

81. ONLY WHEN YOU
KNOW THE
QUESTION, WILL YOU
KNOW WHAT THE
ANSWER MEANS.
81

82. “At its core, all business is
about making bets on
human behavior.”
—Ben Wiseman, WSJ
82

83. Make better
choices.
83

84. We are
imperfect
humans
designing things
for other
imperfect
humans.
84

85. The most
measurable data
may not be
valuable.
85

86. Know your
reason.
86

87. What and Why
Flickr/johnwiechecki
87

88. 88

89. Thank you!
erika@muledesign.com
@mulegirl
www.muledesign.com
Brief books for people who make websites No.
9
JUSTENOUGH
RESEARCH
Erika Hall
89