"Big data" has been around for a few years now but for every hundred people talking about it there’s probably only one actually doing it. As a result Big Data has become the preferred vehicle for inflated expectations and misguided strategy. As always, language holds the key and the seed of the issue is reflected in the expression itself. "Big Data" is not so much about a quality of the data or the tools to mine it, it’s about a new approach to product, policy or business strategy design. And that’s way harder and trickier to implement than any new technology stack. In this talk I look at where Big Data is going, what are the real opportunities, limitations and dangers and what can we do to stop talking about it and start doing it today.

1. Heavy, Messy,
Misleading
Why Big Data is a human problem,
not a technology one
Francesco D’Orazio, @abc3d
VP Product, PulsarPlatform.com

2. Every talk about Big Data
should start with Twin Peaks

7. There’s more to big data than
the technology behind it.
And the best way to find out
what it is, is to start from the
metaphors of Big Data

8. stream
of data

9. ocean
of data

10. river
of data

11. a data
leak

12. data
firehose

13. data
flood

14. data
tsunami

15. data “is”
fluid

16. data “is”
huge

17. data “is”
powerful

18. data “is”
unpredictable

19. data “is”
uncontrollable

20. Data is the new oil(?!)

21. We are not going to war for it
(yet)

22. Data is not a scarce resource

23. The abundance of data is the
result of the instrumentalisation
of the natural, industrial and
social worlds

25. The Large Hadron Collider can record up to
40 million particle interactions per second

26. The Square Kilometer Array will collect data from the deep
space dating back to more than 13 billion years ago

27. Wolfram Data Science on Facebook Data: how our
topics of discussion change by age and gender

28. Carna Botnet: in just 60 seconds nearly 640 Terabytes of IP
data is tranferred across the globe via the Internet

33. Machine Sensing

36. The sensors on the new Airbus 380 generate 10 terabytes of data
every 30 minutes. That’s 120T every LDN-NYC flight

39. And yet, another reason why data
is not the new oil is that we are
not actually doing it much…

40. 99.5%
Percentage of newly created digital data
that’s never analysed

42. But that’s not strictly true
either…

43. 0.5%
Percentage of newly created digital data
that’s actually being used

44. higher % of teenagers having sex
vs
% of new data being analysed

45. Credit Scores have replaced the handshake
with the bank manager

46. Fair and Isaac came along in 1956. Today they
crunch around 10 billion scores each year

47. Buying advertising used to be about smiles
and jokes over Martini lunches

48. Now it looks more like this…

49. 11 seconds of trading for the FB shares. Already in 2006 one
third of all transactions in EU and US was algorithmic

50. Walmart handles more than 1M customer transactions
per hour, all affected by price elasticity

51. Price Discrimination based on log in info, browser history, device,
A/B testing is common practice for most online retailers

52. 75% of the content Netflix serves is chosen
based on a Netflix recommendation

53. At Buzzfeed every item of content has its own dashboard showing how it
spreads form ‘seed views’ to ‘social views’ and by what ‘viral lift’

54. Upworthy
Systematic experimentation: 15% of the top
10.000 websites uses A/B testing

55. Crowdpac matches candidates and funders based on analysis of public
speeches, contribution and other sources of public data on the candidate

56. LAPD run a pilot to predict where a crime is going to happen next
(‘crime aftershocks’) based on 13 million crimes over 80 years

57. The Dubai police is equipping officers with Google Glass enabled
with face recognition to identify potential wanted criminals

58. Target Data can predict with 75% accuracy the likelihood
that a home will sell in the next 30, 60 or 90 days

61. LinkedIn had a student problem: so they re-arranged the
data they already have for a student audience

62. 99.5%
Why then are we throwing away this
much data?

63. We are still learning to
recognize problems as
data-problems

65. Big Data changes the very
definition of how we
produce knowledge

66. Less > More
Exact > Messy
Causation > Correlation

67. Significant correlation requires
scale. And scale is hard to handle.

68. DNA research is a case in point: DNA data is hard to manipulate and there’s not
enough sequenced DNA available to establish significant patterns

69. Big Data comes with Big Errors

70. Data is rarely normalised.

71. Data is siloed and not verifiable.

72. Big does not equal whole.

73. Big does not equal representative.

74. Data doesn’t speak for
itself. We speak for it.

75. Big Data is still biased and the
result of interpretation.

76. Correlation doesn’t imply
causality.

80. Models are often too simple and
not peer-reviewed.

82. Context is hard to interpret at
scale. Traditional Qual & Quant
have to work with big data.

84. 3 billion queries/day
50 million top keywords identified
5 years of data on flu spread matched
Overestimates by 50%
Didn’t predict pandemics

85. Big Data also means a big
new digital divide.

87. Accessible doesn’t mean ethical.

88. The problems slowing down
the adoption of Big Data
are human problems

89. And that’s because the
biggest innovation in Big
Data is a human innovation

90. An innovation in
decision-making: framing,
solving and actioning a
problem

91. “Data is just like crude. It’s
valuable, but if unrefined it
cannot really be used. It has to
be changed into gas, plastic,
chemicals, etc., to create a
valuable entity that drives
profitable activity; so must data
be broken down, analyzed for it
to have value” Michael Palmer

92. The opportunity in Big Data
is data middleware: turning
crude into gas, plastic,
chemicals

93. But until we invent the new plastic, the new gas, the new chemicals, we
are stuck with the smokescreen. Or even the smoke monster.

94. Thank You
Francesco D’Orazio, @abc3d
VP Product, PulsarPlatform.com