Presentation for CSIRO Big Data Workshop 2012. October 2013, North Ryde, Australia.

1. BIG DATA IN FINANCE
Tomasz Bednarz
1

2. OVERVIEW OF THIS TALK
• Flavors of big data in finance
• Technology map for big data in finance
• Wheel of reincarnation
• A touch of history
• Role of visualisation
• Frameworks
• Real Time HFT
2

3. BIG DATA AGAIN
• Big
data is not a “crystal ball”
• The value from big data can only be extracted
when there is precise business problem to be
addressed
• Business problem has to be understood and well
defined at the first place
• Business to data mapping
• Better business predictions = difficult process
• Analysing big data
• Data scientists to examine the data and extract
critical information such as customers buying
habits - need for experts
• Catalogue data assets (10% of data may actually
mean something)
• Privacy
• No clear access controls to access data?
Volume
Big Data
Variety
Velocity

4. SURVEY
http://strata.oreilly.com/2012/01/enterprise-big-data-survey-results.html

5. FLAVORS OF BIG DATA IN
FINANCE
• Structured
• Market data feeds
• Big and fast
• Order flows
• Execution info
• Unstructured
• News
• Research
Dad Bailey’s news stand in 1939. Comics are on the top left.
5

6. TECHNOLOGY MAP FOR
DATA
Loosely Structured Information
Information
Retrieval
Analytics
Highly Structured Information
6
= text and relationships
Human
Computer
Market
Interface
= high frequency ticks/trading order flow

7. TECHNOLOGY MAP FOR
DATA
Loosely Structured Information
Information
Retrieval
Analytics
= text and relationships
Human
Computer
Market
Interface
fast: trading
Highly Structured Information
7
= high frequency ticks/trading order flow

8. TECHNOLOGY MAP FOR
DATA
Loosely Structured Information
= text and relationships
slow: investment research, portfolio management
Information
Retrieval
Analytics
Highly Structured Information
8
Human
Computer
Market
Interface
= high frequency ticks/trading order flow

9. SKILL ZONES - HUMAN
VERSUS MACHINE
months
Time to
exploit
information
collaborative
hybrid generation
Reinvent tech/self
every ~three years!!!
smart
market
monitoring
CSIRO GPU Cluster
Visualizing Marathon 2011
algo
gurus
ms
pure HPC (HFT)
low
Data Subtlety Complexity. Language Concept Importance.
9
high

10. WHEEL OF REINCARNATION
T. H. Myer and I. E. Sutherland. On the design of display processors. Communications of ACM, June 1968, 410-414.
10

11. STRUCTURED DATA
11

12. NYSE
New York Stock Exchange started as very
low-tech place. In 1779, the NYSE was a
bunch of guys standing around a
buttonwood tree at 68 Wall Street shouting
at each other on days when it didn’t rain or
snow.
In 1794 we see the first big
technological solution: the roof.
12

13. NYSE
Everybody moved inside, to the Tontine
Coffee House at the corner of Water
and Wall streets. Hands, roofs, chalk (oh
technology)!
1n 1823, the Difference Engine was
invented by Charles Babbage “I wish to
God these calculations had been
executed by steam.”

14. TECHNOLOGICAL INVASION
- SIMPLER ERA
Before telegraphy, in 1850s, the sky over
Wall Street was open and clear
BUT yo
u had t
o know
to part
Morse
icipate
Code
in the m
a
(Morse
, 1837) rket!
It took only a short time for telegraphy’s
compression of time and space to transform
the scenery. Everybody had to have it!
14

15. TICKER TAPE
A huge success (as important as the roof, hand
signals, and the telegraph). People use jumbo
magnifying lenses - people traded faster than
the machines, so delay meters were installed.
All that ticker tape also made for nice
parades. Here group of specialists
celebrating the one-millionth bagging of a
buy-side trader.

16. AND MORE TECHNOLOGY
Ticker tapes (an enduring market
visualisation) are still with us today.
Type 80 Card Sorter IBM
1925
Our CSIRAC
1 operation
per second
- 1949

17. ROLE OF VISUALISATION
Dynamic market view: Oculus Info’s
Visible Marketplace - portfolio
visualisation.
!
Not very dynamic on paper.
!
Another reason to be glad we have
web browsers.

18. MAP OF THE MARKET
Map of the Market , the classic big picture of whole market visualization.
It was invented by Marten Wattenberg, now at IBM Many Eyes.

19. Cultural Finance
Finance Cover Letter

20. UNSTRUCTURED DATA
20

21. TECHNOLOGY MAP FOR
DATA
Loosely Structured Information
= text and relationships
slow: investment research, portfolio management
Information
Retrieval
Analytics
Highly Structured Information
21
Human
Computer
Market
Interface
= high frequency ticks/trading order flow

22. INFORMATION SOURCES
• Specialised Industry Media
• Local and International Media
• Direct Corporate Communications
• Research Labs
• Government Agencies
• Social Media
• Crowd Sourcing
!
• HTML / Text Feeds / XML

23. BIG DATA AT NICTA
• Developed
to deliver customized
business intelligence and advanced data
mining to the Enterprise marketplace
• Machine
Learning to obtain deep
insights in real time
• Build
upon a scalable open source
software platforms
• See
http://www.youtube.com/watch?v=TxQPdUt_x3c
• Scoobi
- a Scala productivity
framework for Hadoop (allows you to write
what you want rather than how to do it).
• Source
• Demo
code: https://github.com/nicta/scoobi
of Scoobi tomorrow at the
hands-on afternoon session (by Piotr)
http://www.ambiata.com/index.html

24. http://www.zdnet.com/how-is-big-data-faring-in-the-enterprise-7000002404/
24

25. REAL-TIME REAL-TIME REAL-TIME
• Everything
is moving to real time
• Everything
is moving towards continuous time
• Everything
is moving towards mobility (anywhere, anytime)
• Also
GPU for high-frequency trading (HFT)
• GPU
alongside FPGAs receiving stream of market data to increase the accuracy of
the strategy, or even to suggest change in the algorithm used as market conditions
evolve
• GPU
Direct enables a GPU to talk directly to the FPGA, receiving data from it into
a kernel, which may be preloaded to perform its magic once the data is available
• Hadoop
+ GPUs

26. ACKNOWLEDGEMENTS
• Andrew
Sheppard, Fontainhead
• David
Leinweber, The Lawrance Berkley National Lab, author
of Nerds on Wall Street NOWS
• Colleagues
• Craig
from Westac and Commonwealth banks
Mudge for organising Big Data workshop, and always
great discussions

27. THANK YOU
27