Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2pZSlTn. Sam Adams presents an overview of the architecture LMAX Exchange uses to deliver over $2 trillion a year through their platform, and shares their experience of how taking a scientific approach to testing and tuning software has helped them to build a high-availability stateful system. Filmed at qconlondon.com. Sam Adams is currently the Head of Software Engineering at LMAX Exchange. He has had an eclectic career to date: variously modeling the metabolism of drugs and food additives, creating tools to manage and mine scientific data, and now building a high performance exchange at one of the UK's fastest growing Tech Companies.

1. Low Latency Trading Architecture
Sam Adams
QCon London, March, 2017

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
lmax-trading-architecture
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. don't panic
don't panic
buy
sell
GBP/USD

5. Typical day:
1,000's active clients
100,000's trades occur
100,000,000's orders placed
– very bursty: spikes of 100s / ms
1,000,000,000's market data updates sent

6. End-to-end latency:
50%: 80 µs
99%: 150 µs
99.99%: 500 µs
Max: 4ms(*)

7. System Architecture
Building low latency applications

8. Instruction
Execution
reports
Market
data
* latency sensitive *
* throughput matters *

9. The Disruptor

10. ConsumerProducer
High performance inter-thread messaging

11. public class ArrayBlockingQueue<E>
{
final Object[] items;
int takeIndex;
int putIndex;
int count;
/** Main lock guarding all access */
final ReentrantLock lock;
}
ArrayBlockingQueue vs Disruptor
locking & contention

12. public class ArrayBlockingQueue<E>
{
final Object[] items;
int takeIndex;
int putIndex;
int count;
/** Main lock guarding all access */
final ReentrantLock lock;
}
public class RingBuffer<E>
implements DataProvider<E>
{
// ...
final long indexMask;
final Object[] entries;
final Sequence cursor;
// ...
}
public class BatchEventProcessor<E>
{
final DataProvider<E> dataProvider;
final Sequence sequence;
}
ArrayBlockingQueue vs Disruptor
locking & contention
vs
single writers

13. Claimed: -1
Published: -1
Consumer
Consumed: -1
Waiting for: 0
Producer

14. Claimed: 0
Published: -1
Consumer
Consumed: -1
Waiting for: 0
Producer
Claim slot: 0

15. Claimed: 0
Published: 0
Consumer
Consumed: -1
Waiting for: 0
Producer
Publish slot: 0

16. Claimed: 0
Published: 0
Consumer
Consumed: -1
Available: 0
Processing: 0
Producer

17. Claimed: 0
Published: 0
Consumer
Consumed: 0
Waiting for: 1
Producer

18. Claimed: 3
Published: 3
Consumer
Consumed: 0
Waiting for: 1
Producer
Published: 1-3

19. Claimed: 3
Published: 3
Consumer
Consumed: 0
Available: 3
Processing: 1,2,3
Producer

20. Claimed: 3
Published: 3
Consumer
Consumed: 3
Waiting for: 4
Producer

21. Supports dependency graphs between consumers

22. Messaging

23. Asynchronous Pub/Sub messaging:
- UDP Multicast: low latency, scalable, unreliable
- Services publish / subscribe to topics
- topic = unique multicast group
- Informatica UMS (aka 29 West LBM) provides * some reliability *

24. Asynchronous Pub/Sub messaging:
- Push based
- If you miss a message, it is gone
- Late-join: no history

25. javassist generated proxies to interfaces
public interface TradingInstructions
{
void placeOrder(PlaceOrderInstruction instruction);
void cancelOrder(CancelOrderInstruction instruction);
}
See GeneratedRingBufferProxyGenerator in disruptor-proxy for inter-thread version
https://github.com/LMAX-Exchange/disruptor-proxy
Event:
long sequence
byte operationIndex
byte[] data
int length

26. See GeneratedRingBufferProxyGenerator in disruptor-proxy for inter-thread version
https://github.com/LMAX-Exchange/disruptor-proxy
public void placeOrder(PlaceOrderInstruction arg0)
{
// ...
event.initialise(sequence, 1); // operation index
marshaller.encode(arg0, event.outputStream());
// ...
}
Publisher proxy:
Event:
long sequence
byte operationIndex
byte[] data
int length

27. See GeneratedRingBufferProxyGenerator in disruptor-proxy for inter-thread version
https://github.com/LMAX-Exchange/disruptor-proxy
Invoker invokers[];
TradingInstructions implementation;
public void onEvent(Event event)
{
Invoker invoker = invokers[event.getOperationIndex()];
invoker.invoke(event.getInputStream(), implementation);
}
public void invoke(InputStream input, TradingInstructions implementation)
{
PlaceOrderInstruction arg0 = marshaller.decode(input);
implementation.placeOrder(arg0);
}
Subscriber proxy:
Event:
long sequence
byte operationIndex
byte[] data
int length

28. Matching Engine

29. For speed:
All working state held in memory
Remove contention: single threaded

30. Don’t block business logic: buffer for outbound I/O

31. Don’t block network thread: buffer incoming events

32. All state in volatile memory:
Save on shutdown / Load on startup

33. Recover from unclean shutdown
Journal incoming events to disk, replay on startup

34. Replicate events to hot-standby
for resiliency
Manual fail-over
(also to offsite DR)

35. Holding all your state in memory
No database
No roll-back
Up-front validation is critical
Never throw exceptions
- result is inconsistent state

36. System must be deterministic
All operations event sourced
time sourced from events
collections must be ordered
no local configuration

37. Determinism bugs are really nasty
Only an issue if we have
to fail-over or replay
Primary is the source of truth

38. Gateways

39. Same principles:
- non-blocking / message passing
- minimise shared state

40. Stream Processing

41. Matching Engine
Order Book

42. All Orders[ ]
Order Added
Order Cancelled
Order Added
Trade
Trade
Order Added
...
Matching Engine
Order Book

43. All Orders[ ]
Order Added
Order Cancelled
Order Added
Trade
Trade
Order Added
...
Matching Engine
Order Book
Market Analysis
Order Book Image
Event Store
AML Alerts
Order Book Image

44. Where latency doesn’t matter...
- How big are the bursts?
- Buffers are your friend
Does data loss matter?
Market Analysis
Order Book Image
Event Store
AML Alerts
Order Book Image

45. More Reliable Messaging

55. Handling buffer wraps
‘better never than late’
- reset & late join
persistent data loss
- recover from event store
- journal replay and gap-fill

56. Low latency applications:
mechanical sympathy

57. [sam@box ~]$ lstopo

58. Machine (126GB)
NUMANode P#0 (63GB)
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (2
L1d
L1i (
Core
PU
PU
Main Memory
L1/L2 Caches
CPU Core / Hyper Threads

59. Machine (126GB)
NUMANode P#0 (63GB)
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (25
L1d (3
L1i (3
Core
PU
PU
CPUs are faster than memory
Intel Performance Analysis Guide:
L1 CACHE hit, 4 cycles
L2 CACHE hit, 10 cycles
local L3 CACHE hit, ~40-75 cycles
remote L3 CACHE hit, ~100-300 cycles
Local Dram ~60 ns
Remote Dram ~100 ns

60. Memory system optimised for:
Temporal locality
Spatial locality
Equidistant locality

61. Reference vs Primitives
Long[] vs long[]

62. public class Cash
{
long value;
}
Calculations with money
- double: inexact
- BigDecimal: expensive
Fixed-point arithmetic with long
But I want type-safety...

63. long price1 = 1250000L;
long quantity1 = 1520L;
// BUG
long price2 = quantity1;
Prices, precision: 6dp
1250000L → 1.250000
Quantities, precision: 2dp
1520L → 15.20

64. https://checkerframework.org/
With Type Annotations & Units Checker:
@Price long price1 = 1250000L;
@Qty long quantity1 = 1520L;
// Compilation error
@Price long price2 = quantity1;
Prices, precision: 6dp
1250000L → 1.250000
Quantities, precision: 2dp
1520L → 15.20

65. public class HashMap<K,V>
{
Node<K,V>[] table;
static class Node<K,V>
{
K key;
V value;
Node<K,V> next;
}
public class Long2ObjectOpenHashMap<V>
{
long[] keys;
V[] values;
}
java.util vs fastutil
Map<Long,X> vs LongMap<X>

66. public class HashMap<K,V>
{
Node<K,V>[] table;
static class Node<K,V>
{
K key;
V value;
Node<K,V> next;
}
public class Long2ObjectOpenHashMap<V>
{
long[] keys;
V[] values;
}
java.util vs fastutil
Map<Long,X> vs LongMap<X>

67. False sharing: revisit the Disruptor
public class ArrayBlockingQueue<E>
{
final Object[] items;
int takeIndex;
int putIndex;
int count;
/** Main lock guarding all access */
final ReentrantLock lock;
}

68. public class RingBuffer
{
// ...
final Object[] entries;
final Sequence cursor;
// ...
}
public class Sequence
{
long p1, p2, p3, p4, p5, p6, p7;
long value;
long p9, p10, p11, p12, p13, p14, p15;
}
False sharing: revisit the Disruptor

69. public class RingBuffer
{
// ...
final Object[] entries;
final Sequence cursor;
// ...
}
public class Sequence
{
@Contended
long value;
}
False sharing: revisit the Disruptor
Java 8:

70. Removing Jitter:
GC & Scheduling

71. GC Options:
Zero garbage
Massive heap, GC when convenient
Commercial JVM – Azul Zing

72. GC Options:
Zero garbage
Massive heap, GC when convenient
Commercial JVM – Azul Zing

73. GC Options:
Zero garbage
Massive heap, GC when convenient
Commercial JVM – Azul Zing

74. JVM
OS
Avoiding scheduling jitter

75. JVM
OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46

76. JVM
OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46
Remove reserved CPUs from the kernel
scheduler
isolcpus=0,2,4,6,8,24,26,28,30,32

77. JVM
OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46
Create CPU sets for system, application
# cset set --set=/system --cpu=18,20,...,46
# cset set --set=/app --cpu=0,2,...,40
/
/system/app

78. OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46
Processes default to the / CPU set
/
/system/app

79. OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46
Move all threads into /system CPU set
# cset proc --move -k --threads --force
--from-set=/ --to-set=/system
/
/system/app

80. JVM
OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46
Launch application in /app CPU set,
taskset to run in pool
$ cset proc --exec /app
taskset -cp 10,12...38,40
java <args>
/
/system/app

81. JVM
OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46
Move critical threads onto their own cores
using JNA / JNI
sched_set_affinity(0);
sched_set_affinity(2);
...

82. JVM
OS
Socket P#0
L3 (30MB)
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#0
PU P#0
PU P#24
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#1
PU P#2
PU P#26
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#2
PU P#4
PU P#28
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#3
PU P#6
PU P#30
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#4
PU P#8
PU P#32
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#5
PU P#10
PU P#34
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#8
PU P#12
PU P#36
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#9
PU P#14
PU P#38
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#10
PU P#16
PU P#40
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#11
PU P#18
PU P#42
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#12
PU P#20
PU P#44
L2 (256KB)
L1d (32KB)
L1i (32KB)
Core P#13
PU P#22
PU P#46

83. Summary

85. round-trip a correlation ID

86. round-trip a correlation ID
25µs

87. Thank You!
sam.adams@lmax.com
https://www.lmax.com/blog/staff-blogs/
p.s. we’re hiring!

88. The End.

89. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
lmax-trading-architecture