A talk by C. Scott Andreas (@cscotta) of Boundary on "the network" and designing / deploying distributed applications.
This session offers a deep-dive into how application-level problems manifest at the network level. Some of these cases range from basic network partitions and node outages to sophisticated application-level changes such as garbage collections on managed runtimes, classes of bugs which evade conventional monitoring but constitute partial failures, changes in network activity based on database partitioning, load balancing, and sharding, and other warning signs that crop up at layer three long before wreaking havoc at layer seven as customer-visible failures begin to occur. Combining application-level metrics with network analytics is a powerful cocktail for identifying hot spots quickly, and connecting the dots out to the client closes the loop.
21. LEMMA 1:
there exists no possible way for
applications to communicate
except via the network
taco.cat/oscon12
22. LEMMA 2:
applications are unable to fulfill
their purpose without communic-
ating and participating in a cluster
taco.cat/oscon12
23. LEMMA 3:
the network can be represented
as a time series and a graph
taco.cat/oscon12
24. LEMMA 4:
nearly all modes of failure in
distributed systems can be
identified and predicated this way
taco.cat/oscon12
25. WHAT CAN ONE OBSERVE?
poor load balancing
network partitions
bad deploys
GC pauses
good deploys impaired nodes
variance security breaches
taco.cat/oscon12
32. FURTHER READING
– Network Flow Analysis
– Gigascope
(No-Starch Press)
(AT&T Research)
– Eamonn Keogh
– Dynamic Time Warping
(“Atomic Wedgie”)
(algo)
– K-Snap
– Studying Complex
(“Efficient Aggr. for Graph Summ.”)
– “Medians and Beyond” (Shrivastava) Adaptive Systems (Holland)
– Exponential Smoothing:
The State Space Approach (Hyndman)
– HyperLogLog / Count-Min Sketch
taco.cat/oscon12
33. BONUS!
a spec for thai chili salsa and reference implementation
– Half a thing of cilantro
– 28oz peeled tomatoes
– 6 cloves of garlic
– Half a bag of hot thai chilis
– 2 tablespoons of salt
– Half a cucumber
– 2 tablespoons of white vinegar
– Handful of radishes
– Juice from half a lime
– One green bell pepper
– Bit of parsley
– A small red onion
– Some lemongrass
taco.cat/oscon12
34. BONUS!
U P
a spec for thai chili salsa and reference implementation
L
–
L
Half a thing of cilantro
– 28oz peeled tomatoes
A
– 6 cloves of garlic
– Half a bag of hot thai chilis
–
T
2 tablespoons of salt
I
– Half a cucumber
– 2 tablespoons of white vinegar
–
X
Handful of radishes
I
– Juice from half a lime
– One green bell pepper
–
M
Bit of parsley
– A small red onion
– Some lemongrass
taco.cat/oscon12
35. A Look at the Network
Searching for Truth in Distributed Applications
oscon 2012 - portland oregon
taco.cat/oscon12