As websites grow in scale and complexity there is a high likelihood that the number of services involved in responding to a given web or API request also grows. This increases the complexity of answering seemingly simple questions such as “why is this request slow?” One approach to gain better insight is to trace through all the service calls made and create a tree representing the overall execution profile. Etsy recently built such a system, called CrossStitch. In the process we discovered that distributed tracing is not rocket science, in fact there are existing open source projects well-suited to do a lot of the heavy lifting required. You don’t need to adopt something like Zipkin and ingest all the dependencies that implies into your existing infrastructure. One realization, obvious in hindsight, is that building a distributed tracing system will face/solve some distributed problems. Additionally, if you get such a system up and running you are very well placed to roll your own log indexing system like Splunk.

1. What Etsy Learned
Building a Distributed
Tracing System
Paul Wright
Core Platform, Etsy
@wrighty

2. What is Etsy?

3. Etsy is the marketplace we
make together

4. (It’s a busy marketplace)

5. Over 1 Million Sellers
$1.35 Billion GMS* in 2013
*Gross Merchandise Sales

7. Spoilers

8. What did we learn?

9. How did we build it?
What did we learn?

10. Why did we need it?
How did we build it?
What did we learn?

11. Why did we need it?
How did we build it?
What did we learn?

12. First, some context

15. Aversion to Services*

16. * With exceptions
(search, payments, photos)

17. Single Threaded
Short Lived
Shared Nothing

18. This raises a problem:
Sequential Execution

19. Application complexity
dictates lower bound of
response time

20. True for both
website and API

21. (API powers native
mobile apps)

23. How to balance this tension
with constraints of the
stack?

24. Single Threaded
Short Lived
Shared Nothing

25. The somewhat tangential
organisational problem:

26. Native mobile apps always
play catch up to web

27. Feature development:
web-first,
mobile-sometime-maybe

28. Increasingly disconnected
from reality

29. Need an approach to
unify platforms

30. The answer:

31. Keep doing what we
understand

32. HTTP calls, but
asynchronously

34. Build a lightweight API
framework to support this
approach

35. (Borrow API design ideas
from Netflix)

36. New API offers
two types of endpoint:
Component and Bespoke

37. Component:
Generic resources for all
clients

38. Bespoke:
Custom endpoints for
specific client

39. Both can make concurrent
HTTP calls to other
Component endpoints

40. Advocate for API-First
mindset across
organisation

41. New programming model

49. Bespoke
Homepage
Render
Homepage
APIWeb

50. Bespoke
Homepage
Activity
Feed
New
Favorites
Our Picks
For You
Sidebar
Render
Homepage
APIWeb

51. Bespoke
Homepage
Activity
Feed
New
Favorites
Our Picks
For You
Sidebar
Blog
Post
Featured
Shop
Render
Homepage
APIWeb

52. Bespoke
Homepage
Activity
Feed
New
Favorites
Our Picks
For You
Sidebar
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Blog
Post
Featured
Shop
Render
Homepage
APIWeb

53. May have accidentally built
a distributed RPC system

54. Single Threaded
Short Lived
Shared Nothing

55. Many Single Threaded
Many Short Lived
Still Shared Nothing?

56. Existing introspection tools
built for a simpler time

57. Need a way to visualise
execution tree

58. Why did we need it?
How did we build it?
What did we learn?

59. First step: Research

60. Short answer: read Dapper

61. http://research.google.com/pubs/pub36356.html

62. Slightly longer answer:
Read Dapper, study Zipkin,
start prototyping

67. Build vs. Buy

68. Build vs. Buy
(or Adopt)

69. In this instance, we decided
to build

70. Distributed Tracing
Overview

71. All implementations
operate on a trace

72. A trace is a tree of execution

73. Each tree node is a span

74. A span is a series of
observations about a single
service call

75. Bespoke
Homepage
Activity
Feed
New
Favorites
Our Picks
For You
Sidebar
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Blog
Post
Featured
Shop
Render
Homepage
APIWeb

76. Bespoke
Homepage
Activity
Feed
New
Favorites
Our Picks
For You
Sidebar
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Blog
Post
Featured
Shop
Render
Homepage
API
Web

77. Story Card
Story Card
Story Card
Story Card
Bespoke Homepage
Activity Feed
New Favorites
Our Picks For You
Sidebar
Blog Post
Featured Shop
Render Homepage
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
Story Card
0ms 330ms

78. What did we build?

79. Instrumentation
Data Collection
Data Retrieval
Visualisation

80. Request tree construction

81. Clients pass down tree
location data

84. All non-HTTP service calls
are instrumented
only from client’s POV

85. Instrumentation
Data Collection
Data Retrieval
Visualisation

86. Client Send
Server Receive
Server Respond
Client Receive

87. Annotate server execution
for high level context

88. Server execution resulting
in low-level service calls is
handled by child spans

89. Terminating annotations
trigger span logging

90. Instrumentation
Data Collection
Data Retrieval
Visualisation

91. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

92. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

93. json

94. msgpack
gzip
base64

95. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

96. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

97. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

98. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

99. logstash elasticsearch
Ajax API
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
httpd /
PHP
httpd /
PHP
httpd /
PHP
httpd /
PHP
cross_stitch_spans.log
Web node

101. Instrumentation
Data Collection
Data Retrieval
Visualisation

113. Why did we need it?
How did we build it?
What did we learn?

114. There’s a number of moving
parts, but this is not
rocket science

115. Most of the code is specific
to your environment

116. (Hence I’m not announcing
any open source release)

117. But that code is simple to
write, and can be done
piecemeal

118. New internal tools are no
different from new
products

119. There’s an adoption curve,
it’s your job to accelerate it

120. First version was
end-to-end complete

121. Get cross-discipline input

122. Treat it like product
development

123. Observing both client and
server POVs is powerful

124. Network latency is easier to
observe

125. Areas of few observations
are interesting

128. Shared nothing isn’t free

131. But caching is cheap

134. There will be WTFs

136. You won’t always
solve them

137. Doing the simplest thing
possible works

138. Hacky logging runs for 0.1%
of all requests

139. Future directions

140. BIG DATA

141. Hybrid API client

142. Instrumenting other
services from server POV

143. Sample more requests -
maybe 100%?

144. Why did we need it?
How did we build it?
What did we learn?

145. Thanks!
@wrighty