Open source tools like OpenTelemetry, OpenTracing, and W3C Trace Context are helping to standardize distributed tracing and observability. This allows developers to understand problems in microservices architectures by propagating unique trace IDs and collecting metrics and traces across services. While open source tools are useful for development and pre-production, commercial solutions are needed to handle production workloads at scale with additional features like access control and automated instrumentation. Standardization through open source is key to managing today's complexity in distributed systems.
6. 6
Why I am doing this talk
Distributed
Tracing
Observability
W3C Trace
Context
OpenCensus
OpenTracing
OpenTelemetry
Metrics
Span
Trace
7. 7
Application
In the Beginning there was the Monolith
Presentation
Business Logic
Data Access
Database
Services
Presentation
API Gateway
Auth Inventory CartAccount
Offers Shipping CheckoutStatus
Wire
8. 8
Developmentin a Microservices World
Cart
Dev
Preproduction
Cart Auth InventoryAccount
Offers Shopping CheckoutStatus
Push
Cart
• Latency
• Response Time
• Error Rate
• Number of queries
KPI’s
12. 12
Complecity has movedto the NetworkLayer
Client API GW Service
Service
Service
Service
Service
ServiceCart
Which requests lead to an error in our cart service?
Trace
a42b a42b
a42b
a42b
a42b
a42b
a42b
a42b
a42b = Trace Context
13. 13
A Trace is a Tree of Spans
Trace
Span
Span
Span
Click GW API
Spans represent a single operationand containmetadatalike the HTTP method, or a databasequery, or an error code
JDBC
Span: callDB()
Span: JDBC call
14. 14
Trace Context Propagation
Cartsa42b a42b
Extract
Inject
In process propagation
Auto Instrumentation
• Zipkin
• Sleuth
• OpenTelemetry
• Commercial
• …
15. 15
Trace Context Header Formats
• Proprietary
• B3 Header (Zipkin)
• W3C Trace Context
What’sthe header name and what does it contain?
16. 16
W3C Trace Context
Service A API GW Service B
Trace
Service C
OpenTelemetry AWS Zipkin OpenTelemetry
Goal: All monitoring systems and middlewaresagree on one format for trace context propagation
Span
Span
Span
Span
17. 17
W3C Trace Context Format
traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01 tracestate:
rojo=00f067aa0ba902b7,congo=t61rcWkgMzE
Version TraceID ParentID Flags
18. 18
Data collection
So far we just instrumentedthe code to propagatecontext but no data has been collected
Trace
Span
Span
Span
Agent Agent Agent
Click GW API
TraceContext TraceContext
MonitoringSystem Storage
25. 25
Solving our Cart Problem
Client API GW Service
Service
Service
Service
Service
ServiceCart
Trace
Client
Service
Currency
Cart
API GW
GET: Currency=EURO
26. 26
What we did to Solve the Problem
1. We used metrics to learn about a problem
2. We used distributedtracing to pass along a unique ID per trace
• For that,we used auto instrumentationto extract and inject the trace ID
3. We used a monitoring system and its agentsto collected traces and we could filter transactionsthatproduced an
error
4. We looked into the metadata of such a transactionto identify how it differs from succeeding ones
33. 33
OpenTelemetry – Developer usecases
• Cloud nativemicroservices architectures are hard
to trace and debug during development
• In developmentOpenTelemetry can be used to
either
• manuallycreate spans to trace certain
execution paths
• use provided auto-instrumentation tomake
a system observable
• As backend and UI, Jaeger is the most popular
tool. It’s open source and solely displaystraces
34. 34
OpenTelemetry – in Production
• Provides just a fraction of what modern tools provide
• Traces
• Metrics
• Logs
• Topology
• Behavior
• Code level visibility
• Metadata
• Manual instrumentation codeneeds to be kept up-to-date
• A backend needs to be maintained
• No support model if instrumentation breaksproductioncode
• No enterprise features (access control, throttling, scaling, …)
38. 38
What happens when we add support for a new framework?
• Today, our engineers reverse engineer frameworks to add
instrumentationsupport to them
• Every time an update is released, the instrumentationcode is
being tested.
• In case of issues, it goes back to the developmentteam who
needs to fix it and deploy an update.
• The whole process is automated and transparent to the customer ☺
• This is costly and time consuming
40. 40
“We want every platform and library to
be pre-instrumented with
OpenTelemetry and we’re committed to
making this as easy as possible.”
Sergey Kanzhelev (Google)
41. 41
What is Observability and how does it differ fromMonitoring?
1. In control theory, observability is a measure of how well internal states of a system can be inferred from
knowledgeof its external outputs.
Source: Wikipedia
2. In software development, observabilityisachieved by adding code (instrumentation)that emits telemetry
data.
3. Monitoringis the act of displayingand analyzing this telemetry data.
4. Monitoringalone can tell you that there is a problem.
E.g. ”We see that some users experience a 50% higher response time on check-out”
5. Observablityhelps finding the root cause (the why) by providingdatathat can be correlatedand analysed
freely even if this problem is completelynew to you (unknown unknowns)
E.g. “The response time of the checkout increases exponentially with the number of items in the basket,
because of a misplaced for loop that executes the same database query times the number of items for every
item in the basket”
42. 42
Putting it all Together
• Metrics can help you to learn that there is a problem
• Distributedtracing becomes increasingly importantto understandmulti-tier execution paths and root causes
of problems
• Developersnow rely on metrics and traces to understandhow their service functionsin their microservice
architectures
• Pure Open Source solutionsare viable for pre-prod environments
• Standardization isthe only way to tackle today’scomplexity and Open Source is the key driver
• Vendorsare prepared to tap intodata collected by Open Source standard toolsto add enterprise features on
top to support web-scale workloads