Loki is a prototype OpenTracing implementation written in Go thats takes the Prometheus service-discovery and pull based approach to distributed tracing.
6. In reality, this is attempt #2
• Prototype “Weave Tracer” circa
2015
• Concept didn’t require application
instrumentation
• Used ptrace to intercept syscalls
and infer application behaviour
• Kinda worked, for a very limited
definition of “worked”
11. • Jobs must know where monitoring is
• Can overwhelm graphite with too many
samples
Graphite Prometheus
scraping
your jobs
Prometheus
pushing
your jobs
Graphite
• Tell Prometheus where jobs are (via service
discovery)
• Prometheus can back off when overwhelmed
• Prometheus knows the identity of each job
13. http://job/traces
your job
Loki client library
spans
scraping
Loki
• Client library keep pending
spans in an in-memory ring
buffer.
• /traces HTTP handler
grabs all the in-memory
spans and serialises them
using Thrift.
• Spans will be dropped if not
collected frequently enough.
14. • Retrieval library ‘knows’ identity of scraped
endpoints, adds that to received spans
• … jobs don’t need to know their own
identity
• … can be consistent with identity used in
Prometheus
• Naive in memory storage implementation
• … makes queries slow, as its just a loop.
• Zipkin-compatible API endpoints
• UI _is_ the Zipkin UI
Loki
Prometheus retrieval library
In memory
storage
Zipkin API
Zipkin UI
15. Its all open source:
https://github.com/weaveworks-experiments/loki
…and it’s written in go
24. Client Library
• Make is support multiple scrapers
• Move away from thrift to protos
• More languages
• Useful HTML /traces
Loki Server
• Local storage with BoltDB
• Make queries faster
• Make it distributed, use cloud
storage
TODO