Go code instrumentation with Prometheus.

1. Globalcode – Open4education
Instrumenting Go applications with Prometheus
André Carvalho / @andresantostc
globo.com / tsuru.io
1

2. Globalcode – Open4education
André Carvalho
Developer @ globo.com
Open Source
tsuru maintainer
other contributions
2

3. Globalcode – Open4education
Agenda
Intro
Motivation
Instrumentation
Prometheus
Demo
3

4. Globalcode – Open4education
4
Why?
Service

5. Globalcode – Open4education
5
Why?
Service
Users

6. Globalcode – Open4education
6
Why?
Service
Users
Dependencies

7. Globalcode – Open4education
7
Why?
Service
Users
Dependencies
Your service is slow!

8. Globalcode – Open4education
8
Why?
Service
Users
Dependencies
Your service is slow!
Is the problem here?

9. Globalcode – Open4education
9
Why?
Service
Users
Dependencies
Your service is slow!
Or here?

10. Globalcode – Open4education
10
Why?
Service
Users
Dependencies
Your service is slow!
Maybe here?

11. Globalcode – Open4education
11
Why?
Service
Users
Dependencies
Your service is slow!
It has to be here! Or not…

12. Globalcode – Open4education
Ability to monitor or
measure the level
of a product's
performance, to
diagnose errors
and to write trace
information.
12
Instrumentation

13. Globalcode – Open4education
How & what?
USE
Brendan Gregg
Utilization
Saturation
Error rate
RED
Tom Wilkie
Request rate
Error rate
Duration
13

14. Globalcode – Open4education
14
Prometheus

15. Globalcode – Open4education
Prometheus
Monitoring and Alerting system (https://
prometheus.io/)
Created by SoundCloud
Written in Go
Open Source
Member of the Cloud Native Computing
Foundation
15

16. Globalcode – Open4education
Prometheus
Pull based (HTTP)
Easier to maintain and scale
Flexible query language
Service discovery
Kubernetes, AWS, Azure, Consul, …
Several exporters already available
Node, JMX, Mysqld, …
Easy to build your own!
16

17. Globalcode – Open4education
Prometheus
17

18. Globalcode – Open4education
connections_current_open{instance="192.168.50.10:9080",job="router"} 165
Prometheus Metrics
18

19. Globalcode – Open4education
connections_current_open{instance="192.168.50.10:9080",job="router"} 165
Name
Prometheus Metrics
19

20. Globalcode – Open4education
connections_current_open{instance="192.168.50.10:9080",job="router"} 165
Labels
Prometheus Metrics
20

21. Globalcode – Open4education
connections_current_open{instance="192.168.50.10:9080",job="router"} 165
Value
Prometheus Metrics
21

22. Globalcode – Open4education
Go, Prometheus!
Official client
https://github.com/prometheus/client_golang
Out of the box runtime metrics
Goroutines, GC latency, OS threads
Supported metrics
Counter, Gauge, Histogram and Summary
prometheus.Collector interface
22

23. Globalcode – Open4education
github.com/andrestc/demo-go-instrument
Demo
23

24. Globalcode – Open4education
24
GET /city/{name}/temp
Redis
api.openweathermap.org
Queue
Demo
localhost:8080

25. Globalcode – Open4education
25
Redis
api.openweathermap.org
USE
RED
RED
Utilization
Saturation
Error rate
Request rate
Error rate
Duration
Demo
GET /city/{name}/temp
localhost:8080

26. Globalcode – Open4education
Demo
26

27. Globalcode – Open4education
Obrigado!
github.com/andrestc
@andresantostc
27