Go monitoring at Lazada

1. GO Monitoring at Lazada
Rinat Takhautdinov & Ninh Dang
Senior Software Engineers at PDP Vertical Team – Lazada

2. Golang meetup # 11
About Us – PDP Vertical Department
- We’re doing product development at Lazada
- Product Detail Page
- Home Page
- Login/Signup
- Work mostly with Golang and PHP

3. Golang meetup # 11
- Overview of time-series metrics monitoring
- Big picture
- Prometheus
- Grafana
- How monitoring helps us at Lazada
- Use case 1: Product Ratings & Reviews performance optimization
- Use case 2: Shaping the product for Buyer-Seller Communication
- Conclusions
Agenda

4. Golang meetup # 11
- Overview of time-series metrics monitoring
- Big picture
- Prometheus
- Grafana
- How monitoring helps us at Lazada
- Use case 1: Product Ratings & Reviews performance optimization
- Use case 2: Shaping the product for Buyer-Seller Communication
- Conclusions
Agenda

5. Golang meetup # 11
Monitoring: General Architecture
http://service:port/metrics

6. Golang meetup # 11
What does Prometheus do?
• Time series database
• Written in Go - no external
dependencies
• Powerful functional query
language
• Pull, not push
• Thousands of targets
• Hundreds of thousands of
samples per second
• Millions of time series
• Distributed (or not)
a) can be run as a single node
b) can be run as a cluster for
redundancy & performance

7. Golang meetup # 11
Exporters and direct instrumentation
Client libraries
https://prometheus.io/docs/instrumenting/clientlibs/
Official
Go (https://github.com/prometheus/client_golang)
Java (JVM)
Ruby
Python
Unofficial
.NET / C#
Node.js
Bash
Exporters
https://prometheus.io/docs/instrumenting/exporters/
Official
Node/system metrics exporter
Graphite exporter
Collectd exporter
JMX exporter
HAProxy exporter
StatsD bridge
AWS CloudWatch exporter
Hystrix metrics publisher
Mesos task exporter Consul exporter
Direct instrumentation
Kubernetes
Kubernetes-Mesos
Etcd
gokit
go-metrics instrumentation library

8. Golang meetup # 11
Metrics and labels
Metric types:
- counter
- gauge
- histogram
- summary
var httpResponsesTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Namespace: "my_super_puper_production",
Subsystem: "http_server",
Name: "http_responses_total",
Help: "The count of http responses issued, classified by
code and method.",
},
[]string{"code", "method"},
)
Listing 1
CAUTION: Remember that every unique combination of key-
value label pairs represents a new time series, which can
dramatically increase the amount of data stored. Do not use
labels to store dimensions with high cardinality (many different
label values), such as user IDs, email addresses, or other
unbounded sets of values.

9. Golang meetup # 11
Powerful query language
topk(3, sum(rate(bazooka_instance_cpu_time_ns[5m])) by (app, proc))
sort_desc(sum(bazooka_instance_memory_limit_bytes -
bazooka_instance_memory_usage_bytes) by (app, proc))
Expression browser

10. Golang meetup # 11
Grafana

11. Golang meetup # 11
Grafana

12. Golang meetup # 11
Grafana

13. Golang meetup # 11
- Overview of time-series metrics monitoring
- Big picture
- Prometheus
- Grafana
- How monitoring helps us at Lazada
- Use case 1: Product Ratings & Reviews performance optimization
- Use case 2: Shaping the product for Buyer-Seller Communication
- Conclusions
Agenda

14. Golang meetup # 11
Product Ratings & Reviews

15. Golang meetup # 11
Problem – High level
- Lazada has many campaigns throughout the year
- How do we prepare for the campaigns?
- Stress tests + monitoring
- Find & fix bottlenecks
- Verify results
- What happened to us?
- Site was broken, checkout not working
- Backend service alert (Review API)

16. Golang meetup # 11
Review API architecture

17. Golang meetup # 11
What do we monitor for our Review API?
- CPU / memory
- Goroutines
- Avg RT, RT per handler
- Error rates
- Throughput [per clients]
- DB connection
- Cache size (number of items)
- Cache hits / misses (utilization)
- …

18. Golang meetup # 11
What do we monitor for our Review API?

19. Golang meetup # 11
What do we monitor for our Review API?

20. Golang meetup # 11
What we found in our case?
- Number of cache items went up very high
- Review API version fragmentation
- Many calls on old version API endpoints

21. Golang meetup # 11
Optimization in action
1. Reduce number of calls from clients.
2. Force all clients move to use the latest versions.
3. Deprecate outdated handlers (keep multi-version of API is expensive)

22. Golang meetup # 11
Feel rock?

23. Golang meetup # 11
Business metrics monitoring for product development
Product delivering focused team
Many kind of KPIs how to check product health need to tracked
A lot of ideas how to make our products better
Limited capacity to do it
Smooth rollout and looking for better way

24. Golang meetup # 11
Business metrics monitoring for product development

25. Golang meetup # 11
Business metrics monitoring for product development
- Helps us to validate and evolve our MVP
- Transparent for everyone (e.g. dev team, country PMs)
- Real time tracking product health

26. Golang meetup # 11
Conclusions
- Having monitoring is extremely critical for any business
- If you have many teams, having standard is important
- Define most important metrics for each microservice
- Have some TV screens with some graphs near your team
corner!

27. Golang meetup # 11
Q&A

28. Golang meetup # 11
- WE ARE HIRING!
- 4 x Senior Backend Developers (GO/PHP)
- Contact us for more information:
- ninh.dang@lazada.com
- rinat.takhautdinov@lazada.com