The value of observability in a production microservice system

1. DATE OR VENUE
The value of
observability in a prod
microservice ecosystem
XPDAYS2019 – 23/11/2019

2. System overview
150

3. System overview
150 15

4. System overview
150 15

5. WHOAMI
mgovorischev maksym_govorischev
mgovorischev@gmail.com

6. System architecture

7. System architecture

8. System architecture

9. System architecture

10. System architecture

11. System architecture

12. System architecture

13. System architecture

14. Potential failure points

15. Potential failure points

16. Potential failure points

17. Potential failure points

18. Potential failure points

19. Issue mitigation protocol – priority 1
Service interruption

20. Issue mitigation protocol – priority 1
Service interruption
Acknowledge alert (45 mins)

21. Issue mitigation protocol – priority 1
Service interruption
Acknowledge alert (45
mins)
Begin resolution (+30
mins)

22. Issue mitigation protocol – priority 1
Service
interruption
Acknowledge
alert (45 mins)
Begin
resolution (+30
mins)
Update every
120 mins (MS
Teams)

23. Issue mitigation protocol – priority 1
Service
interruption
Acknowledg
e alert (45
mins)
Begin
resolution
(+30 mins)
Update
every 120
mins (MS
Teams)
Publish
post-
mortem (72
hours)

24. How it looks like in production
500
POST /subscriptions

25. How it looks like in production
500
POST /subscriptions

26. Here comes observability
observability
is a measure of how well internal states of a system can
be inferred from knowledge of its external outputs.

27. Three pillars of observability

28. Monitoring

29. Datadog metrics collection
Service
Datadog
agent
ECS Cluster
Cluster instance
ECS Task

30. Monitoring abstractions
aws.ecs.service.cpuutiliz
ation
Metric
Service CPU Utilization
Monitor
Overall Service State
Dashboard

31. Metrics

32. Monitors

33. Average vs percentiles

34. Metrics, that we pay attention to
• Resource utilization – CPU, Memory, container pool

35. Metrics, that we pay attention to
• Resource utilization – CPU, Memory, container pool
• API responses – response time (SLA), status codes

36. Metrics, that we pay attention to
• Resource utilization – CPU, Memory, container pool
• API responses – response time (SLA), status codes
• Structure of load – (IP vs PUA vs SSO vs Legacy auth)

37. Dashboards

38. Correlation between metrics

39. Logging with Sumologic

40. Log aggregation flow
Service
ECS Cluster
Cluster instance
ECS Task
HTTP Source

41. Things to log
1
POST /subscription
Incoming requests

42. Things to log
1
POST /subscription
2
Lookup customer
Create customer
Taxes and coupons
Create subscription
Activate user
Workflow steps

43. Things to log
1
POST /subscription
2
Lookup customer
Create customer
Taxes and coupons
Create subscription
Activate user
3
Outcoming
requests

44. Things to log
1
POST /subscription
2
Lookup customer
Create customer
Taxes and coupons
Create subscription
Activate user
3
Outcoming
requests
Anomalies

45. Things to log
1
POST /subscription
2
Lookup customer
Create customer
Taxes and coupons
Create subscription
Activate user
3
4 Data manipulation

46. Things to log
1
POST /subscription
2
Lookup customer
Create customer
Taxes and coupons
Create subscription
Activate user
3
4 Data manipulation
Audit trail

47. Things to log
1
POST /subscription
2
Lookup customer
Create customer
Taxes and coupons
Create subscription
Activate user
3
4
5
Availability

48. Things not to log

49. PII Data
Linked Linkable

50. Additional notes
• INFO, WARN, ERROR – enough

51. Additional notes
• INFO, WARN, ERROR – enough
• Internal vs external logs

52. Additional notes
• INFO, WARN, ERROR – enough
• Internal vs external logs
• Introduce correlation IDs

53. Distributed tracing with Zipkin

54. Zipkin collection flow
App code Trace instrumentation Http Client Zipkin Collector
GET /foo
Service A

55. Zipkin collection flow
App code Trace instrumentation Http Client Zipkin Collector
GET /foo
record tags and
timestamp
Service A

56. Zipkin collection flow
App code Trace instrumentation Http Client Zipkin Collector
GET /foo
record tags and
timestamp
add trace headers
Service A

57. Zipkin collection flow
App code Trace instrumentation Http Client Zipkin Collector
GET /foo
GET /foo
X-B3-TraceId: aa
X-B3-SpanId: 6b
record tags and
timestamp
add trace headers
Invoke
request
Service A

58. Zipkin collection flow
App code Trace instrumentation Http Client Zipkin Collector
GET /foo
200 OK
GET /foo
X-B3-TraceId: aa
X-B3-SpanId: 6b
record tags and
timestamp
add trace headers
record duration
Invoke
request
200 OK
Service A

59. Zipkin collection flow
App code Trace instrumentation Http Client Zipkin Collector
GET /foo
200 OK
GET /foo
X-B3-TraceId: aa
X-B3-SpanId: 6b
record tags and
timestamp
add trace headers
record duration
async span report
Invoke
request
200 OK
Service A

60. Zipkin metadata

61. Zipkin collection flow
Service A
Service B
Service C
Zipkin Collector
Query Service
DB
Zipkin Web UI

62. Trace example

63. Spring Boot Configuration

64. NodeJS configuration

65. Additional notes
• Proactive vs reactive usage

66. Additional notes
• Proactive vs reactive usage
• Use 20/80 approach proactively

67. Additional notes
• Proactive vs reactive usage
• Use 20/80 approach proactively
• Nice to have: combination of tracing and logging

68. Additional notes
• Proactive vs reactive usage
• Use 20/80 approach proactively
• Nice to have: combination of tracing and logging
• Custom annotations and naming where needed

69. You need to watchout!

70. Opsgenie for the rescue

71. Opsgenie integration with other tools

72. Opsgenie integration with other tools
alerts

73. Opsgenie integration with other tools
alerts notifications

74. On-call schedules

75. Escalation policies

76. Closing notes
• 2/3 is fine, but…

77. Closing notes
• 2/3 is fine, but…
• Make observability a daily routine

78. Closing notes
• 2/3 is fine, but…
• Make observability a daily routine
• Bounded context & code ownership

79. Closing notes
• 2/3 is fine, but…
• Make observability a daily routine
• Bounded context & code ownership
• Our progress: 3 weeks -> -2 hours