OpenTelemetry for API Monitoring Danielle Kayumbi, Software Engineer - Capgemini Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

1. OpenTelemetry
For API Monitoring

3. Danielle KAYUMBI HODIEB
Software Engineer
danielleKayumbi

7. API Security
breaches

8. Impact
the business

9. AWS
Historic Outage
December 7th 2021

10. https://aws.amazon.com/fr/about-aws/global-infrastructure/localzones/locations/

11. . . .

12. Deliver packages

13. Netflix, Disney+ ...
Stream movies

14. Play online
Video games

15. https://www.supermarktblog.com/2017/11/13/antippen-statt-anstellen-wie-vapiano-und-mcdonalds-den-bestellprozess-im-schnellrestaurant-umkrempeln/
Order in fastfood
restaurants

16. Delta, Southwest ...
Buy airline tickets

17. Medical exams
Were delayed

18. And so where
Education exams

22. Deploy in prod
New automatic job
Scale up

23. Huge volume of requests

24. Huge volume of retry calls

25. Congestion

26. 9 hours to fix

27. How to prevent
& discover ?

28. Identify outages
Errors
Latency
Bottlenecks
Third parties

29. Alert !
Send
Relevant messages

30. Distributed systems

39. Amazon
Microservices
architecture
https://www.divante.com/blog/10-companies-that-implemented-the-microservice-architecture-and-paved-the-way-for-others

40. Netflix
More than
700
microservices
https://www.divante.com/blog/10-companies-that-implemented-the-microservice-architecture-and-paved-the-way-for-others

41. Observe the system

42. Monitoring the
operations from
time to time

43. Monitoring
is too
deterministic

44. Analyse data continuously

45. Understand the relationships
between systems

46. Observability
The key success

47. Monitoring
Observability

48. Traces Metrics
Logs
Telemetry
data

49. Generate, collect, export
Telemetry data

51. Help make a system
observable

52. Accepted in 2019
Moved to matury level in 2021

53. OpenTracing OpenCensus

54. Fundamentals

55. Distributed tracing

56. https://eng.blackbuck.com/distributed-tracing-at-blackbuck/
Follow request journey across various systems

57. Log

58. Log
Message emitted by services or
other components

59. [2024-04-18T09:05:08.906198+00:00]
app.INFO: Entity has been
successfully persisted
Log (Syslog sample)

60. Log
Lack contextual information
Tracking code execution

61. Log
Correlated with trace & spans
Included as part of a span

62. Span

63. Span
Unit of work or operation.
Tracks specific operations
that a request makes.

64. {
“name”: “get-customer-cart”
“context”: {
“trace_id”: “7bba9f33312b3dbb8b2c2c62bb7abe2d”,
“span_id”: “086e83747d0e381e”
},
“parent_id”: “”,
“start_time”: “2024-04-18 16:04:01.209458162 +0000 UTC”,
“end_time”: “2024-04-18 16:04:01.209458162 +0000 UTC”,
“status_code”: “OK”,
“attributes”: {
“http.method”: “GET”,
“user_id”: “123”,
“cart_id”: “456”
}}

65. Sensitive services
(command/query handlers…)
Controllers
(Http, AMQP actions…)
Repositories
(database, message broker, http client…)

66. Inject span

67. Annotation

69. Distributed trace

70. Distributed trace
Records the paths taken by requests,
Through multi-service architecture.
Trace = collection of spans

71. https://opentelemetry.io/

72. Cart
service
Product
service
API [GET]
product by id
API [GET]
cart by id

73. Get cart operation
Get products
Retrieve cart db
Get product operation
Retrieve product db
Trace
Cache
product
Context
Product service
Cart service

74. Context propagation
Keep correlation between signals
Handle by instrumentation librairies
W3C Recommendation
for interoperability

75. https://www.datadoghq.com/

76. OpenTelemetry collectors

77. https://docs.lightstep.com/docs/quick-start-collector

78. https://www.elastic.co/blog/best-practices-instrumenting-opentelemetry

79. https://www.datadoghq.com/

80. Service Level
Management
Four golden signals
Monitoring
Dashboards

81. Four
Golden
Signals
Latency
Traffic
Saturation
Errors

82. Latency
https://cloud.google.com/blog/products/management-tools/the-right-metrics-to-monitor-cloud-data-pipelines

83. Traffic
https://cloud.google.com/blog/products/management-tools/the-right-metrics-to-monitor-cloud-data-pipelines

84. Errors
https://www.datadoghq.com/

85. Saturation
https://cloud.google.com/blog/products/management-tools/the-right-metrics-to-monitor-cloud-data-pipelines

86. https://www.datadoghq.com/

87. Service level

88. Service Level Objective
“Less than 1% of users
should experience an idle of 5s”

89. Service Level Indicator
Defined quantitative measure

90. Availability, latency
“How many requests could be handled ?”

91. AWS outage resolution

92. https://www.thousandeyes.com/blog/aws-outage-analysis-dec-7-2021

93. Comprehensive observability

95. Thank you !
danielleKayumbi