Apply best parts of microservices to serverless
•
3 likes•1,637 views
Whenever a new paradigm comes along, we often cast the previous incumbents as relics to be forgotten by history, only to then repeat the same mistakes as they once did. On the surface Serverless has revolutionised how we build and run software, but deep down we are still building microservices and face the same challenges. As more of us adopt Serverless and build increasingly complex systems using this new paradigm, it's important to take a moment to reflect on the lessons others have learnt about building microservices and how they can be applied to our Serverless applications.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apply best parts of microservices to serverless
Similar to Apply best parts of microservices to serverless (20)
More from Yan Cui
More from Yan Cui (20)
Recently uploaded
Recently uploaded (20)
Apply best parts of microservices to serverless
- 1. applying the best parts of Microservices to Serverless
- 4. follow @DAZN_ngnrs for updates about the engineering team We’re hiring! Visit engineering.dazn.com to learn more.
- 6. 2006
- 7. 2010
- 8. 2010
- 9. 2016
- 10. 2016
- 11. SQL NoSQL OOP Functional On Premise Cloud Waterfall Agile Monoliths Microservices
- 17. what’s this?
- 18. what’s this? this solves all my problems!
- 19. what’s this? this solves all my problems! this is rubbish!
- 20. what’s this? this solves all my problems! this is rubbish! I’m starting to get it..
- 21. what’s this? this solves all my problems! this is rubbish! I’m starting to get it.. I know what I’m doing
- 22. SQL NoSQL OOP Functional On Premise Cloud Waterfall Agile Monoliths Microservices Server-ful Serverless
- 23. “those who cannot remember the past are condemned to repeat it” - George Santayana
- 24. what’s this? this solves all my problems! this is rubbish! I’m starting to get it.. I know what I’m doing
- 25. lesson 1. don’t fly blind
- 30. These are the four pillars of the Observability Engineering team’s charter: • Monitoring • Alerting/visualization • Distributed systems tracing infrastructure • Log aggregation/analytics “ ” http://bit.ly/2DnjyuW- Observability Engineering at Twitter
- 31. NO ACCESS to underlying OS
- 33. user request user request user request user request user request user request user request critical paths: minimise user-facing latency handler handler handler handler handler handler handler
- 34. user request user request user request user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead
- 35. user request user request user request user request user request user request user request critical paths: minimise user-facing latency StatsD handler handler handler handler handler handler handler rsyslog background processing: batched, asynchronous, low overhead NO background processing except what platform provides
- 37. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now?
- 38. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now? UTC Timestamp Request Id your log message
- 40. one log group per function one log stream for each concurrent invocation
- 41. logs are not easily searchable in CloudWatch Logs me
- 42. CloudWatch Logs
- 43. CloudWatch Logs AWS Lambda ELK stack
- 45. you need to use structured logging me
- 47. CloudWatch Logs $0.50 per GB ingested $0.03 per GB archived per month
- 48. CloudWatch Logs $0.50 per GB ingested $0.03 per GB archived per month 1M invocation of a 128MB function = $0.000000208 * 1M + $0.20 = $0.408
- 49. DON’T leave debug logging ON in production
- 50. you need to sample debug logs in production me
- 51. volume of logs observability all debug logs no debug logs sampled debug logs hurts mean time to resolution (MTTR) during a production incident
- 52. volume of logs observability all debug logs no debug logs sampled debug logs $$$$$$
- 53. always log the invocation event on error me
- 64. my code send metrics internet internet press button something happens
- 65. those extra 10-20ms for sending custom metrics would compound when you have microservices and multiple APIs are called within one slice of user event
- 66. Amazon found every 100ms of latency cost them 1% in sales. http://bit.ly/2EXPfbA
- 67. console.log(“hydrating yubls from db…”); console.log(“fetching user info from user-api”); console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”); console.log(“MONITORING|1489795335|8|count|yubls-served”); timestamp metric value metric type metric namemetrics logs
- 68. CloudWatch Logs AWS Lambda ELK stack logs m etrics CloudWatch
- 70. API Gateway send custom metrics asynchronously
- 71. SNS KinesisS3API Gateway … send custom metrics asynchronously send custom metrics as part of function invocation
- 74. ? functions are often chained together via asynchronous invocations
- 76. ? SNS Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 SES tracing ASYNCHRONOUS invocations through so many different event sources is difficult
- 77. X-Ray
- 79. do not span over API Gateway
- 80. narrow focus on a function good for homing in on performance issues for a particular function, but offers little to help you build intuition about how your system operates as a whole.
- 81. don’t span over async invocations good for identifying dependencies of a function, but not good enough for tracing the entire call chain as user request/data flows through the system via async event sources.
- 82. don’t span over non-AWS services
- 85. correlation-IDs
- 87. correlation IDs* * eg. request-id, user-id, yubl-id, etc.
- 88. kinesis client http client sns client
- 90. lesson 2. no shared DBs
- 91. shared DBs create TIGHT COUPLING between services
- 92. build loosely-coupled system through events
- 93. service A service B service C service D bounded context bounded context
- 94. service A service B service C service D bounded context bounded context
- 95. service A service B service C service D bounded context bounded context
- 96. service A service B service C service D bounded context bounded context
- 97. lesson 3. spiky load between services
- 98. service A service B
- 99. downstream systems might not be as scalable
- 101. service A service B Kinesis Lambda
- 102. service A service B Kinesis Lambda concurrency == no. of shards
- 104. service A service B Kinesis Lambda retried until success
- 106. lesson 4. failures are inevitable
- 107. complex distributed systems fail in.. well, complex, sometimes cascaded ways..
- 108. the only way to truly know your system’s resilience against failures is to test it through controlled experiments
- 109. there are more inherent chaos and complexity in a Serverless architecture
- 110. smaller units of deployment but A LOT more of them!
- 111. more difficult to harden around boundaries serverful serverless
- 113. ? Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 more intermediary services, and greater variety too SNS SES
- 114. ? Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 more intermediary services, and greater variety too each with its own set of failure modes SNS SES
- 115. more configurations, more opportunities for misconfiguration serverful serverless
- 116. more unknown failure modes in infrastructure that we don’t control
- 117. often there’s little we can do when an outage occurs in the platform
- 120. missing fallback when downstream is unavailable
- 121. FAILURE INJECTION
- 123. inject failures
- 124. inject failures validate failure handling
- 126. Recap
- 128. “those who cannot remember the past are condemned to repeat it” - George Santayana
- 129. don’t fly blind
- 130. no shared DBs
- 131. amortize spiky load between services