Managing microservices at scale is all about the tooling: smart deployment, testing in production, label based routing, critical path verification.

1. Microservices: next steps
Boyan Dimitrov,
Platform Automation Lead @ Hailo @nathariel

3. Back in 2011 we started simple
We quickly found out that supporting monoliths is hard:
• Hard to maintain the codebase
• Hard to build new features
• Hard to scale the dev teams
Failure to deliver business value
Frontend Backend
MySQL

4. TEST
PREPRO
D
LIVEBuild
Pipeline
So basically we had problems with this
weeks

5. So in 2013 we ended up
doing…

6. At present we have
• Microservices ecosystem (99.9% written in Go)
• Designed specifically for the cloud – different building blocks and
components will constantly be in flux, broken or unavailable
• 200+ services in production

7. TeVPC
Auto Scaling
S3
OrchestrationEnv DNS
Release AutoScaling
Discovery
Monitoring
CFEC2
Route 53
Redshift
ComputeEIP
Routing
Core
Platform
Provisioning
Login
Services
Cloud Provider
Whisper
Config

8. TEST
PREPRO
D
LIVEBuild
Pipeline
Did this thing get any faster
Hailo Warp Speed ( measured in many μs or something )

9. Until

10. The number of services increased

11. I need 10+ services to actually test a workflow locally!
What about all services dependencies (config, database schemas, etc )?
As a developer:

12. Meanwhile in staging
Staging

13. Staging

14. So we decided to give every developer an environment
ENV ENV
ENV

15. Environment Service
SIE MIE
Infrastructure
Core Platform
Single server on
AWS
Hundreds of servers
/ single AWS region
CloudFormation
Orchestration layer
On demand environments
Single Instance Environment Multi instance environment

16. Environment Service
SIE MIE
Infrastructure
Core Platform
Single server on
AWS
Hundreds of servers
/ single AWS region
Release Service
ANY ENV
(PROD)
Services
Config
*Data
clone
ETA: ~12 min ETA: ~40 min
CloudFormation
Orchestration layer
On demand environments
Single Instance Environment Multi instance environment

18. SIE
Pre Prod
master only
All of this so we can do
SIE
MIE
MIE
SIE
MIE
SIE Live
Orchestration

19. SIE
Preparing for…
SIE
MIE
MIE
SIE
MIE
SIE Live
Orchestration

20. But what do you need to test in production
in a microservices world?

21. Tooling

22. Reliable service provisioning and insights

23. Service provisioning at present
service name version
auto scaling group

24. Service A Service A
Traffic
5% 95%
Smart traffic shaping

26. Identify and measure your most critical workflows

27. Robomon
You need to identify your critical workflows

28. How do we know the critical path hasn’t changed?

29. Platform tiering

30. We tier down everything and check if the critical path succeeds
Robomon

31. And we only alert if this stops being the case
XX
X
Robomon

33. And one more thing…

34. Introducing label based routing
• A routing abstraction that allows you
to “group” services and isolate the
traffic between them
• Completely transparent for the
service itself
• Only traffic that is specifically tagged
for this group will ever hit a service
belonging to the group
• All traffic is contained within the
group if there is a recipient service
available there

35. service.foo
service.foo
Introducing label based routing
• Labels are forwarded with the request
context
• Even if a service needs to call
another service outside the group,
any further requests on the same call
chain will still “stick” to existing group
members

36. Fully dynamic
• Membership can be
configured on the fly
• You can have as many groups
as you want

37. What is this ?
• A way to flexibly curate traffic and test complex workflows in production
• A way to let developers deploy services simultaneously and still achieve
isolation
• A/B testing on steroids
• A very powerful tool when you get something wrong
• Many more use cases (multitenant platforms…)

38. What this is NOT…
• A tool that lets you tightly couple services (a.k.a. “distributed monoliths”)
and get away with it)
• A tool for releasing backwards incompatible changes
• A replacement for feature flags*

39. service.nearest-driver
service.driver-index
service.eta service.traffic
API
Layer
As an example…

40. service.nearest-driver
service.driver-index
service.eta service.traffic
API
Layer
As an example…

41. service.nearest-driver
service.driver-index
service.eta service.traffic
API
Layer
We can test app behavior with complex workflows without
affecting the end user in production

42. service.nearest-driver
service.driver-index
service.eta service.traffic
API
Layer
We can tag traffic at our API layer

43. service.nearest-driver
service.driver-index
service.eta service.traffic
service.experiment
API
Layer
We can tag traffic as part of an experiment

44. The reality…

45. Robomon will automatically verify the active service groups as well
Robomon

46. @nathariel
boyan@hailocab.com
@nathariel
boyan@hailocab.com
@HailoTech
Thanks!