This document appears to be a slide deck presentation on testing in production and microservices testing automation. It contains over 60 slides on topics such as blue-green deployments, canary releases, dark launches, feature flagging, service virtualization techniques for testing, and strategies for gradually rolling out changes to production. The slides include diagrams, code snippets, and quotes to illustrate different concepts in testing microservices in production environments.

1. Testing in Production.
Microservices testing &
automation
Alex Soto Bueno
@alexsotob

2. @alexsotob
2
Buenas tardes, buenas noches,
señoritas y señores
To be here with you tonight.
Brings me joy, que alegria
— Miguel
“

3. @alexsotob
3
Who Am I?
Alex Soto

4. @alexsotob
Raffle Rules (applicable in the real)
4
1. Follow: @alexsotob
2. With picture of the session
3. Mention @alexsotob
4. With hashtag #DevNationLive

5. @alexsotob5
Questions

6. @alexsotob6
I think that is the
most stupidest thing
I ever heard it.
— Gideon Grey
“

7. @alexsotob7

8. @alexsotob8
https://puppet.com/resources/whitepaper/state-of-devops-report
High-performing
organizations are decisively
outperforming their lower
performing peers ...

9. @alexsotob9
https://puppet.com/resources/whitepaper/state-of-devops-report
46X46X More Deployments

10. @alexsotob10
https://puppet.com/resources/whitepaper/state-of-devops-report
440X440X Shorter Lead Times

11. @alexsotob11
https://puppet.com/resources/whitepaper/state-of-devops-report
96X96X Mean Time to Recover

12. @alexsotob12
Change is good.
— Rafiki
“

13. @alexsotob13
OPS PMO DBASEC
BIZ DEV
QA

14. @alexsotob14
Continuous
integration
and testing
Build
Test
Continuous
delivery and
deployment
Release
Deploy
Continuous
operations
Operate
Monitor
Continuous Improvement
Continuous Improvement

15. @alexsotob15
QA Bottleneck

16. @alexsotob16
I don’t want to survive.
I want to live.
— Captain B. McCrea
“

17. @alexsotob17
We are on Production

18. @alexsotob18
Shit! We break Production Environment

19. @alexsotob19
When you find out
you can live without it,
go along not thinking
about it.
— Baloo
“

20. @alexsotob20
Production is not sacrosanct anymore

21. @alexsotob21
- Unit Tests
- Component Tests
- Static Analysis
- Coverage Tests
- Benchmark Tests
- Contract Tests
- Acceptance Tests
- Mutation Tests
- Smoke Tests
- UI/UX Tests
- Penetration Tests
- Integration Tests
- Tap Compare
- Load Tests
- Shadowing
- Config Tests
- Canarying
- Dark Canaries
- Monitoring
- Feature Flagging
- Exception Tracking
- Feature Graduation
- Teeing
- Profiling
- Logs
- Chaos Testing
- Monitoring
- A/B Testing
- Tracing
- Auditing
- OnCall Experience
- Journey tests
Cindy Sridharan
Pre-Production
Testing In Production
Deploy Release Post Release
The New ¿Pyramid?

22. @alexsotob22
There is a lot of grey area in
make me a prince.
— Gene
“

23. @alexsotob23
Regression-free versions (tap compare)

24. @alexsotob24
Diferencia OpenDiffy

25. @alexsotob25
Simple Comparison

26. @alexsotob26
Blue-Green Deployments

27. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
BUILD
SCM

28. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
BUILDSCM

29. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
SCM

30. Blue/Green Deployment
CD/CD PRODUCTION ROUTER USERS
SCM

31. @alexsotob31
Canary Release

32. @alexsotob32
CI/CD PRODUCTION ROUTER USERS
SCM

33. @alexsotob33
CI/CD PRODUCTION ROUTER USERS
SCM

34. @alexsotob34
CI/CD PRODUCTION ROUTER USERS
SCM

35. @alexsotob35
CI/CD PRODUCTION ROUTER USERS
SCM

36. @alexsotob36
CI/CD PRODUCTION ROUTER USERS
SCM

37. @alexsotob37
CI/CD PRODUCTION ROUTER USERS
SCM

38. @alexsotob38
CI/CD PRODUCTION ROUTER USERS
SCM

39. @alexsotob @AndyGeeDe39
Dark Launches

40. @alexsotob40
DEVELOPMENT QA STAGING PRODUCTION ROUTER USERS
SCM
INTERNAL USERS
Dark Canaries

41. @alexsotob41
DEVELOPMENT QA STAGING PRODUCTION ROUTER USERS
SCM
Shadowing Traffic

42. @alexsotob42
Stubbing collaborating services
Service A
Service
Virtualization
(Pod Virtualization)
X
Service B v1
Service B v2
Host:”serviceb-shadow”
Service C Service D…Z
Prod DB

43. @alexsotob43
Synthetic transactions in mirroring traffic
Service A
Service B v1
Service B v2
Host:”serviceb-shadow”
synthetic:true
rollback

44. @alexsotob44
Virtualizing test-cluster’s database in mirroring
Service A
Service B v1
Service B v2
Host:”serviceb-shadow”
Prod DB
Virtual DB
read
Throwaway
write

45. @alexsotob45
Feature Graduation

46. @alexsotob46

47. @alexsotob47

48. @alexsotob48
This is the circle of sadness.
Your job is to make sure
that all sadness stays
inside of it.
— Joy
“

49. @alexsotob49

50. @alexsotob50

51. @alexsotob51

52. @alexsotob52
It's time to see
what I can do
To test the limits
and break through.
— Elsa
“

53. @alexsotob53
Istio — ‘Sail’
(Kubernetes — the ‘Helmsman’)

54. @alexsotob54
Put on your Sunday clothes
there's lots of world
out there.
— Wall-E
“
[http://bit.ly/istio-tutorial]
[http://bit.ly/istio-cheat-sheet-v1]

55. @alexsotob55
Every adventure
requires a first step.
— Alice
“

56. @alexsotob56
Start with the most sympathetic and innovative groups
MarketGrowth
Time
2.5% 13,5% 34% 34% 16%
The Chasm
Early
Adopters
Innovators
Early
Majority
Late
Majority
Laggards
The Technology Adoption Curve (Source: Moore and McKenna, Crossing The Chasm)

57. @alexsotob57
No chance, no way
I won't say it, no, no.
— Meg
“

58. @alexsotob58
And Now What?
Evangelism
Test Guardians
Bridge To Management
Define Strategies
Write Tests
No Manual Tests, No Maintenance Tasks

59. @alexsotob59
Now,
you see, the world
is full of temptations .
— Jimmy Cricket
“

60. @alexsotob60
If I go there is just no telling
how far I’ll go.
— Moana
“

61. @alexsotob
Hay un amigo en mí,
cuando salgan a volar,
hay un amigo en mí
— Toy Story
“
@alexsotob
asotobue@redhat.com
http://www.lordofthejars.com/
lordofthejars

62. https://bit.ly/2oXNwz9
DevNation London Survey

63. @alexsotob63
About Images
http://disney.wikia.com/wiki/File:TLK_Simba.png
http://disney.wikia.com/wiki/File:Tiana.12.png
http://insideout.wikia.com/wiki/File:FEAR_Fullbody_Render.png
http://disney.wikia.com/wiki/File:Judy_Hopps_Zootopia.png
https://mbtskoudsalg.com/explore/disney-up-png/#gal_post_3621_up-balloons-png.png
http://parody.wikia.com/wiki/File:Jiminy_cricket_disney.png
http://zootopia.wikia.com/wiki/File:Gideon_Adult_Transprent.png
http://lifeofheroesrp.wikia.com/wiki/File:Aladdin.png
http://disney.wikia.com/wiki/Captain_B._McCrea
http://pixar.wikia.com/wiki/Slim
http://pixar.wikia.com/wiki/Auguste_Gusteau
http://hero.wikia.com/wiki/File:Joy_(Inside_Out).png
http://disney.wikia.com/wiki/File:Diamond_baloo.png
https://www.flickr.com/photos/photogreuhphies/7383113142/in/photolist-cfqojb-pC8jJB-6hCtMr-a4hoxN-9gPWs9-6EHZzs-dV9aSt-81JzLb-85TxfK-qsw4px-81JAH3-9LBbLF-kAeQJ
qdoArQ-pC7bTE-cDWcPJ-doZAgW-aZhdma-oFEs5b-a1865x-23TdcqE-7oLqCo-8TWx33-9Aazdw-5hFSPs-pUCGJQ-oGPNnf-pCa8P9-bAdBVK-51ooTn-nsZnCK-bvV1nu-92f6Ys-p1rKvB-
http://www.stickpng.com/img/comics-and-fantasy/the-incredibles/edna-e-mode
https://www.kisspng.com/png-dory-nemo-palette-surgeonfish-disney-infinity-3-0-916908/
https://www.kisspng.com/png-frozen-olaf-s-quest-elsa-kristoff-anna-anna-frozen-825204/
https://www.kisspng.com/png-eve-wii-wallpaper-wall-e-png-photos-421264/
https://www.kisspng.com/png-rapunzel-free-png-image-63202/
https://www.kisspng.com/png-ratatouille-french-cuisine-film-animation-pixar-ra-631822/
https://www.kisspng.com/png-peter-pan-tinker-bell-wendy-darling-clip-art-fly-655477/
https://gallery.yopriceville.com/Free-Clipart-Pictures/Cartoons-PNG/Transparent_Snow_White_PNG_Clipart#.W_PigJNKgnU
https://www.kisspng.com/png-princess-jasmine-aladdin-genie-youtube-abu-aladdin-961044/
https://www.kisspng.com/png-a-bug-s-life-flik-youtube-film-streaming-media-bug-2538893/
https://www.kisspng.com/png-buzz-lightyear-jessie-toy-story-sheriff-woody-tim-103824/
https://www.kisspng.com/png-alice-s-adventures-in-wonderland-the-mad-hatter-wh-689107/
https://www.kisspng.com/png-russell-youtube-carl-fredricksen-pixar-film-ups-1441690/
http://disney.wikia.com/wiki/File:Bambi_-_Render.png
https://www.clipartmax.com/middle/m2H7H7Z5m2i8G6G6_joy-upside-down-transparent-png-stickpng-inside-out-inside-out-characters-sadness/
https://www.kisspng.com/png-bolt-ariel-the-walt-disney-company-walt-disney-ani-6179624/
https://www.clipartmax.com/middle/m2H7i8b1b1m2A0Z5_4-png-toy-story-slinky-png/
https://dlpng.com/png/115503
https://clipart.info/elsa-frozen-disney-png-6355
http://www.pngall.com/ariel-png
CC 3
Not Commercial
Usage
Not Commercial
Usage
CC 4