Given at CraftConf 2015: http://craft-conf.com/2015 Accompanying talk repo: https://github.com/randommood/Craftconf2015 Video: http://www.ustream.tv/recorded/61449003 Let's face it, testing distributed systems is hard. Large number of inputs, partial failures, and asynchrony make these systems seemingly impossible to verify. Formal testing methods are onerous and system correctness continues to elude us. Despite the difficulty, we need to have a way to change our systems with confidence. Integration and unit tests are vital to our sanity and we know it's good for us to use continuous integration to ensure functional consistency across our products. Come to this talk for a fresh and practical look into the various aspects of distributed systems testing and integration. We will cover CI pipelines: their strengths, weaknesses, and ways to improvement them. Get ready to rediscover the untapped power of your integration practices and develop a burning desire to make them more awesome!

1. Testing & Integration
THE REMIX!

2. Ines
Sombra
ines@fastly.com
@Randommood
!
"
#

3. Globally distributed & highly available

4. DS Testing
Continuous
Integration
Bring Home
"
!
$
Our Agenda
DS =
Distributed
Systems

5. Visual Guide ^
Distributed Systems Testing
" $!

6. Why do we test?
We test to gain
confidence
that our system
is doing the
right thing
(now & later)
" $!

7. MEDIUM
Unit
Integration
System
Acceptance
SMALL
Unit
Integration
(maybe)
COMPLEX SYSTEM
Fault injection
Stress /
performance
Canary
Regression
Unit
Integration
System
Acceptance
Compatibility
Many Types of Tests " $!

8. Nondeterminism
Timing
Failures
Unbounded inputs
Ordering
Many states
Concurrency
No centralized view
Challenges of DS Testing " $!

9. Behavior is aggregate
Components tested in isolation
also need to be tested
together
?
!
A
!
B
" $!Challenges of DS Testing

10. Reliability of a linear system is the
product of the reliability of each of the
system’s components
" $!Challenges of DS Testing

11. .95 * .95 * .95 = .85
Your system is only
85% reliable
Component A
95% reliable
!
Component B
95% reliable
!
Component C
95% reliable
!
" $!Challenges of DS Testing

12. Hierarchy of Errors*
BYZANTINE FAILURES
(fail by doing whatever I want)
OMISSION FAILURES
(fail by dropping messages)
CRASH FAILURES
(fail by stopping)
* Stolen from Henry Robinson’s PWLSF talk
Deadlocks
Livelock / starvation
Under specification
Over specification
" $!

13. Distributed Systems Testing
Difficult to approach &
many factors in play
Behavior is aggregate
Goal is to gain
confidence of proper
system behavior now
& later
" $!

14. Continuous Integration
CI for
short
" $!

15. Waiting till the end of a project to integrate
leads to 🔥🔥 hell 🔥🔥 (a special kind)
" $!

16. What is CI?
Development practice
Integrate code to
master branch daily
Each integration is
automatically verified
" $!

17. Why CI Today?
Often overlooked/
Taken for granted
“Eat your veggies”
Impacts your risk
& quality
" $!

18. Commit frequently
100% test must pass
Fix broken tests
Automation is 👑
CI Practices
" $!

19. CI gives us
Rapid feedback
Rapid deployment
Repeatable
automated testing
Reduce risk
" $!

20. Revisiting Tests
Differentiated
by required setup
which affects
running time
Unit
Integration
System
Functional
(also Smoke /Acceptance)
" $!

21. CI Challenges
Complex systems /
dependencies
Multiple architectures
Type of applications &
types of tests
" $!

22. CI Challenges " $!
Overfitted/ brittle / slow tests
Writing good tests is hard
Maintenance
Consider Regulations (PCI)

23. “Manage
the CI
System
or be
controlled
by manual
processes”
Is it
worth it?
" $!

24. You integrate to master
very often (daily)
All of your tests pass &
you fix broken builds!
Success indicators: #
commits / build status
Am I doing CI right? " $!

25. CI In the Wild
" $!

26. Tools & Services
How to
pick?
Security, Cost
Language
support
Services &
integrations
Scalability
" $!

27. " $!

28. CI In the Wild
Branches & pull
requests. Merges
& deploys
Consolidated
view of build
statuses across
all codebases
" $!

29. Master
PRs
Kick off a
suite
Stacks
OS + Our
Image
Scenario
Live setup +
assertions
Suite
Collection of
scenarios
My Assertions:
Installing, backing
up/restoring,
setting up,
replicating,
promoting, etc
Any prod
deploy
Kicks off a
suite
" $!
Dredd
Tests
Systems,
boundaries, &
integration
CI In the Wild

30. CI In the Wild " $!
* Stolen from Google’s talk on
their Continuous Build System

31. " $!CI In the Wild

32. * Slide stolen from Seth Falcon’s ChefConf 2015 talk
CI In the Wild: CD tools* " $!

33. CI Matters
Predictor of
professional
maturity at the
organizational &
individual level
Adds visibility &
allows us to manage
risk and quality
" $!
🏆

34. How are we doing?

35. Bringing it home
" $!

36. " $!Getting Started
Test the full distributed system. This means testing
the client, system, and provisioning code!
HAVE DECENT TESTS
ADD VISIBILITY
VERIFY &
ENHANCE

37. CI & Code Quality
Automated inspection
as part of the build
Customizable by Org
Test coverage
Code duplication
" $!

38. CI & Quality " $!
Languages help you: go
cov, race detector,
go lint
Infrastructure testing
matters
But beware of
misguided processes
😩

39. CI & Quality
Cyclomatic Complexity
Number (CNN)
Afferent/Efferent
Coupling
Software instability as
“quantifiable”
" $!

40. And more! " $!
Use it for docs
generation
Architectural/design
diagrams too !
Accessibility testing
Used to test mobile
architectures too
📄

41. Pro-Tips " $!
Add CI early
Different machine
for integrating &
put everything you
need to built in git
Parallelize builds
to scale up or down

42. Rapid feedback is
critical. Slow is 😭💀
Staggered builds: run
fast tests first, then
second phase kicks in
The more likely a test is
to fail the sooner you
should execute it
" $!Pro-Tips

43. Tl;DR
DS TESTING
Different
types of tests
Can be tricky
Behavior is
aggregate
CI PRACTICES
Everyone commits
to baseline daily
Automate build &
make it self-
testing
Keep builds fast
Everyone can see
builds & status
ON QUALITY
CI can be used to
drive good coding
standards
Some metrics
available to gauge
stability / quality
Keep track of them

44. Rantifesto:
Build a
Culture of
Quality
✨The technology
is just a tool but
your culture is
really the key ✨
Lower the risk
of change with
tools & culture
" $!

45. “Quality
means doing
it right when
no one is
looking”
Henry Ford

46. Thank
You!
Special thanks to: Caitie
McCaffrey, Jordan West, Jon
Hyman, Bruce Spang, Devon
O’Dell, Kelsey Gilmore-Innis,
Aysylu Greenberg, Alan
Kasindorf, Paul Reed, André
Arko, Mike O’Neill, Thomas
Mahoney, Eric Kustarz, Ian
Fung, Neha Narula, Karl
Smith, and Greg Bako.
ines@fastly.com | @Randommood | github.com/Randommood/CraftConf2015

47. Questions?
ines@fastly.com | @Randommood | github.com/Randommood/CraftConf2015