My presentation from Velocity 2015 - how to scale organizations and technology

1. Systems Thinking
Patterns for scaling unstable systems
@_siddharth_ram
Siddharth Ram
Chief Architect, Small Business Group
Intuit

2. Intuit Confidential and Proprietary2
Intuit’s Mission
To improve our customers’ financial lives so profoundly …
they can’t imagine going back to the old way
Consumers Small Businesses Accounting Professionals

3. Intuit Confidential and Proprietary6
Recognized as One of the World’s Leading Companies
FORTUNE 100 BEST COMPANIES TO WORK FOR
2013%
MOST ADMIRED: SOFTWARE INDUSTRY MOST INNOVATIVE COMPANIES
2014%2015
31
Ranked
#
12 Years in a Row
2004 2005 2006 2007 2008 2009 2010 2011 2012 2015
WORLD’S MOST ADMIRED COMPANIES
2013 2014
14 Years in a Row
2
Ranked
#

4. • #1 Business management solution for small businesses and
accountants worldwide
• More than 5M paid subscribers
• Available via Web, native tablets/mobile apps
• Available in 140 countries
• Translated into 12 languages
About QuickBooks

6. 1. Optimism bias

8. Actual Perform
anceEstimated Performance
Overestimation
Underestimation
TestScore
Quartiles

9. Unskilled and Unaware of It:
How Difficulties in Recognizing One's Own
Incompetence Lead to Inflated Self-Assessments
The Dunning-Kruger Effect

11. 2. Wirth’s law
The rate of slowdown of software
is faster than the rate of speedup of hardware

12. 3. Hofstadter’s law
It always takes longer than you expect, even
when you take into account Hofstadter’s law

13. 4. Complexity
Complexity should be useful
-Larry Wall

14. 5. The End to End Principle
http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf
Reliability from unreliable parts

16. Stability vs Scale

17. Scale
Stability
Complexity

18. Pattern 0
Attitude towards failures

19. Everything
always works
I do my job.
Others need to do theirs
Everything is broken
My service works even
when my dependent
services are broken
Blind Faith
Trust but verify
Infrastructure
is broken
Software is broken
My service works
when there are
infrastructure
problems
My Service works even
when my hardware &
software dependencies are
broken
God’s country
Dunning-Kruger
Optimism bias
Circuit Breakers
Destructive Testing

20. Fallacies of Distributed
computing
The network is reliable
• Transport cost is zero
• It is easy to coordinate nodes in a network
• The network is homogenous
• There is one administrator
• Bandwidth is infinite
• Topology does not change
• The network is secure
• Latency is zero
Hofstradter’s law

21. Pattern 1
Scale out , Scale up

22. Scale out AND scale up
Wirth’s law

23. Pattern 1A:
Understand How to Scale

24. Horizontal duplication
Services
Sharding
(0,0,0)
Monolith
(1,0,0)
Read replicas
(0,0,1)
Sharding
(0,1,0)
Services
(1,1,1)
Infinite Scale
C Axis
Organizational
Scale
H
ow
to
Scale

25. Pattern 2
Everyone is Full Stack

26. The rule of least power
Atwoods Law
The less powerful the language, the
more you can do with the data stored
in that language
Occam’s Razor Simpler is better
Any application that can be written in
JavaScript, will eventually be written in
JavaScript.

27. 4 Tier Applications - add NodeJS
Clients
NodeJS
ClientsClients
Java
Data
Atwoods Law
UsefulComplexity
FullStack

28. FullStackDevOps - own it end to end
You built it, you run it - https://queue.acm.org/detail.cfm?id=1142065
Giving developers operational responsibilities has greatly enhanced the
quality of the services, both from a customer and a technology point of view.
The traditional model is that you take your software to the wall that separates
development and operations, and throw it over and then forget about it. Not
at Amazon. You build it, you run it. This brings developers into contact with
the day-to-day operation of their software. It also brings them into day-to-
day contact with the customer. This customer feedback loop is essential for
improving the quality of the service
– Werner Vogels

29. Pattern 3:
Performance is a feature

30. Pattern 3a:
Understand what/how to
measure

31. • Time to first byte
• Start render
• onLoad()
Perform
ance
What do you measure?

32. • Average
• Median
• TP
• RUM
Perform
anceHow do you measure?

33. Pattern 3b:
Design your edge architecture

34. Service 1
CDN/
Edge Server
Firewall Gateway
Facade
Service 2
Service 2
Cluster
Router
Perform
ance

35. Pattern 4
Search & Destroy Statefulness

36. State kills. Kill State
Session and state destroy the ultimate value promised by
multi-tenancy within Internet (SaaS, commerce, and so on)
applications.
- 50 Principles for Scaling Web Sites
Don’t simplify by creating unnecessary complexity
KillState

37. Pattern 5
Loose Couple to lose coupling

38. Pattern 5a
Service Decoupling

39. unnecessary complexity
TightC
oupling

40. FacadeFacadeCache
{
X-identity:
X-i18n:
X-role:
X-permission:
}HTTP
Headers
{
X-identity:
X-i18n:
X-role:
X-permission:
X-TX-Count:
}
1
2
3
Roles
Permissions
Identity
Loose
C
oupling

41. Pattern 5a
Bus & Event Sourcing
Loose
C
oupling

42. Async
The “Bus”
Highly
Reliable…?
Loose
C
oupling

43. The “Bus”
Unreliable,
high
throughput
recovery logic
End to End Principle
Loose
C
oupling

44. Unreliable,
high
throughput
recovery logic
Event Source
Message
Bus
ConsumersConsumersConsumers
Loose
C
oupling

45. Embrace failures
Embrace useful complexity
Understand Scaling patterns
Know how to measure
performance
Be full Stack
Destroy State
Use async and
event sourcing
Plan your edge architecture
culture
Technology