Keynote at DevOpsDays Cuba Successful Internet companies are built on a foundation of excellent culture, efficient organization, and solid technology. As a company needs to scale, all of these parts of the foundation need to grow and scale with it. This session covers modern best practices at innovative companies in Silicon Valley for scaling culture, organization, and technology. Driven primarily by the presenter's experience ranging from small Valley startups to Google and eBay, it discusses: * Organizing small, fast-moving engineering teams * Building a scalable system out of smaller microservices * Maintaining a culture of ownership and collaboration * Developing effective engineering processes of continuous integration and continuous delivery

1. Evolving Architecture and
Organization Together
Lessons from Google and eBay
Randy Shoup
@randyshoup
linkedin.com/in/randyshoup

2. Scalable Software
Development
Organization
TechnologyCulture
Process

3. Scalable Software
Development
Small
Teams
MicroservicesDevOps
Continuous
Delivery

4. Scalable Software
Development
Small
Teams
MicroservicesDevOps
Continuous
Delivery

5. Conway’s Law
• Organization determines architecture
o Design of a system will be a reflection of the communication paths within the
organization
• We can engineer the system we want by engineering the
organization

6. Small
“Service” Teams
• Amazon “2 Pizza” Teams
o No team should be larger than can be fed by 2 large pizzas
o Typically 4-6 people
• Aligned to Business Domains
o Clear, well-defined area of responsibility
o Single service or set of related services
o Minimal, well-defined “interface”
@randyshoup linkedin.com/in/randyshoup

7. Full-Stack
Teams
• All disciplines required for the team’s function
o Design
o Development
o Quality and Performance
o Maintenance
o Operations
• Depend on other teams for supporting services, libraries,
and tools
@randyshoup linkedin.com/in/randyshoup

8. “DevOps is a reorg”
-- Adrian Cockcroft, Netflix

9. Scalable Software
Development
Small
Teams
MicroservicesDevOps
Continuous
Delivery

10. “Tell us how you did
things at Google and
eBay.”
“Sure, I’ll tell you, but
you have to promise
not to do them!
[… yet]”

11. Architecture Evolution
• eBay
• 5th generation today
• Monolithic Perl  Monolithic C++  Java  microservices
• Twitter
• 3rd generation today
• Monolithic Rails  JS / Rails / Scala  microservices
• Amazon
• Nth generation today
• Monolithic Perl / C++  Java / Scala  microservices

12. No one starts with microservices
…
Past a certain scale, everyone
ends up with microservices

13. Getting to rearchitect a system
is a sign of success, not failure.

14. If you don’t end up regretting
your early technology
decisions, you probably over-
engineered.

15. Microservices
• Single-purpose
• Simple, well-defined interface
• Modular and independent
A
C D E
B

16. Microservices
• Single-purpose
• Simple, well-defined interface
• Modular and independent
• Isolated persistence (!)
A
C D E
B

17. Microservices
Each unit is simple
Independent scaling and
performance
Independent testing and
deployment
Can optimally tune performance
(caching, replication, etc.)
Pros
Many cooperating units
Many small repos
Requires more sophisticated tooling
and dependency management
Network latencies
Cons

18. Why
Rearchitect?
• Velocity
o Teams step on each others’ toes, and can no longer develop independently
o Difficult for new engineers to be productive
• Scaling
o Vertical scaling of the monolith no longer works
o Parts of the system need to scale independently of others

19. Why
Rearchitect?
• Deployment
o Parts of the system need to deploy independently of others
o Monolithic release is too slow, too complicated, too risky

20. “The only thing a Big Bang
migration guarantees is a big
*Bang*.”
-- Martin Fowler

21. Carving up the
Monolith
• Look for (or create) a “seam” in the
monolith
o This is often the hardest part (!)
• Wall it off behind an interface
• Write automated tests around the
interface
• Replace implementation with an
independent component
•  Rinse and Repeat

22. Scalable Software
Development
Small
Teams
MicroservicesDevOps
Continuous
Delivery

23. You Build It, You Run It.
-- Werner Vogels

24. Ownership
• End-to-end Ownership
o Team owns service from design to deployment to retirement
• Responsible for
o Features
o Quality
o Performance
o Operations
o Maintenance

25. Cross-Functional
Collaboration
• Open communication
o Individuals encouraged to work directly with each other
o Prefer informal cooperation over formal channels
• Best decisions made through partnership
o Agreement on goals and priorities makes it easier to agree on tactics
o Given common context, well-meaning people will generally agree
o “Disagree and Commit”

26. None of us is as smart as all of
us.
-- Japanese proverb,
as quoted by Bob Taylor

27. Scalable Software
Development
Small
Teams
MicroservicesDevOps
Continuous
Delivery

28. Continuous
Integration
• Small incremental changes
o Faster feedback loop
o Easy to understand, code-review, test, roll back
• Larger changes are much riskier
o Risk of code change is nonlinear in the size of the change
o Ex. Initial memcache service submission
• Main code branch always shippable
o Increased rate of change while reducing risk of change

29. Continuous
Delivery
• More solid systems
o Release smaller units of work
o Faster to repair, easier to diagnose
o Small change to roll back or roll forward
• Enables experimentation
o Small experiments and rapid iteration are cheap
• Enabled by
o PaaS, containers, cloud

30. Scalable Software
Development
Small
Teams
MicroservicesDevOps
Continuous
Delivery

31. ¡Muchas Gracias!
• @randyshoup
• linkedin.com/in/randyshoup