It's hard to imagine a modern developer workflow without a sufficiently advanced build system: Make, Gradle, Maven, Rake, and many others. In this talk, we'll discuss the evolution of build systems that leads to distributed build systems. Then, we'll dive into how we can build a scalable system that is fast and resilient, with examples from Google. We'll conclude with the discussion of general challenges of migrating systems from one architecture to another.

1. Building a Distributed
Build System
at Scale
Aysylu Greenberg
September 17, 2016

2. Aysylu Greenberg
@aysylu22

3. Building
Distributed Build System
at Google Scale

4. Building
Distributed Build System
at Google Scale

5. What does a build system do?
Photo by smemon / CC BY

6. Build Scenario
Project with dependencies
Makefile
main.o : main.c defs.h
cc -c main.c
main.c :
curl –O http://remote/main.c

7. Build Scenario
Project with dependencies
Find dependencies
Makefile
main.o : main.c defs.h
cc -c main.c
main.c:
curl –O http://remote/main.c

8. Build Scenario
Project with dependencies
Find dependencies
Build project with the dependencies
Download build artifacts

9. Test Scenario
Project with dependencies
Find dependencies
Build project with the dependencies
Download build artifacts Run the test
Get the results of the test

10. Building
Distributed Build System
at Google Scale

11. Building
Distributed Build System
at Google Scale

12. Scale
• Engineers: >30,000 developers in 40+ offices

13. Scale
• Engineers: >30,000 developers in 40+ offices
• Commits: 15K by humans + 30K by robots/day

14. Scale
• Engineers: >30,000 developers in 40+ offices
• Commits: 15K by humans + 30K by robots/day
• Source code: 2 billion LOC

15. Scale
• Engineers: >30,000 developers in 40+ offices
• Commits: 15K by humans + 30K by robots/day
• Source code: 2 billion LOC
• Builds and tests: 5M per day through BuildRabbit

16. Scale
• Engineers: >30,000 developers in 40+ offices
• Commits: 15K by humans + 30K by robots/day
• Source code: 2 billion LOC
• Builds and tests: 5M per day through BuildRabbit
• Petabytes of output artifacts

17. Scale
• Engineers: >30,000 developers in 40+ offices
• Commits: 15K by humans + 30K by robots/day
• Source code: 2 billion LOC
• Builds and tests: 5M per day through BuildRabbit
• Petabytes of output artifacts
• 1 repository

18. Working in One Repository
Photo by flyingblogspot / CC BY SA

19. Repository of Artifacts vs Building from Source
Photo by flyingblogspot / CC BY SA

20. Working in One Repository
• Linear revision history

21. Working in One Repository
• Linear revision history
• Extensive code sharing & reuse
• Everything can be cross-referenced
• Large-scale refactoring for codebase modernization
• Easier collaboration across teams

22. Working in One Repository
• Linear revision history
• Extensive code sharing & reuse
• Simplified dependency management
• No confusion about authoritative version of the file
• No forking of shared libraries
• Components for library releases = Git subtree or Git
subcomponents to separate release from WIP versions

23. Working in One Repository
• Linear revision history
• Extensive code sharing & reuse
• Simplified dependency management
• Predictable, repeatable builds from source

24. Working in One Repository
• Linear revision history
• Extensive code sharing & reuse
• Simplified dependency management
• Predictable, repeatable builds from source
• Optimizations to avoid compiling same artifacts

25. Working in One Repository
• Linear revision history
• Extensive code sharing & reuse
• Simplified dependency management
• Predictable, repeatable builds from source
• Optimizations to avoid compiling same artifacts
• Decouple each team's processes as much as
possible

26. Building
Distributed Build System
at Google Scale

27. Building
Distributed Build System
at Google Scale

28. Build System Story
Photo by imanka / CC BY

29. Towards Distributed Build System

30. Towards Distributed Build System

31. Towards Distributed Build System

32. Towards Distributed Build System

33. Towards Distributed Build System

34. Towards Distributed Build System

35. BuildRabbit:
Distributed Build System

36. BuildRabbit: Distributed Build System
IN THE CLOUD

37. What does BuildRabbit do?

38. Continuous
integration
system
BuildRabbit
Source
System
Google
engineers &
teams
Release
infrastructure
Integration
testing
infrastructure
Build
artifact
storage
Blaze

39. Building
Distributed Build System
at Google Scale

40. Building
Distributed Build System
at Google Scale

41. From Push to Pull: Client / Server
Client for
User
BuildRabbit
Scheduler
BuildRabbit
Worker

42. From Push to Pull: Build Service
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
event
RPC
stream

43. From Push to Pull: Build Service
User
Persistent
Queue
BuildRabbit
Worker
Build Artifacts
Build Progress
Info
event
RPC
stream

44. Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker

45. Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Progress
InfoBuild Artifacts

46. Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Progress
InfoBuild Artifacts
RESILIENCE
LOGIC
RESILIENCE LOGIC

47. Continuous
integration
system
BuildRabbit
Source
System
Google
engineers &
teams
Release
infrastructure
Integration
testing
infrastructure
Build
artifact
storage
Blaze

48. Migration with Zero Downtime
Photo by moonjazz / CC BY SA

49. MIGRATE BACKENDS FIRST

50. Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Progress
InfoBuild Artifacts

51. FOCUS ON MIXED MODE
MIGRATE BACKENDS FIRST

52. TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
MIGRATE BACKENDS FIRST

53. Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Progress
InfoBuild Artifacts

54. DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
MIGRATE BACKENDS FIRST

55. Client
for
User
BuildRabbit
Scheduler
BuildRabbit
Worker
User
Persistent
Queue
BuildRabbit
Worker
Build Progress
InfoBuild Artifacts

56. MAXIMUM VISIBILITY INTO SYSTEM STATE
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
MIGRATE BACKENDS FIRST

57. PRACTICE THE LAUNCH
MAXIMUM VISIBILITY INTO SYSTEM STATE
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
MIGRATE BACKENDS FIRST

58. PRACTICE THE LAUNCH, A LOT
MAXIMUM VISIBILITY INTO SYSTEM STATE
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE
MIGRATE BACKENDS FIRST

59. HAVE A SOLID ROLLBACK PLAN
PRACTICE THE LAUNCH, A LOT
MAXIMUM VISIBILITY INTO SYSTEM STATE
DECOUPLE LAUNCH OF SERVICES
TARGET LAUNCH-FRIENDLY CLIENTS
FOCUS ON MIXED MODE

60. Distributed System Design Patterns
Photo by oslointhesummertime / CC BY

61. Distributed vs Centralized Control Plane
Photo by oslointhesummertime / CC BY

62. Distributed Design Patterns: Distributed Control
Design implementation is scaled from 1 machine
Bigger overhead during evolution

63. Distributed Design Patterns: Centralized Control
Distributed design from the start
Bigger overhead during design phase

64. Building a Distributed
Build System
at Scale
Aysylu Greenberg
@aysylu22