Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
# 
Frank Compagner 
Guerrilla Games
# 
How we learned to stop worrying and love 
the development cycle 
• Background 
• Iteration 
• Teamwork 
• Stability 
• ...
# 
• Killzone Shadow Fall – PS4 launch title 
• 200 local users, 150 remote 
• Head revision: 1.8 TB (1.6 M files) 
• Depo...
# 
Server 
Proxy Proxy Proxy Proxy 
Build Farm (60) 
Users (200) 
Proxy 
Cambridge 
Proxy 
Guilford 
Proxy 
San Diego 
Pro...
# 
Commit 
Server 
Edge 
Server 
Build Farm (60) Users (200) 
Cambridge 
Proxy 
Guilford 
Proxy 
San Diego 
Proxy 
Amsterd...
# 
Submits 
GB / week
# 
• Monitors all vital stats of server and proxies 
• Simple Python script, produces simple html 
• Works for Windows and...
#
#
# 
Code 
Compile 
Test Programmer 
Link
# 
Code 
Compile 
IDE and Tools 
Test Programmer 
Link 
Distribute 
Incremental linker 
Code 
Hotloading
# 
Model 
Artist 
Test Process 
Hotloading
# 
Model 
Tools 
Artist 
Test Process 
Hotloading Distribute
# 
Programmer 
Rigger 
Tester 
Team 
Designer Animator
# 
Programmer 
Rigger 
Animator 
Animator 
Tester 
Designer 
Modeler 
Programmer 
Programmer 
VFX artist 
Designer 
Audio ...
#
# 
• Does not look like this
# 
• It’s more like this
# 
• OK, like this
# 
• Personal branch is fine (sandbox / streams / Git) 
• Feature branches sometimes work 
• Give control over your enviro...
# 
• For us, branch switching is always expensive: 
– Amount of data and change 
– Unmergeable files 
– Code-Data dependen...
#
# 
Programmer 
Rigger 
Tester 
Team 
Designer Animator
# 
Programmer 
Submit Sync 
Rigger 
Tester 
Team 
Submit 
Submit 
Designer Animator 
Submit 
Submit 
Sync 
Sync 
Sync 
Syn...
# 
Programmer 
Test 
Submit Sync 
Rigger 
Tester 
Team 
Submit 
Submit 
Designer Animator 
Submit 
Submit 
Sync 
Sync 
Syn...
# 
• Test pre submit (user) 
• Test post submit (builder) 
• Test in parallel 
• Test as fast as possible 
• Still acciden...
# 
Build machines 
write 
Label 42: change 1200 
Label 41: change 1180 
Label 40: change 1155 
Label 39: change 1150 
… 
L...
Select 
tests 
Sync to 
get label 
# 
Labels that 
pass tests 
Labels that 
fail tests 
• Custom safe-sync tool shows labe...
#
# 
• We branch for every release (often every week) 
• Nobody works on the release branch 
• Everybody works on the trunk ...
• Cherry picking rules! 
• Gives you a lot of control 
• But also lots of confusion 
• And mistakes are easy 
• Good tools...
# 
• Iterate as fast as you can 
• Don’t branch too much 
• Test everything 
• Sync should be safe 
• Release all the time...
# 
Frank Compagner 
frank@guerrilla-games.com
# 
RESOURCES 
Perforce Workshop (P4 monitor, cache cleaner): 
https://swarm.workshop.perforce.com/files/guest/frank_compag...
Upcoming SlideShare
Loading in …5
×

Working Well Together: How to Keep High-end Game Development Teams Productive

5,327 views

Published on

During the production of PlayStation 4 launch title Killzone: Shadow Fall, Guerrilla Games struggled to finish in time as the size and scope of the game increased. Hear about the improvements they made to their build pipeline and walk away with key takeaways for making teams more productive by enabling collaboration and cooperation through good tools and processes, minimizing distance between developers, providing accurate and accessible information on the state of the project.

Published in: Technology
  • Login to see the comments

Working Well Together: How to Keep High-end Game Development Teams Productive

  1. 1. # Frank Compagner Guerrilla Games
  2. 2. # How we learned to stop worrying and love the development cycle • Background • Iteration • Teamwork • Stability • Releasing Iterate Launch
  3. 3. # • Killzone Shadow Fall – PS4 launch title • 200 local users, 150 remote • Head revision: 1.8 TB (1.6 M files) • Depot size: 8 TB • Peak change: 180 GB / week • 60 dedicated build servers • All in Perforce since 2004
  4. 4. # Server Proxy Proxy Proxy Proxy Build Farm (60) Users (200) Proxy Cambridge Proxy Guilford Proxy San Diego Proxy Amsterdam Shanghai
  5. 5. # Commit Server Edge Server Build Farm (60) Users (200) Cambridge Proxy Guilford Proxy San Diego Proxy Amsterdam Shanghai Edge Server Edge Server
  6. 6. # Submits GB / week
  7. 7. # • Monitors all vital stats of server and proxies • Simple Python script, produces simple html • Works for Windows and Linux servers • Proxies cleaned by another Python script • Both available in the Perforce Workshop
  8. 8. #
  9. 9. #
  10. 10. # Code Compile Test Programmer Link
  11. 11. # Code Compile IDE and Tools Test Programmer Link Distribute Incremental linker Code Hotloading
  12. 12. # Model Artist Test Process Hotloading
  13. 13. # Model Tools Artist Test Process Hotloading Distribute
  14. 14. # Programmer Rigger Tester Team Designer Animator
  15. 15. # Programmer Rigger Animator Animator Tester Designer Modeler Programmer Programmer VFX artist Designer Audio Engineer Modeler Designer Animator
  16. 16. #
  17. 17. # • Does not look like this
  18. 18. # • It’s more like this
  19. 19. # • OK, like this
  20. 20. # • Personal branch is fine (sandbox / streams / Git) • Feature branches sometimes work • Give control over your environment • But create distance to others • Team work requires frequent branch switching
  21. 21. # • For us, branch switching is always expensive: – Amount of data and change – Unmergeable files – Code-Data dependencies • Branches add complexity, we already have that • Team effort, we don’t want distance • Almost everybody works on the trunk • So why isn’t it always broken?
  22. 22. #
  23. 23. # Programmer Rigger Tester Team Designer Animator
  24. 24. # Programmer Submit Sync Rigger Tester Team Submit Submit Designer Animator Submit Submit Sync Sync Sync Sync
  25. 25. # Programmer Test Submit Sync Rigger Tester Team Submit Submit Designer Animator Submit Submit Sync Sync Sync Sync Test Test Test Test
  26. 26. # • Test pre submit (user) • Test post submit (builder) • Test in parallel • Test as fast as possible • Still accidents do happen User B Sync Sync Build Game Submit Sync Test Game Sync Build Tools Submit Sync Cook Data < 30 minutes User A Submit Sync Test Game … Sync Upload Cook Data Upload Test
  27. 27. # Build machines write Label 42: change 1200 Label 41: change 1180 Label 40: change 1155 Label 39: change 1150 … Labels.xml read Safe-Sync Tool • Each game build creates a label • Stored in Labels.xml file in depot • Builds add their result to the label • Builds can add artifacts to a label • Nobody gets latest (not safe) • Everybody uses safe-sync tool
  28. 28. Select tests Sync to get label # Labels that pass tests Labels that fail tests • Custom safe-sync tool shows labels and test results • Use it to sync to a known good label – This includes artifacts from other builds
  29. 29. #
  30. 30. # • We branch for every release (often every week) • Nobody works on the release branch • Everybody works on the trunk • Changes are merged up from main to release release main = release = change
  31. 31. • Cherry picking rules! • Gives you a lot of control • But also lots of confusion • And mistakes are easy • Good tools can fix this • Give flexibility • And control # (See whitepaper for details)
  32. 32. # • Iterate as fast as you can • Don’t branch too much • Test everything • Sync should be safe • Release all the time • Control what you release Speed Control
  33. 33. # Frank Compagner frank@guerrilla-games.com
  34. 34. # RESOURCES Perforce Workshop (P4 monitor, cache cleaner): https://swarm.workshop.perforce.com/files/guest/frank_compagner ReleaseManager: http://www.perforce.com/resources/presentations/user-conference-talks/ tools-for-smooth-release-cycle http://www.perforce.com/sites/default/files/guerrilla-games-whitepaper.pdf

×