Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Uber mobility - Build & Release

Speakers: Chirayu Krishnappa & Daniel Liem (Uber), Christian Legnitto (Pocketship), Jacek Suliga (LinkedIn), Robbert Van Ginkel & Gautam Korlam (Uber), Rachel Brindle (Pivotal Labs)

Tools for building and releasing mobile applications provided by Google and Apple are geared towards smaller mobile application development teams. However, once development teams start to grow, the requirement from your build and release tooling change drastically. This meetup was all about building & releasing mobile applications at scale: How does your build infrastructure need to evolve to support increasingly complex code-bases? How can you build an automated release pipeline that tracks every build and rolls out your mobile applications safely? What kind of unexpected problems can arise from adopting new technologies when you have a larger team?

Uber mobility - Build & Release

  1. 1. Daniel Liem & Chirayu Krishnappa Ship Fast & Stable @ Uber Scale 11.1.2016
  2. 2. 99.99%
  3. 3. ● Have a dedicated Release team ● Aggressive weekly release cadence ● Build cuts (CI) from Master ● Nightly (alpha) vs. Beta / Production builds ● Internal beta dogfooding vs. External Beta testing ● FF (Feature Flagging) wherever possible ● Avoid alphafixes, betafixes, hotfixes & rollbacks ● Soft Upgrades vs. Force Upgrades Our Process
  4. 4. Staged Rollouts iOS Android
  5. 5. Staged Rollouts iOS Sanity Tests / Patches Android Sanity Tests / Patches
  6. 6. Staged Rollouts iOS Sanity Tests / Patches Upload to iTunes, enable Testflight (Weekend dogfooding) Android Sanity Tests / Patches β channel (Weekend dogfooding)
  7. 7. Staged Rollouts iOS Sanity Tests / Patches Upload to iTunes, enable Testflight (Weekend dogfooding) Wait for Approval Android Sanity Tests / Patches β channel (Weekend dogfooding) Staged Rollouts (1% → 10% → 50% → 100%)
  8. 8. Staged Rollouts iOS Sanity Tests / Patches Upload to iTunes, enable Testflight (Weekend dogfooding) Launch 100% (All users at once!) Wait for Approval Android Sanity Tests / Patches β channel (Weekend dogfooding) Staged Rollouts (1% → 10% → 50% → 100%) (Partner app only) Soft Upgrade + Force Upgrade
  9. 9. How many apps?
  10. 10. How many teams? Developers!
  11. 11. How long does it take to “deploy to production”?
  12. 12. Build, sign, and more. Mostly deterministic.
  13. 13. Submit to App Store for approval
  14. 14. WAIT … !
  15. 15. Approved! Now what?
  16. 16. WAIT … ! Your users will upgrade…eventually.
  17. 17. http://www.publicdomainpictures.net/view-image.php?image=139317&picture=snail-man
  18. 18. SPEED Not the speed of your mobile app
  19. 19. SPEED Speed of deployment
  20. 20. SPEED Speed of reactions when you discover issues
  21. 21. SPEED Speed of rollbacks.
  22. 22. What are rollbacks?
  23. 23. How many versions of your app are out there?
  24. 24. Adoption across versions
  25. 25. Need for a trusted system
  26. 26. Goal: Develop at full speed!
  27. 27. High quality bar! Goal: Develop at full speed
  28. 28. High quality bar! Goal: Develop at full speed Because slow to deploy/rollback
  29. 29. Signals
  30. 30. Signals
  31. 31. Soon to be 100s of signals Signals
  32. 32. How can you stay on top? How do you react to these signals?
  33. 33. Ticketing system at the core If there’s a failure that should block the train, there’s a ticket for it.
  34. 34. Block specific versions Verify patched versions Track mitigation tasks Weekly reports
  35. 35. Intelligent Subsystems crash detection at alpha stages vs. production Every alpha crash gets a ticket
  36. 36. Feature Flagging Automate to track features per version and turn off based on classified crashes
  37. 37. Reminders E.g. When a new build is ready (also in-app “upgrade” notifications)
  38. 38. Notifications/Alerts We’re pushing at 4pm It’s 2pm and we have new / unresolved tickets.
  39. 39. Production alerts are separate e.g. Spike in crashes, E2E, App Store ratings dropped.
  40. 40. https://pixabay.com/en/go-button-3d-icon-sign-symbol-1067074/
  41. 41. Stage Everything Rollout to employees Test flight, alpha, beta channels
  42. 42. Stage Everything Alpha channel rollout? Stage it. 1%, 10%, 50% 100%
  43. 43. Stage Everything In app upgrade prompt? Stage that too! 1%, 10%, 100%.
  44. 44. In a nutshell… Deploys are slow Collect all signals Ticketing at core Automated reminders Ad hoc notifications / alerts
  45. 45. Thank you Proprietary and confidential © 2016 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber. Ship Fast & Stable @ Uber Scale Daniel Liem & Chirayu Krishnappa
  46. 46. Building At Uber Scale Like a ‘BAUS’ Robbert van Ginkel & Gautam Korlam 11.1.2016
  47. 47. Overview Challenges with mobile development at scale Team Size Build Time Infrastructure Improving Developer Experience while tackling scale Architecture Workflow
  48. 48. Mobile Scale @ Uber
  49. 49. Several Hundred Mobile Developers Hundreds of commits daily 50% of code changes every month Shared modular codebase with hundreds of modules
  50. 50. Team Size
  51. 51. Commits
  52. 52. Architecting for Scale Code architecture Features are built as Plugins and shared between apps Code infrastructure and tooling Monorepo helps with modularization and sharing Regressions block the whole team Always keep master green Guard as much as possible at compile time Fail fast
  53. 53. Workflow at Scale Asynchronous change merging Submit queue Stacked Diffs Run expensive code quality checks pre merge UI Tests Deep static analysis - Infer etc. Performance regressions on real devices - cold start, battery, network etc.
  54. 54. Build Time
  55. 55. Waiting for builds...
  56. 56. With more modules, come more problems
  57. 57. Build Tools Scaling Issues Cocoapods Does not scale well with more targets (15 min pod install time) Xcode Incorrect incremental builds (non deterministic and hard to debug) Xcode project file merge conflicts Gradle Does not scale well with large android projects (15 min for a single line change) Android Studio performance degrades
  58. 58. Building at Scale Both iOS and Android use Buck to build at Uber Incremental everywhere Scale non exponentially as more code is added Cache immutable state - avoid rebuilding Transparent Dependency Management Works well for monorepo iOS - Clean ~4x faster, Incremental ~20x faster Android - Clean ~6x faster, Incremental ~30x faster Remote Build Cache
  59. 59. Infrastructure
  60. 60. Uber’s CI Infrastructure CI capacity needs increased exponentially 400+ Busy Executors on CI hourly 50k+ CI Jobs run per day 600 400 200 100 CI Executors with Time
  61. 61. Optimizing the CI Pipeline Perform relevant checks at the right stage Code Formatting - pre diff Build, Unit Tests - diff UI, Static Analysis - pre merge Use CI resources effectively Remote build artifact caching Build in elasticity to meet peak demand
  62. 62. Open Source
  63. 63. Projects to Watch OkBuck - Gradle plugin that lets you use gradle projects with buck Buck Http Cache - A distributed build artifact cache service Coming soon: Swift Support in Buck https://github.com/uber/buck-http-cache 1https://github.com/uber/okbuck , Slide Deck
  64. 64. Takeaway Invest in the right build tools early on Scaling hardware only works till a certain point Having shared workflow/tools across platforms helps a lot in the long run Fail earlier and keep master always green
  65. 65. Scaling the Build Process at Uber Robbert van Ginkel & Gautam Korlam Thank You
  66. 66. In-product features for release engineers Christian Legnitto christian@pocketship.com
  67. 67. App ↑ ↑ ↑ http://www.pocketship.com
  68. 68. In-product features for release engineers
  69. 69. In-product
  70. 70. Release Engineers Product Engineers App
  71. 71. Empathy Agency
  72. 72. Agency Fix Release Engineering pain Gain more code context and confidence Improve Release Engineering processes Influence company direction
  73. 73. Empathy Build times and CI turnaround Tests, test infrastructure, and tooling Release process overhead Shipping to the world
  74. 74. Empathy “One of us”
  75. 75. Specific features
  76. 76. Specific features Company Product Release
  77. 77. Crash, OOM, hang reporting
  78. 78. Telemetry / analytics
  79. 79. Feature flags (A/B testing) if (whatever) { doA(); } else { doB(); }
  80. 80. Feature flags (A/B testing) if (whatever) { doA(); } else { doB(); }
  81. 81. Feature flags (A/B testing)
  82. 82. Bug reporter
  83. 83. Bug reporter
  84. 84. Bug reporter
  85. 85. Bug reporter
  86. 86. Bug reporter
  87. 87. Bug reporter
  88. 88. Promotion framework
  89. 89. Promotion framework
  90. 90. Promotion framework
  91. 91. Promotion framework
  92. 92. Promotion framework Test the next version of Facebook Get access to new features and bug fixes before everyone else.
  93. 93. Promotion framework
  94. 94. Promotion framework
  95. 95. Promotion framework
  96. 96. Automatic employee updater
  97. 97. React Native
  98. 98. React Native JS Bundle
  99. 99. React Native JS Bundle JS Bundle ↑
  100. 100. React Native
  101. 101. WebViews
  102. 102. WebViews
  103. 103. Version number scheme
  104. 104. Version number scheme
  105. 105. Version number scheme v20
  106. 106. Version number scheme v20 #500 1#500 2#500 3#500 4#500 5#500 6 … 14 APKs
  107. 107. Version number scheme v20 #500 1#500 2#500 3#500 4#500 5#500 6 … v21 #504 6#504 7#504 8#504 9#505 0#505 1 …
  108. 108. Version number scheme Ship production weekly Slow rollout Alpha and beta Internal employee dogfooding
  109. 109. Version number scheme
  110. 110. Version number scheme 100.0.0.20.70
  111. 111. Version number scheme 100.0.0.20.70 Major version
  112. 112. Version number scheme 100.0.0.20.70Major version
  113. 113. Version number scheme 100.0.0.20.70Major version Hotfix
  114. 114. Version number scheme 100.0.0.20.70Major version Hotfix
  115. 115. Version number scheme 100.0.0.20.70Major version Hotfix Beta
  116. 116. Version number scheme 100.0.0.20.70Major version Hotfix Beta
  117. 117. Version number scheme 100.0.0.20.70Major version Hotfix Alpha Beta
  118. 118. Version number scheme 100.0.0.20.70Major version Hotfix Beta Alpha
  119. 119. Release Engineers Product Engineers App
  120. 120. Release Engineers Product Engineers App
  121. 121. Christian Legnitto christian@pocketship.com
  122. 122. THE DARK SIDE OF ENTERPRISE SWIFT Jacek Suliga 🍺 Mobilize @ LinkedIn
  123. 123. Builds Perf & App Size Launch Performance Maintenance Cost
  124. 124. Builds Perf & App Size Launch Performance Maintenance Cost
  125. 125. 8-23 MB 10-30%
  126. 126. 20% slower
  127. 127. Builds Perf & App Size Launch Performance Maintenance Cost
  128. 128. WWDC 2016, 406, “Optimizing app startup time”
  129. 129. Builds Perf & App Size Launch Performance Maintenance Cost
  130. 130. 0. 5 1. 0 1. 1 1. 2 2. 0 2. 1 Open Sourc e 2. 2 2. 3 3. 0 6/14 9/14 10/14 4/15 6/15 9/15 12/15 3/16 6/16 9/16
  131. 131. LinkedIn Swift 3 migration party
  132. 132. 3. 0
  133. 133. 0. 3
  134. 134. Any Questions?!
  135. 135. Automating Mobile Releases Rachel Brindle
  136. 136. TERMS • User: Someone who uses your app. • Environment: Place where a user can download your app • Staging: HockeyApp, Testflight, etc. • Production: Enterprise MDM, App Store, Play Store • Automated Deployment: Using CI to push to environment z
  137. 137. Processz
  138. 138. WHY • Frees up the deployer to do other things • Consistent deploys • Shorter release cycle • Documents $ git push pushed g4edeff4 $ # wait $ rake check_if_deployed Latest is g4edeff4 z
  139. 139. • Make sure tests pass • Build for release • Gather metadata • Screenshots • Release Notes • Other? • Upload to environment
  140. 140. • Make sure tests pass • Build for release • Gather metadata • Screenshots • Release Notes • Other? • Upload to environment

×