Building an application is a very intense and complicated process. Sometimes it could lead to unacceptable results when you can wait for temporary product eternity. Tools could be different, applications could be different, but techniques will be the same. Optimization is very important thing even when your process is standartized and strong. During that seesion I'll talk about: - Build is the most valuable product in DevOps - Tests, Sniffers, Performance tests and other things are minor in comparison to builds - How to get rid of long waits for small changes or fixes - How to don't waste time for waiting for build - How to incorporate measurement tools - How to solve feature branch hell and don't spent tons of time for merge conflicts - Make builds for enterprise and big data databases - Other interesting things from DevOps live :) Optimisation strategy shouldn’t be strict and shouldn’t ruin current process or block the team from performing operations. Given those answers, we can move forward like a thunder and achieve whatever we want.

1. Alexander Schedrov aka sanchiz
DrupalCamp Lviv, Sep 2016
Build your application in
seconds and optimize workflow
as much as you can using
DevOps

2. What is build?
Build is the process of integrating, building
and compiling the software that is produced.

3. – Кузьма Иванович, водовоз
“Без воды - и не туды, и не
сюды”

4. – Кузьма Иванович, водовоз
“Без воды - и не туды, и не
сюды”

5. – Щедров Александр, билдовоз :)
“Без билда - и не туда, и не
сюда”

6. How to accelerate
builds?

9. What is a big project?

10. Big data?

11. Big codebase?

12. Big team?

13. Big client?

14. Big money!

15. Big money!

16. What is the most valuable
thing for client in project?

17. Clean and structurized
code?

18. Custom modules?

19. Tests coverage?

20. Product, speed and
deliverability!

21. – Andrii Podanenko, FFW
“Success depends on how fast
you can show changes to client.”

22. Our project

23. In terms of build
• $$$ :)
• Huge database
• ~300mb *.sql.gz
• ~ 4gb *.sql
• D8 multisite - 3 sites
• Automated tests
• Code sniffers

24. In terms of workflow
• Builds queue
• ~50 per day
• Server space
• 80GB droplet
• ~10 active builds + 5 feature dev sites

25. Old build timeline

26. PR build timeline

27. Primary build timeline

28. + 2 subsites :(

29. Let The Machines Do
The Work

30. Lazy builders

31. Primary build and
secondary builds
1. Primary build
2. Other builds in parallel
• Slave sites
• Sniffers
• Tests

32. Lazy builder timeline

33. Post build actions

34. GitHub API for lazy builders
sudo jo state=pending target_url=http://URL/job/PR_BUILDER_TESTS/$
{BUILD_NUMBER}/console description="Tests run has been started."
context="CIBox tests" > pending.json
curl -u [bot_name]:[token] https://api.github.com/repos/[org]/
[repo]/statuses/${PR_SHA1} --request POST --data @pending.json
Pending status

36. GitHub API for lazy builders
# Run tests.
cd /var/www/build${PARENT_BUILD_NUMBER}
ansible-playbook tests.yml -i 'localhost,' --connection=local
# Set status and post comment to GitHub.
cd ${WORKSPACE}
sudo jo body="Behat test results file URL: http://URL/build$
{PARENT_BUILD_NUMBER}/build_reports/behat_report.html" >
comment.json
curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/
[repo]/issues/${PR_ID}/comments --request POST --data @comment.json
sudo jo state=success target_url=http://URL/job/PR_BUILDER_TESTS/$
{BUILD_NUMBER}/console description="Tests run has been finished."
context="CIBox tests" > success.json
curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/
[repo]/statuses/${PR_SHA1} --request POST --data @success.json
Tests + success status

37. https://wiki.jenkins-ci.org/display/JENKINS/Post+build+task

38. GitHub API for lazy builders
cd ${WORKSPACE}
sudo jo state=failure target_url=http://URL/job/PR_BUILDER_TESTS/$
{BUILD_NUMBER}/console description="Tests run has been failed."
context="CIBox tests" > failed.json
curl -u [bot_name]:[bot_token] https://api.github.com/repos/[org]/
[repo]/statuses/${PR_SHA1} --request POST --data @failed.json
Failed status

41. Build hierarchy
Primary Build
Slave 1 Slave 2 Sniffers Tests
Comment + status
on GitHub
Comment +
status
on GitHub
Comment +
status
on GitHub
Comment +
status
on GitHub
Comment +
status
on GitHub

43. Database optimisation

44. Optimize SQL import
SQL import is the most intensive operation
• Warm up databases for build?
• Bandwidth speed for downloading dumps
locally?
• Maintenance?

45. Virtualization with Docker

46. Create Docker Image
1. Run local Docker
repository(per project)
2. Pull base image cibox/mysql
3. Import database from
production dump
4. Create/Update image after
import
5. Push to local Docker
repository(per project)

47. Build pipeline
1.Pull changes from Docker Hub + Custom
Repository
2.Run Docker container
3.Specify in settings container’s IP
4.Container will be killed via cleaner

48. Benefits
1. Seconds to spin up container
2. Non-blocking operations for concurrent builds
3. Pull only changes(commits) to local environment
4. Update Docker Image daily

49. New build timeline

50. New build timeline

51. New Primary build timeline

52. Drush cr optimization

53. Why it faster now?
• ~2 seconds to spin up container
• ~2 seconds to clear Drupal 8 cache
• Non-Blocking operation in filesystem
• Isolated MySQL
• Run fast tests on build
• Run full tests on demo/dev site

54. Statistics
Start date

55. Numbers
• Primary build
• ~15 mins -> ~3 mins
• Overall time
• ~3 mins before testing of main product
• ~6-8 mins before testing of secondary products
• ~6-8 mins before sniffers & test results

56. Metrics - measure
your success

57. https://wiki.jenkins-ci.org/display/JENKINS/Timestamper

58. Time in comments

59. https://wiki.jenkins-ci.org/display/JENKINS/Global+Build
+Stats+Plugin

60. Remote access API
http://[JENKINS_URL]/job/[JOB_NAME]/api/json?tree=allBuilds[*]

61. DevOps life

62. – Chang Xiao, FFW
“I swear to god our vagrant is
like the nerdiest thing ever.”
“It should be like making a
ANSCII sandwich, brewing some
tea, waiting for your local
environment to be fully baked.”

64. Drupal.org: https://www.drupal.org/u/sanchiz
GitHub: https://github.com/Sanchiz
Blog: http://sanchiz.net
Email: alexander.schedrov@gmail.com
Twitter: @alexschedrov
Thank you!