Доклад об одном из самых больших монолитных инстансов Jenkins в мире: один мастер переваривает больше 100 тысяч билдов в день и управляет в пике ~2500 executor'ов. В докладе будут подняты следующие вопросы: * Как развернуть CI в облаке? * Как с помощью memcache экономить на железе для CI? * Тормозит ли Jenkins сам по себе? * Лайфхаки по использованию разных плагинов. * Нужно ли мониторить GC у Jenkins? * Какие плагины делают жизнь лучше, а какие - сильно хуже? * Как сделать мониторинг Jenkins'a ~из скотча и линейки~ с FluentD + InfluxDB + Grafana. И наверняка что-то еще...

1. Continuous Integration
on Steroids
Akbashev Alexander
Highload++ | November 07, 2016

3. Agenda
01. CI in HERE
02. Monitoring
03. Scalability
04. Jenkins
05. Nightmares Plugins
06. Morale
07. Q&A

4. 01
Continuous Integration in
HERE

5. Every change goes through validation pipeline
Gerrit
Gerrit
Plugin
Pre-submit
Trigger
Pre-submit
Trigger
Build
Build
Build
Build
Build
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests

6. Feedback goes from tests back to Gerrit
Gerrit
Gerrit
Plugin
Pre-submit
Trigger
Pre-submit
Trigger
Build
Build
Build
Build
Build
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests

7. Feedback comes from every pipeline
Gerrit
Gerrit
Plugin
Pre-submit
Trigger
Pre-submit
Trigger
Build
Build
Build
Build
Build
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests
Tests

8. Numbers
100k+ builds per day ~1.5k concurrent builds 1.3-2.5k executors
• Each “build” is
execution of one build/
test job
• Total number correlates
with number of commits
• Number of builds is not
so important as number
of commits
• Big throughput is
extremely important
• Morning commit
• Before lunch
• “Last attempt for today”
• Raised on-demand
• Health checks
• Jenkins strategy is not
optimized for cloud

9. 02
Monitoring

10. Collects information about every build in system
Groovy
Event
Listener
Plugin
Jenkins
build
Fluentd InfluxDB Grafana

11. Collects information about every build in system
Groovy
Event
Listener
Plugin
Jenkins
build Fluentd InfluxDB Grafana

12. JVM stats are the best “canary”
Groovy
Event
Listener
Plugin
Jenkins
build
Fluentd InfluxDB Grafana
Jenkins
JVM

14. 03
Scalability

15. What do we want to achieve?

16. What do we want to achieve?
Keep feedback time (< 20 min.)

17. What do we want to achieve?
Keep feedback time (< 20 min.)
Test as much as possible

18. What do we want to achieve?
Keep feedback time (< 20 min.)
Test as much as possible
… with debug symbols

19. What do we want to achieve?
Keep feedback time (< 20 min.)
Test as much as possible
… with debug symbols
… and code coverage information

20. What do we want to achieve?
Keep feedback time (< 20 min.)
Test as much as possible
… with debug symbols
… and code coverage information
and on physical devices

21. How to scale
Increase number of executors
Minimize job execution time
Smart testing

22. How to increase number of executors?
EC2 Plugin
TestDroid

23. How to minimize job execution time

24. How to minimize job execution time
Split tests by type

25. How to minimize job execution time
Split tests by type
Parallel execution

26. How to minimize job execution time
Split tests by type
Parallel execution
Node as cache storage

27. How to minimize job execution time
Split tests by type
Parallel execution
Node as cache storage
Shared compiler cache

28. How to minimize job execution time
Split tests by type
Parallel execution
Node as cache storage
Shared compiler cache
Profiling!

29. 04
Jenkins

30. Is Jenkins so slow or we are doing something wrong?

31. Is Jenkins so slow or we are doing something wrong?
Jenkins is ok.

32. Is Jenkins so slow or we are doing something wrong?
Jenkins is ok.
But…

33. Surprise #1
Rotation costs a lot

34. Surprise #2
It works much better with nginx
less jenkins.access.log | tail -n1000 | grep urt="-" | wc -l
407

35. Surprise #3
Some buttons are very dangerous

36. Surprise #3
Some buttons are very dangerous

37. Slave
Slave
One fundamental issue
Master
Slave
Slave
Slave
Slave
Slave
Slave
Users

38. What can you find in heap dump of OOM-Killed Jenkins?

39. What can you find in heap dump of OOM-Killed Jenkins?
Console logs

40. Console logs
Should be less than X MB
Verbose output goes to file
“>” and “tee” are amazing!

41. What can you find in heap dump of OOM-Killed Jenkins?
Console logs

42. What can you find in heap dump of OOM-Killed Jenkins?
Console logs
Build history

43. Build history
2000 entities or 3 days
Efficient rotator

44. What can you find in heap dump of OOM-Killed Jenkins?
Console logs
Build history

45. What can you find in heap dump of OOM-Killed Jenkins?
Console logs
Build history
Build artifacts

46. Build artifacts
Push to S3 directly from slaves
Don’t store anything on master

47. 05
Nightmares Jenkins Plugins

48. Limit of number of builds
20K

49. Groovy Event Listener Plugin
all events
synchronized
groovy compilation
fixed since 1.010 (Mar 10, 2016)

50. Limit of number of builds
40K

51. Warnings Plugin
Just another parser of console log
parseConsole is “deprecated”
parseFile is allowed
0 warnings are very appreciated :)

52. Limit of number of builds
60K

53. Timestamper Plugin
Tail needs not only “tail”
fixed since 1.8.5 (Aug 31, 2016)

54. Limit of number of builds
60K

55. EC2 Plugin
Full list of all images in AWS
fixed since 1.35 (Jun 30, 2016)

56. Limit of number of builds
90K

57. Robot Framework Plugin
Green chart costs 100 times more
Replaced by xUnit Plugin

58. Limit of number of builds
120K

59. Build Failure Analyzer Plugin
One regexp
One stream
One thread
PR-57 is not accepted yet

60. Limit of number of builds
140K

61. Cleanup Workspace Plugin
`ü` breaks everything
PR-29 is not accepted yet

62. 06
Morale

63. Final recommendations

64. Final recommendations
Think about scalability at first place

65. Final recommendations
Think about scalability at first place
Flakiness could be a huge problem

66. Final recommendations
Think about scalability at first place
Flakiness could be a huge problem
Reduce memory allocations

67. Final recommendations
Think about scalability at first place
Flakiness could be a huge problem
Reduce memory allocations
Cache as much as possible

68. Final recommendations
Think about scalability at first place
Flakiness could be a huge problem
Reduce memory allocations
Cache as much as possible
Failing builds can be expensive

69. Workflow
Slowness? Profile! Fix! Contribute!

70. Open source collaboration
Let’s make our life better ;)

71. Full list of our contributions related to this talk
• Jenkins
• ccache
• clcache
• EC2 Plugin
• S3 Plugin
• FluentD Plugin
• BuildRotator Plugin
• Groovy Event Listener Plugin
• Timestamper Plugin
• Robot Framework Plugin
• Build Failure Analyzer Plugin
• JVM GC Log Plugin for
FluentD

72. Thank you
Contact
Akbashev Alexander
GitHub: Jimilian
E-mail: alexander.akbashev@here.com