Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

This is a follow-up presentation to my talk at CloudBees | Jenkins Automotive and Embedded Day 2016, where I was presenting Pipeline usage strategies for use-cases in the Embedded area. In this presentation I talk about Jenkins Pipeline features for automation frameworks and talk about lessons learned in several project.

Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

  1. 1. Jenkins Pipeline @ Scale Building Automation Frameworks for Systems Integration CloudBees | Jenkins Automotive and Embedded Day Böblingen, April 05, 2017 Oleg Nenashev, CloudBees
  2. 2. © 2017 CloudBees, Inc. All Rights Reserved. 2 About me. Hardware/Embedded @oleg_nenashev oleg-nenashev LibreCoresSt. Petersburg Polytechnic University
  3. 3. © 2017 CloudBees, Inc. All Rights Reserved. 3 About me. Jenkins project @oleg_nenashev oleg-nenashev
  4. 4. © 2017 CloudBees, Inc. All Rights Reserved. 4 Oleg’s “Hall of Shame”(c) • Jenkins Core • Remoting • Plugins
  5. 5. © 2017 CloudBees, Inc. All Rights Reserved. 5 2016 http://bit.ly/nenashev-automotive-day-2016
  6. 6. © 2017 CloudBees, Inc. All Rights Reserved. 6 ØHow/why do we build frameworks for Embedded? ØWhat has changed in Pipeline since 2016? ØHow to effectively use and develop Pipeline? ØLessons learnt Agenda
  7. 7. © 2017 CloudBees, Inc. All Rights Reserved. 7 • Pipeline 101 • Pipeline Generic Features • Blue Ocean • Declarative Pipeline • Automation Infrastructure • Agents, Tools, Hardware Peripherals • Added-value Pipeline features in CloudBees Jenkins Enterprise What is NOT in the talk? Ask the Experts, Jenkins Online Meetup Birds of Feather, Ask the Experts
  8. 8. © 2017 CloudBees, Inc. All Rights Reserved. 8 What is a typical Embedded Product? ` Hardware External IPFirmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main product” Documentation
  9. 9. © 2017 CloudBees, Inc. All Rights Reserved. 9 Embedded area. Welcome to the automation hell! • Almost everything needs to be automated • Multiple technologies and tools • Integration testing is important • Unstable hardware & networks • Legacy stuff is everywhere • …
  10. 10. © 2017 CloudBees, Inc. All Rights Reserved. 10 OK, Automotive… Source: http://www.popsautoelectric.com/auto-electric/ • System of systems • High level of integration • Components by 3rd parties • Many standards • High quality requirements
  11. 11. © 2017 CloudBees, Inc. All Rights Reserved. 11 1. The most popular CI automation server in the world 2. It’s a framework => flexible • Swiss Army Knife of Automation 3. It’s open source => customizable 4. Commercial support is available 5. Useful plugins Why Jenkins?
  12. 12. © 2017 CloudBees, Inc. All Rights Reserved. 12 Система Docs Plugins Configs Infra- structure Samples Demos How do we build CI Systems? Infra- structure Infra- structure • Agents (FKA ”slaves”) • Tools • Hardware peripherals • Computing grids
  13. 13. © 2017 CloudBees, Inc. All Rights Reserved. 13 Approach: Using “classic” Job types: • Freestyle, Matrix, JobDSL, … Classic Approaches Keywords: • Script steps to run the logic • Custom Tools Plugin to manage tools • Throttle Concurrent Builds and Lockable Resources to dispatch access to hardware • Naginator Plugin to restart builds on-demand • Parameterized Trigger / Copy artifacts to parallelize tests and to collect results JUC2015/London https://jenkins.io/solutions/embedded/
  14. 14. © 2017 CloudBees, Inc. All Rights Reserved. 14 Classic Approaches Spaghetti Automation • Barely traceable dependencies • Complex build flow • Complex configurations • Duplication • Expensive maintenance
  15. 15. © 2017 CloudBees, Inc. All Rights Reserved. 15 ü Store build definitions with the project ü Track changes together with the project ü Tests together with the project Configuration as Code
  16. 16. © 2017 CloudBees, Inc. All Rights Reserved. 16 Configuration as Code in Jenkins Job ConfigurationJenkins system configuration
  17. 17. © 2017 CloudBees, Inc. All Rights Reserved. 17 Jenkins System Configuration… as Code External Tools Jenkins CLI and REST API python- jenkins jenkins-client (java) Configuration Management systems Ansible, Chef Docker Solutions within Jenkins project Groovy Boot Hooks Scriptler Plugin SCM Sync Configuration Unstable Just examples…
  18. 18. © 2017 CloudBees, Inc. All Rights Reserved. 18 Configuration as Code for jobs Job DSL Job Builder Plugin Jenkins Pipeline Pipeline is offered by default in Jenkins 2.0 … Groovy YAML Groovy
  19. 19. © 2017 CloudBees, Inc. All Rights Reserved. 19 Pipeline is a… Ø Groovy DSL for automatic flow description * • Alternatives – Job DSL, Jenkins Job Builder Ø Job type in Jenkins (Pipeline Job) • Job definitions can be stored in SCM • Automatic job creation from repositories and organizations Ø Ecosystem for automation as code • ~20 dedicated plugins • Integration in many other plugins • Suggested by default in Jenkins 2 // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial 19
  20. 20. © 2017 CloudBees, Inc. All Rights Reserved. 20 Why Jenkins Pipeline?
  21. 21. © 2017 CloudBees, Inc. All Rights Reserved. 21 Why Jenkins Pipeline? • Robustness - master restarts & network • Parallelization across multiple nodes: • parallel() • node() • Ability to store the shared logic in standalone Pipeline libraries • Security • Two modes: Scripted and Declarative • Visualization: Stage View, Blue Ocean
  22. 22. © 2017 CloudBees, Inc. All Rights Reserved. 22 Система Docs Plugins Configs Infra- structure Samples Demos Automation Framework Vision Infra- structure Infra- structure SCM Infrastructure as Code Configuration as Code Documentation as Code Pipeline Libraries Pipelines
  23. 23. © 2017 CloudBees, Inc. All Rights Reserved. 23 Developer Experience • Encapsulation • Simple UI • Most of the code is in GitHub • No need in classic UI for 90% of users • Notifications to email/GitHub
  24. 24. Jenkins Pipeline in Embedded 2016
  25. 25. © 2017 CloudBees, Inc. All Rights Reserved. 25 2016 http://bit.ly/nenashev-automotive-day-2016
  26. 26. © 2017 CloudBees, Inc. All Rights Reserved. 26 2016 – Summary • Jenkins can be used to run tests on hardware in embedded and automotive projects • Jenkins Pipeline is helpful for test runs • Pipeline lacks some critical features
  27. 27. © 2017 CloudBees, Inc. All Rights Reserved. 27 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016
  28. 28. © 2017 CloudBees, Inc. All Rights Reserved. 28 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016
  29. 29. © 2017 CloudBees, Inc. All Rights Reserved. 29 Management of resources • Lockable Resources Plugin – since 2016 • Throttle Concurrent Builds – coming soon • Pipeline Milestones
  30. 30. © 2017 CloudBees, Inc. All Rights Reserved. 30 PARTIAL - Better support of tools • withEnv() for environment variables • tool() step in Pipeline was improved • Tool reference in the Job • Section in Declarative Pipeline • Custom tools is not fully integrated • Follow JENKINS-30680 • ETA – Q2, 2017
  31. 31. © 2017 CloudBees, Inc. All Rights Reserved. 31 DONE – Pipeline Libs (Pipeline Package Manager) • Early 2016 • Global Pipeline Library • Pipeline Remote Loader Plugin • Late 2016 • Pipeline Shared Libraries • Support of @Grab – Pipeline libs can pull common Groovy libs Library Structure:
  32. 32. © 2017 CloudBees, Inc. All Rights Reserved. 32 • Pipeline Shared Libraries • Libs are located in SCM • Versioning • Definition for scopes: ▸ Jenkins system-wide ▸ Folder ▸ Job
  33. 33. © 2017 CloudBees, Inc. All Rights Reserved. 33 Pipeline Library Definition [1/2]
  34. 34. © 2017 CloudBees, Inc. All Rights Reserved. 34 Pipeline Library Definition [2/2]
  35. 35. © 2017 CloudBees, Inc. All Rights Reserved. 35 Using Pipeline Libraries • Implicit Loading • Via Jenkins Global or Folder configuration • Explicit Loading • @Library
  36. 36. © 2017 CloudBees, Inc. All Rights Reserved. 36 Using Pipeline Libraries com.mycorp.libs.make Jenkinsfile makeProject() makeProject() implementation - global variable com.mycorp.libs.coverity com.mycorp.libs.cppunit Globally-defined library, automatic update @Library() definition
  37. 37. © 2017 CloudBees, Inc. All Rights Reserved. 37 Early adoption of any framework http://devopsreactions.tumblr.c om/post/150674592321/pettin g-your-incomprehensible- piece-of-code • It is working! • In several pet projects…
  38. 38. Lessons Learnt. What does NOT work well?
  39. 39. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  40. 40. © 2017 CloudBees, Inc. All Rights Reserved. 40 Pipeline Developer Experience • No Debugger • No IDE • No built-in test framework • Limited static analysis features •Pipeline DSL != Groovy
  41. 41. © 2017 CloudBees, Inc. All Rights Reserved. 41 How do I work? Intellij IDEA Filesystem SCM Plugin (with patch) Disclaimer: There may be better ways • Syntax file • Built-in documentation Jenkins Test Instance Source Code (local .git repos) • Production System Snapshot • Custom Pipeline Library Settings • Repositories for libraries • Jenkinsfile repos
  42. 42. © 2017 CloudBees, Inc. All Rights Reserved. 42 Getting the Syntax File
  43. 43. © 2017 CloudBees, Inc. All Rights Reserved. 43 Filesystem SCM Plugin • Pending pull request: • https://github.com/jenkinsci/filesystem_scm-plugin/pull/2 Library definition in the test instance !!!
  44. 44. © 2017 CloudBees, Inc. All Rights Reserved. 44 Pipeline Testing • No standard test framework offered by the Jenkins project • External project: https://github.com/lesfurets/JenkinsPipelineUnit “mvn verify” has been invoked… And the execution passed… For the source Jenkinsfile…
  45. 45. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  46. 46. © 2017 CloudBees, Inc. All Rights Reserved. 46 Error Handling • IMHO: Pipeline is designed for Success Path scenarios When it “Just works” When it does not
  47. 47. © 2017 CloudBees, Inc. All Rights Reserved. 47 Error Handling. Top-level Scripted Pipeline Declarative Pipeline • Global try/catch() When it does not…
  48. 48. © 2017 CloudBees, Inc. All Rights Reserved. 48 Error Handling When it “Just works” When it does not The interesting part… Failover, repeat
  49. 49. © 2017 CloudBees, Inc. All Rights Reserved. 49 Error Handling. Hardware failover • Pipeline allows re-running unreliable stages on other nodes. System failure (script returns 253) Node #1 Node #2. . . . . . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 253")) { // Fatal error fail("Test run failed") } } } Try all possible boards:
  50. 50. © 2017 CloudBees, Inc. All Rights Reserved. 50 Error handling. Hardware failover for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } ??? • Hard to extract error cause • Special handling is required for every case • Cannot just use existing parsers • Build Failure Analyzer • Log Parser • Warnings • Try-catch blocks pollute the code • Hard to test
  51. 51. © 2017 CloudBees, Inc. All Rights Reserved. 51 Context retention • No way to propagate any data • Solution - @NonCPS steps within the library steps
  52. 52. © 2017 CloudBees, Inc. All Rights Reserved. 52 Error handling. Hardware failover Declarative Pipeline Encapsulation via Libraries Simple use-case Other • Exception try/catch • timeout() wrapper • Proper failover • Logic in @NonCPS
  53. 53. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  54. 54. © 2017 CloudBees, Inc. All Rights Reserved. 54 Declarative Pipeline • It is great for simple use-cases • Publishers & Notifiers • We do not need a framework for simple use-cases • Lack of features • “How do I pass option for the Docker Agent” • “How do I declare a resource requirement for Throttling?” • … • Scripted Pipeline is a recommended fallback, but there is no automatic migration
  55. 55. © 2017 CloudBees, Inc. All Rights Reserved. 55 Declarative Pipeline. Recommendations • Use Declarative Pipeline with caution • Complex agent definitions – use Scripted Pipeline • Complex throttling – use Scripted Pipeline • Post-build steps – work well • Tool definitions – work well • …
  56. 56. Lessons Learnt. What does NOT work well? • Development Tools • Non-Success Path handling • Declarative Pipeline • Manual release management
  57. 57. © 2017 CloudBees, Inc. All Rights Reserved. 57 Pipeline vs. Application Lifecycle
  58. 58. © 2017 CloudBees, Inc. All Rights Reserved. 58 Pipeline vs. Application Lifecycle BUT: • Non-CD process with manual promotion • Dependent products • Snowflake development stages
  59. 59. © 2017 CloudBees, Inc. All Rights Reserved. 59 Solutions • Input() step + milestone() for synchronization • Multiple Pipelines • Release() step is coming soon • Release Plugin Pipeline 1 Pipeline 2 Run Selector Staging Artifact Storage External ALM tool CD for the Component 1 CI/CD for the Component 2
  60. 60. Lessons learnt • No area-specific issues so far • High barrier to entry • It worth investment
  61. 61. © 2017 CloudBees, Inc. All Rights Reserved. 61 What I need? 2017 • Development Tools for Pipeline • IDE, Debugger, Test Tools • Community Pipeline Libraries • Common Embedded cases without plugins • Promotion engine for Pipeline • Ideally: Merge of standard Promoted Builds and Release plugins • Plugins for area-specific tools
  62. 62. © 2017 CloudBees, Inc. All Rights Reserved. 62 Takeaways • Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances • Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases • Barrier to entry is still high
  63. 63. © 2017 CloudBees, Inc. All Rights Reserved. 63 More info • Embedded solutions page: • https://jenkins.io/solutions/embedded/ • Links to the articles and videos • Some categorization Embedded Solutions Page …
  64. 64. © 2017 CloudBees, Inc. All Rights Reserved. 64 • Jenkins project: • Website: https://jenkins.io • Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Pipeline: • Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md • Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md • Examples: https://github.com/jenkinsci/pipeline-examples • CloudBees website: • Website: https://www.cloudbees.com • CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform Useful Links
  65. 65. © 2017 CloudBees, Inc. All Rights Reserved. 65 Thank you! Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev
  66. 66. Software at the speed of ideas THANK YOU! www.cloudbees.com

×