Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pipeline 101 Lorelei Mccollum

Boston Jenkins Meetup Pipeline 101 Talk

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Pipeline 101 Lorelei Mccollum

  1. 1. Pipeline 101 Lorelei McCollum
  2. 2. Agenda • What is Jenkins Pipeline? • Getting Started • Using SCM with Pipeline • Pipeline Examples & Best Practices • Shared Libraries • Vars & Steps & Helper Functions • Developing and Testing Pipeline scripts • Advanced Reporting • Q & A
  3. 3. What is Jenkins Pipeline? • A new way to design/create/write Jenkins Jobs • No more Job Config UI • All Code • Replaces Job Config UI Build steps with code • Uses a Groovy CPS library to compile the code • Allows to save states to disk as the program runs to survive restarts • Groovy is the syntax of the pipeline script • Not efficient due to the nature of the CPS compiler • Doesn’t support all fancy syntax operations of the groovy language
  4. 4. Example of a FreeStyle Job: • Copies Artifacts from another Job • Executes some Shell Command • Injects some properties from Shell step into the Build • Triggers another job • Conditional statements as well • Maybe a check out & Build These Build Steps are how you design/configure your Jenkins Jobs
  5. 5. Creating a Pipeline Job Job Config Just no longer has all those build steps you can add, just a place for your code
  6. 6. Source Control for Pipeline • Pipeline scripts can be inline • Best practice is to put them in source control, also forces code review…. • Can be in source control with the product the job is for • Can be in separate source control project for just Pipeline code • This is the option we took, our shared library, external resource files and the pipeline scripts themselves are all in one git repo • Source control forces the pipeline scripts to be executed in Groovy Sandbox
  7. 7. What is a groovy sandbox? • All over the Jenkins interface • Means this groovy pipeline script runs in the master Jenkins JVM space without restriction, and has access to everything • Recommended to always use a groovy sandbox • Rogue scripts can and will take down your Jenkins master • Some methods may require Admin Approval • Admin Approval can also tell you if the function an engineer wants to use is dangerous and you can deny it • Admin Approvals can be a pain in current design…. But worth it!
  8. 8. Admin approvals we allowed • method groovy.lang.Script println java.lang.Object • method hudson.model.Action getIconFileName • method hudson.model.Actionable getAction java.lang.Class • method hudson.model.Actionable getActions • method hudson.model.Actionable getActions java.lang.Class • method hudson.model.Run getDescription • method hudson.model.Run setDescription java.lang.String • method hudson.tasks.junit.CaseResult getClassName • method hudson.tasks.junit.CaseResult getSuiteResult • method hudson.tasks.junit.SuiteResult getFile • method hudson.tasks.junit.TestObject getFailCount • method hudson.tasks.junit.TestObject getSkipCount • method hudson.tasks.junit.TestObject getTotalCount • method hudson.tasks.test.AbstractTestResultAction getFailCount • method hudson.tasks.test.AbstractTestResultAction getResult • method hudson.tasks.test.AbstractTestResultAction getTotalCount • method hudson.tasks.test.TestResult getFailedTests • method hudson.tasks.test.TestResult getSkippedTests • method java.text.Format format java.lang.Object • method java.text.NumberFormat setMaximumFractionDigits int • method java.time.Duration toMinutes • method java.util.Collection remove java.lang.Object • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder$BadgeManager createSummary java.lang.String • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder$BadgeManager getBuild • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder$BadgeManager getEnvVars • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildSummaryAction appendText java.lang.String boolean • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildSummaryAction appendText java.lang.String boolean boolean boolean java.lang.String • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildSummaryAction getIconPath • method org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildSummaryAction getText • new java.util.ArrayList • new java.util.TreeMap • staticMethod java.lang.Boolean parseBoolean java.lang.String • staticMethod java.lang.Boolean valueOf boolean • staticMethod java.lang.Math min int int • staticMethod java.lang.String valueOf int • staticMethod java.text.NumberFormat getInstance • staticMethod java.time.Duration between java.time.temporal.Temporal java.time.temporal.Temporal • staticMethod java.time.Instant now • staticMethod java.time.LocalDateTime now
  9. 9. Examples of how to write Pipeline scripts • Copy Artifacts • Execute Shell • Conditional • Archive Artifacts • File I/O • Test Results • Trigger Other Jobs
  10. 10. Basics of Pipeline Scripts: • Node Blocks • Determine where to run this part of the job… what Jenkins Agent/Node to use an executor on • Stage Blocks • Organize segments of your Pipeline script • Allows for easy code readability and for the Dashboard of the Job when the job is running • Can easily see what part of the Pipeline script is being executed • Checkpoint Statements • Almost like “saving your work”, identifies a good place in the script to save, so that if you wanted to restart the pipeline at a later time you could resume from this point
  11. 11. **sudo code example
  12. 12. Other Pipeline Examples CheckOuts Parallel Blocks **sudo code example
  13. 13. Error Handling in Pipeline Scripts • error(“Job Failed”) • Stops your pipeline script from executing as if it was aborted • Try/Catch Blocks • Can handle errors and report yourself • Allows you to handle if someone “Aborts the Build” and gracefully clean up and report • Notify people • Email, slack all supported in pipeline • Groovy Postbuild plugin • Gives you access to manager.buildFailure() – marks the build red • Gives you access to manager.buildUnstable() – marks the build yellow • Allows you to control the result
  14. 14. Snippet Generator For Plugins Installed: • Helps you understand and learn the syntax to invoke them in your Pipeline • You provide how you would invoke via Job Config UI (Old way to define Jenkins Jobs) and it shows you the pipeline equivalent!
  15. 15. Global Variables available to your scripts These are recognized in your scripts and have methods/variables available to you • env • env.NODE_NAME • env.WORKSPACE • env.BUILD_URL • env.JOB_URL • params • This builds parameters if it’s a Parameterized build • currentBuild • Next slide shows examples • scm • manager • Groovy PostBuild Plugin available to Pipeline Scripts • Shared Library Vars • Ones you create, will discuss later when we look at Shared Libraries • Other Plugins you may have installed that contribute here
  16. 16. One pipeline script for multiple jobs? • Found I had a lot of jobs that were similar • As I developed the pipeline scripts, there was a lot in common • Too much in common, where shared libraries didn’t really apply, as it would be a entire function of the job • Prepared environment = perfect solution
  17. 17. How to leverage the Prepared Environment? • The items in prepared env come in in the env. Variable • Can reference them and create what you need • Then all your jobs can all reference the same pipeline script • Makes easy to maintain and modify
  18. 18. Shared Libraries • Used to store helper functions that may be common or used across pipeline scripts • Used to store global variables used throughout your pipeline projects • Used to create your own “steps” for pipeline scripts to leverage • Helper functions go in the /src folder and include: package com.ibm.shared; • Steps and Variables for directly pipeline access go in the /vars folder • Import and allow shared library access in a pipeline script with this at the top • Access these shared libraries by instantiating the class
  19. 19. /vars/jobinfo.groovy • Note: Lowercase name of the groovy file • In pipeline scripts you can access these things by: • jobinfo.PIPELINE_CONFIG.jobURL • JenkinsJobRef is a class in our shared library that has jobURL and remotePath instance variables to allow access to these items • We wanted once place to reference these hardcoded strings in the event they were to change
  20. 20. /vars/nodescript.groovy • We step out and execute JavaScript resource files with Node • Created a “Step” helper to be used in pipeline • Call from pipline within a nodescript {} block • Where you can set variables to determine what to do
  21. 21. /vars/nodescript.groovy • Allows the pipeline script to overwrite variables if they set them in their block • Checks out the resources project • Copies down the javascript files • Executes them and returns the result
  22. 22. Pipeline Tips • Pipeline is not efficient • Use Strict typing even though Groovy doesn’t necessarily require it • Not meant to do heavy parsing, or to hold complex logic • You may be tempted to write it this way because the groovy programming language allows it • Declarative Pipeline • New type of pipeline script that helps limit this • The pipeline script is running in the master space, node blocks step out the “steps” to your Jenkins Agent executor, but the main pipeline script uses your masters CPU/memory • Heavy logic can impact master to recommended to step it out to other processes, shell, node, external groovy scripts • You will notice for loops, if blocks in Pipeline are not efficient and can take a long time to execute because of the CPS compiler • If you must have these, you could use @NonCPS blocks around those functions, and it will compile the functions normally • CODE REVIEW!!!!
  23. 23. What doesn’t work? • Nested Stages/Parallel blocks of stages • Blue Ocean I think is looking to solve this • Fancy Groovy programming • Stick to the basics, go simple • Shared Libraries cannot use Pipeline steps • If you see a crazy exception and stack, likely you have something not supported in your script • Google is useful in this case, but carefully go through your pipeline script • Pipeline scripts don’t automatically && its result with results of things you do, unless a fatal error, you need to control the result
  24. 24. Test and Develop Pipeline Scripts • Created a JobsUnderDevelopment Folder • Engineers have their own sub folder within that space • Keeps all Dev Jobs separate • Engineers define their Folder Config to point to the SCM branch to load the Shared Libraries • Then any jobs executing in that Folder will use the dev stream of the shared libraries • Pipeline scripts of production jobs can be copied into these folders as well and point to the SCM Branch to load the development pipeline code • Allows multiple engineers to work and test off their SCM branch before delivering pipeline script changes
  25. 25. Advanced Reporting • Customize your build pages of the jobs • Does require Admin Approval of various functions to do this • We modify the build description first of our jobs when they run
  26. 26. • Shared Library Helper function • setDescriptionWithTestResult • Lots of my pipelines use this so made a shared library function • Call it from pipeline script • Returns a String of what summary • Can be used in notification email
  27. 27. Example from Pipeline script, that loads the shared library
  28. 28. Remove items from build page
  29. 29. Sort Items on the build page Our parallel blocks write out what is going on So you know where it is throughout the build But at end of job we want to clean that up
  30. 30. Q & A Helpful websites: • https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md • https://github.com/jenkinsci/workflow-cps-plugin/blob/master/README.md • https://jenkins.io/doc/book/pipeline/ • https://jenkins.io/solutions/pipeline/ • https://go.cloudbees.com/docs/cloudbees-documentation/cookbook/book.html#_continuous_delivery_with_jenkins_pipeline • https://www.cloudbees.com/blog/top-10-best-practices-jenkins-pipeline-plugin • https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md • https://jenkins.io/blog/2017/02/01/pipeline-scalability-best-practice/ • https://jenkins.io/doc/pipeline/examples/ Helpful Plugins • https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Utility+Steps+Plugin • https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Postbuild+Plugin

×