Currently I'm working on a flexible deployment pipeline in Jenkins where I can reuse some templates and create new pipelines within minutes. I'd like to present my solution, talk about the basics of Jenkins and explain why it's the better way to deploy with a CI server.

1. 0Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Jenkins Deployment Pipelines
» Introduction
» Jenkins
» Pipelines
» Hands on
» Improvements
» Downsides

2. 1Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Introduction

3. 2Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Me, myself and I
» Nicole Cordes
» working at CPS-IT GmbH in Berlin
» TYPO3 community member since 2011
» Core and Security Team member
» contributed some Symfony improvements for Windows in Finder
» some libraries and own projects based on Symfony
» Twitter: @IchHabRecht
» GitHub: https://github.com/IchHabRecht

4. 3Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
About this talk
» who or what is Jenkins
» understanding pipelines
» my way of pipeline integration into Jenkins
» useful Jenkins plugins to be more productive
» drawbacks of the solution

5. 4Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Jenkins

6. 5Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Jenkins
» automation (continuous integration) server
» open source project
» written in Java
» previously known as Hudson
» 2011 Oracle bought Sun and wanted to develop a commercial version
» the development was renamed to Jenkins and continued by the Hudson inventors

7. 6Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Jenkins
» easy to install / maintain
» able to run any kind of customized task
» you can define multiple triggers
» expansible by a huge amount of different plugins
» distribute work across multiple machines

8. 7Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Pipelines

9. 8Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Pipelines
» pipeline means multiple stages with different tasks to do
» build stage
» test stage
» deploy stage
» tool stage

10. 9Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Job pool
tool
SonarQube
phpDocumentor
test
PHPUnit
JUnit
Behat
Codeception
PHP Lint
build
composer
Node.js
asset management
deploy
rsync (any tool)
clear cache
warmup cache
database migrations

11. 10Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Understanding workspaces
» build and test stage share a workspace
» build stage ensures a proper state of your project
» test stage runs tests and ensures stability
» deployment has its own workspace
» you don’t want to deploy all dev tools
» tool stage usually uses deployment workspace as well but can have its own one

12. 11Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Build your project
test
PHPUnit
PHP Lint
build
composer
deploy
rsync (any tool)
» Project A (simple project)
test
PHPUnit
PHP Lint
» Project B (advanced project)
build
composer
Node.js
deploy
Deployer
clear cache
tool
SonarQube
phpDocumentor

13. 12Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Job pool
» each task is an own Jenkins job
» you can easily set up new projects from the job pool
» you can add new jobs to the job pool
» you can change one job and it is changed in every project

14. 13Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Hands on
(Project A)

15. 14Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Installing Plugins
» three required plugins to support minimal pipelines
» Git plugin
» Environment Injector Plugin
» Parameterized Trigger plugin
» Copy Artifact Plugin

16. 15Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the first job
» enter template name
» choose to build a “Freestyle project”
» allow template jobs to use artifacts
» choose “Prepare an environment for the run”
» enter some project server configuration
» those properties will be available as ENV vars

17. 16Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the first job
» define your “Source Code Management”
» choose “Git” and enter your information
» enable “Poll SCM” trigger without any schedule
» this allows you to call
http://jenkins/git/notifyCommit?url=https://github.com/IchHabRecht/some-project.git
to trigger your build from any other resource

18. 17Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the first job
» add a build step “Execute shell”
» create a tar file of the Git checkout
» add a post-build action “Archive the artifacts”
» enter your tar file name
» activate “Fingerprint all archived artifacts“

19. 18Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the template jobs
» create a new job “template-build-simple-project” as Freestyle project
» activate “Advanced Project Options“
» use a custom workspace
“${ROOT_JOB_NAME}_build/workspace“
» add a build step “Copy artifacts from another project”
» use project name “${ROOT_JOB_NAME}”
» use specific build “${ROOT_BUILD_NUMBER}”

20. 19Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the template jobs
» add a build step “Execute shell”
» untar the artifact and delete it
» now our build workspace is prepared to run single tasks

21. 20Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the template tasks
» create a new job “template-build-composer-install” as Freestyle project
» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”
» add a build step “Execute shell” with command `composer install`
» create a new job “template-test-phpunit” as Freestyle project
» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”
» add a build step “Execute shell” with command `bin/phpunit`
» depending on your framework you might need more configuration here

22. 21Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create the template tasks
» create a new job “template-test-php-lint” as Freestyle project
» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”
» add a build step “Execute shell” with command
for file in $(find “${WORKSPACE}/” -name *.php); do
php -d display_errors=stderr -l “$file” > /dev/null
done
» create a new job “template-deploy-rsync” as Freestyle project
» use custom workspace “${TEMPLATE_DEPLOY_WORKSPACE}”
» add a build step “Execute shell” with some rsync function
rsync -rvzle ssh -E --delete “${WORKSPACE}/”
“user@${SERVER}:${WEB_ROOT}/”

23. 22Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Create project type templates
» create a new job “template-test-simple-project” as Freestyle project
» no specific settings needed so far as tests using the build workspace
» create a new job “template-deploy-simple-project” as Freestyle project
» activate “Advanced Project Options“
» use a custom workspace
“${ROOT_JOB_NAME}_deploy/workspace“
» add a build step “Copy artifacts from another project”
» use project name “${ROOT_JOB_NAME}”
» use specific build “${ROOT_BUILD_NUMBER}”
» add a build step “Execute shell” and untar the artifact and delete it

24. 23Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Template view

25. 24Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Connecting the deploy jobs
» re-configure “template-deploy-simple-project”
» add a new build step “Trigger/call builds on other projects”
» projects to build “template-deploy-rsync”
» enable “Block until the triggered projects finish their builds”
» add parameters “Current build parameters”
» add parameters “Predefined parameters”
TEMPLATE_DEPLOY_WORKSPACE=${WORKSPACE}

26. 25Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Connecting the deploy jobs

27. 26Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Connecting the test jobs
» re-configure “template-test-simple-project”
» add a new build step “Trigger/call builds on other projects”
» projects to build “template-test-phpunit,template-test-php-lint”
» enable “Block until the triggered projects finish their builds”
» add parameters “Current build parameters”

28. 27Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Connecting the build jobs
» re-configure “template-build-simple-project”
» add a new build step “Trigger/call builds on other projects”
» make sure it is the last step in this job
» projects to build “template-build-composer-install”
» enable “Block until the triggered projects finish their builds”
» add parameters “Current build parameters”
» add parameters “Predefined parameters”
TEMPLATE_BUILD_WORKSPACE=${WORKSPACE}

29. 28Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Connecting the build jobs
» add another new build step “Trigger/call builds on other projects”
» make sure it is the last step in this job
» projects to build “template-test-simple-project”
» enable “Block until the triggered projects finish their builds”
» add parameters “Current build parameters”
» add parameters “Predefined parameters”
TEMPLATE_BUILD_WORKSPACE=${WORKSPACE}
» add a post-build action “Trigger parameterized build on other projects”
» projects to build “template-deploy-simple-project”
» enable “Block until the triggered projects finish their builds”
» add parameters “Current build parameters”

30. 29Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Adding the pipeline to Project A
» re-configure “Project A”
» add a post-build action “Trigger parameterized build on other projects”
» make sure it is the last step in this job
» projects to build “template-build-simple-project”
» enable “Block until the triggered projects finish their builds”
» add parameters “Current build parameters”
» add parameters “Predefined parameters”
ROOT_JOB_NAME=${JOB_NAME}
ROOT_BUILD_NUMBER=${BUILD_NUMBER}

31. 30Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Conclusions
» you can now build Project A which will trigger all other jobs
» currently only the checkout is deployed; you maybe have to add a
“template-deploy-composer-install” job running `composer install –no-dev`
» you can add a new “Project C” as another simple project
» create a new job and enter a name
» Mark “Copy existing Item” and choose “Project A”
» you just need to change the server variables and the git repository
» you are able to extend your job pool and define other project templates

32. 31Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Improvements

33. 32Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
SCM Sync Configuration Plugin
» store your Jenkins configuration in a Git repository
» comment your changes
» bulk edit jobs

34. 33Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Green Balls
» turn the blue status icons into green ones

35. 34Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Role-based Authorization Strategy
» manage and assign roles to users
» global roles like “Administrator” “Developer” “Guest”
» project roles based on project template names
» use client names as project prefixes

36. 35Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Workspace Cleanup Plugin
» delete the workspace before a new build is started
» useful for the “umbrella” template jobs

37. 36Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Email Extension Plugin
» versatile email notification plugin
» send emails to all commiters
» send emails to project members
» attach build logs

38. 37Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Further template improvements
» variable PHP version per project
» Node.js caching
» multiple server support for deployment
» multiple server types according to Git Flow branching model

39. 38Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Downsides

40. 39Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Missing features
» due to the post-build actions the root job never fails
» job pool tasks don’t save state per root template nor Git branch

41. 40Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines
Thank you
for your attention!