This document discusses managing Jenkins jobs programmatically using Python and the Autojenkins library. It describes how to create template jobs that can be used to generate new jobs dynamically based on project metadata. New jobs are created by populating placeholders in the template with values from a project database, allowing jobs to be managed and updated automatically rather than through the Jenkins web UI.

1. Managing Jenkins with Python
Vijay Kumar B.<vijaykumar@bravegnu.org>

2. Continuous Integration
Every developer makes changes to the code and pushes them to the version control system.
The continuous integration server
pulls the code from repository
runs unit tests on the code and verifies its sanity.
Benefits
Accidentally missed out files are detected early.
Code is verified in a clean room environment.
Code can be tested in various target operating systems and configurations.
Build breakages and unit test breakages are immediately notified.

3. Continuous Integration Servers
Jenkins - popular continuous integration server written in Java.
Other open source continuous integration servers:
BuildBot
CruiseControl
Drone.io
GitLab CI

4. Setup @ Zilogic

5. Jenkins Jobs
A job specifies
where to checkout the source from
the commands to build the code
what to do with built artifacts.
Jobs are created through a web interface.

6. Fetching Source Code
Specify where to get the source from.
Specify whether to build periodically, or build when code is changed.

7. Build Steps and Post Build Actions
Specify what commands are to be executed to build and run unit tests.
Specify who should receive notifications, on build failure.

8. Job Representation
No text file representation that can be edited by hand.
Only a web interface to edit jobs.
Job representation is dumped to XML format, using persistence APIs.

9. config.xml
<project>
...
<scm class="hudson.scm.SubversionSCM" plugin="subversion@2.1">
<locations>
<hudson.scm.SubversionSCM_-ModuleLocation>
<remote>https://pike.zilogic.com/svn/proj/ansible-desktop
/trunk</remote>
...
</hudson.scm.SubversionSCM_-ModuleLocation>
</locations>
...
</scm>
...
<triggers>
<hudson.triggers.SCMTrigger>
<spec>H/15 * * * *</spec>
<ignorePostCommitHooks>false</ignorePostCommitHooks>
</hudson.triggers.SCMTrigger>
</triggers>

10. config.xml (Contd.)
...
<builders>
...
<hudson.tasks.Shell>
<command>export BUILD_VERSION=$BUILD_ID
make -f .jenkins.mk before_script
make -f .jenkins.mk script
</command>
</hudson.tasks.Shell>
...
</builders>
<publishers>
<hudson.tasks.Mailer plugin="mailer@1.10">
<recipients>vijaykumar</recipients>
...
</hudson.tasks.Mailer>
</publishers>
...
</project>

11. Problem
Managing Jenkins jobs through Web UI, does not scale.
Jobs are created by copying an existing job, and modifying it!
Violation of the DRY rule: Don’t Repeat Yourself!

12. Solution
Many people are trying to address the issue in various ways:
Job DSL plugin - https://github.com/jenkinsci/job-dsl-plugin/
Jenkins Job Builder - http://docs.openstack.org/infra/jenkins-job-builder/
Autojenkins - https://github.com/txels/autojenkins

13. Autojenkins
Jenkins provides REST API
Create jobs
Fetch config.xml of existing jobs
Update config.xml
Autojenkins
Python package that wraps over the REST API
Provides many useful funtions

14. Template Job
A template job can be created with place holders.
The job is disabled, which means Jenkins will not run it.
Autojenkins can create new jobs based on the template job, and values specified for the
placeholders
def create_copy(self, job, template, enable=True, _force=False,
**context):
"""
Create a job from a template job.
"""

15. Project DB
Description for each project for which CI is required.
name: ansible-desktop
template: python-template
summary: Ansible scripts for updating config. and installed
packages.
owner: vijaykumar
branches:
- 1.0
- 2.0
Used to populate the template, and create the jobs.

16. Per Project Build File
The chunk of what is different across projects in the build steps.
The build steps are put up in a Makefile called .build.mk within the project folder.
The template just invokes a make target in the .build.mk.
As a bonus, the CI build steps are under version control. This is similar to what Travis CI
does.

17. Script to Update Jobs
A class ProjInfo to hold the per project information.
name
summary
owner
template
branches

18. Script to Update Jobs (Contd.)
The following code creates a job for each project.
jenkins = Jenkins(self._config.jenkins_url,
auth=(self._username, self._password))
ci_jobname = "%s-ci" % self._proj_info.name
jenkins.create_copy(ci_jobname,
template_job=self._proj_info.template,
enable=True,
_force=True,
proj_summary=self._proj_info.summary,
proj_name=self._proj_info.name,
owner=self._proj_info.owner)

19. Finishing Notes
Job update script itself runs as a job under Jenkins!
Manage over 150 jobs at Zilogic.
Separate release and CI jobs, for each project.