The Butler is still young – applying modern Jenkins features to the Embedded area

PhD, Hardware Engineering
Background: R&D in Hardware & Embedded
• Intel Labs, Sitronics, Synopsys
• Large-scale automation infrastructures
Jenkins Developer, CloudBees
• CloudBees Jenkins Platform
• Customer support and consulting
Core Team Member, Jenkins Project
• Maintainer of plugins
• Meetup organizer, GSoC Org Admin
Contributor, LibreCores
About me
@oleg_nenashev
oleg-nenashev
2

The Butler is still young –
applying modern Jenkins
features to the Embedded area
Oleg Nenashev
CloudBees, Inc.

Do know about Jenkins?
About you

1. The most popular CI automation
server in the world
2. It’s a framework
• Swiss Army Knife of Automation
3. It’s open-source
• Can be customized
4. Commercial support is available
Who is Mr. Jenkins?

© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 6
Computing
grids
Docs
HW
peripherals
Plugins Configs
Build
agents
Jobs
Jenkins-based
Automation System

Do you use Jenkins?
About you

Jenkins is Popular
http://stats.jenkins-ci.org/jenkins-stats/
• Common: ~70% of teams
use Jenkins
• Almost 150k installations
• ~300 active committers
(over last 6 months)
• ~3 new plugins every week
• ~50 plugin releases every
week
Hudson Jenkins

Many Embedded companies use Jenkins

LTS release lines
Backward compatibility
•Jenkins 1.x was compatible for 8+ years
•Jenkins 2.x is almost compatible with
Jenkins 1.x
It’s also conservative about continuous
improvement
10
Jenkins is Conservative

NEW WEBSITE
NOW IT’S A CD AUTOMATION SERVER
FOCUS ON USER EXPERIENCE
UI IMPROVEMENTS
SECURITY IS ENABLED BY DEFAULT
PIPELINE AS CODE
11
Jenkins 2
http://bit.ly/jenkins20
Stats:
• Released on April 2016
• LTS line – July 2016
• 50% adoption over six months

Jenkins 2 is a milestone, work continues
USER EXPERIENCE
•Better stability of installations
•Better UIs (Blue Ocean, etc.)
ARCHITECTURE
•(?) Pluggable storage
•Remoting changes
•Jenkins in the Cloud – ”new” requirements
PROJECT INFRASTRUCTURE
•Contributor experience
•Better quality of plugins & core finally
12
What’s going on in Jenkins?

GOALS
• Share info about new Jenkins
features
• Share experience about
Pipeline usage in Embedded
• Give stickers & T-shorts for
good questions ;)
NO-GOALS
• Continuous Integration 101
• Jenkins 101
• “Classic” approaches to
Embedded use-cases
About this talk
JUC2015/London
More info:

What is an Embedded Product?
14
`
Hardware
External IPFirmware /
embedded
software
External
software
Development tools
/ EDAs
Demo software Demo hardware
Developer kits
Main Product
Documentation
Integ-
ration
Boom!

Almost everything needs to be automated
Multiple technologies and tools
Unstable hardware and networks
Legacy stuff is everywhere
Limited resources
Hardware engineers are not software experts
15
Embedded Area. Welcome to Automation Hell!
Technical
Non-technical

Approach: Use Free-style and Matrix:
https://jenkins.io/solutions/embedded/
Classic Approaches in Jenkins
JUC2015/London
More info:
Keywords:
• Script steps to run the logic
• Custom Tools - manage tools
• Throttle Concurrent Builds and
Lockable Resources - dispatch access
to the shared hardware
• Naginator - restart builds on-demand
• Parameterized Trigger / Copy Artifact -
parallelize tests and to collect results
16

Spaghetti Automation
• Many job dependencies
• Difficult build flow
tracking
• Complex configurations
• Duplication
• Maintenance is
expensive
Classic Approaches in Jenkins

ü Storage together with the project
ü Tracking together with the project
ü Tests together with the project
…. as code?
Configuration…
Documentation…
Pipeline…
Everything…

Jenkins Pipeline
19
Configuration as Code for Jenkins jobs
Not just a configuration as code

Pipeline is a…
Ø Groovy DSL for automatic flow description *
• Alternatives – Job DSL, Jenkins Job Builder
Ø Job type in Jenkins (Pipeline Job)
• Job definitions can be stored in SCM
• Automatic job creation from repositories and organizations
Ø Ecosystem for automation as code
• ~20 dedicated plugins
• Integration in many other plugins
• Suggested by default in Jenkins 2
// Run on a node having the “my_fpga” label
node("linux && ml509") {
git url:"http://github.com/myorg/myproject.git"
sh "make all"
} http://bit.ly/pipeline-tutorial
20

Snippet generator

Stored in SCM
• Jenkinsfile
Jenkins can create projects from SCM
• Multi-branch Pipeline
• Folder with a project
• Jobs for branches and pull requests
Works even on the organization level
• GitHub, BitBucket, etc.
22
Pipeline…. as code

Multi-Branch Pipeline

Pipeline. Useful Features
Robustness against restarts of Jenkins master
Robustness against network disconnects
• sh() steps are based on the Durable Task plugin
• Jenkins continues execution once the node reconnects
Parallelization across multiple nodes:
• It can be achieved via combination of parallel() and node() steps
• No need to use the trigger/copy chain (but you can do it)
Ability to store the shared logic in standalone Pipeline libraries
Visualization: Pipeline Stage View, Blue Ocean
24

Pipeline. Stage View Plugin
25
New UI: Blue Ocean is coming soon!

Example 2: Running tests in Parallel

Embedded Components Testing Approaches
Software Hardware
Simulators
EDA tools
FPGAs
ASIC prototypesHardware
Accelerators, etc.
x
27

Example. Jenkins Node with an Attached Board
28
VM or
bare-metal
Power
Reset
when it hangs
Deploy bit-files,
Blasting cables
Debugging cables,
firmware upload
We don’t
change them,
huh?
Jenkins node
Drivers
Tools
Peripherals
Attached peripherals
Runtime

Issue #1. Hardware is Unstable
29
Nodes @ Site A Nodes @ Site CNodes @ Site B
“We borrowed FPGAs
here, somebody takes
them randomly.”
“This stuff has
Windows drivers only.”
“No infra team
members here.”
“We run Jenkins
nodes on developer
laptops.”
Frankenboards
“We built them from
the garbage we had.”
L
L
L
L
L

Node Failover
Pipeline allows re-running unreliable stages on
other nodes.
System failure
Node #1 Node #2. . .
. . .
for (def board : boards) {
echo "trying board " + board;
try {
node(board) {
checkout scm
sh ‘./bin/run.sh’
// Call passed => DONE
break;
}
} catch (Exception ex) {
if (ex.message.contains
("exit code 255")) {
// Fatal error
fail("Test run failed")
}
}
}
Try all possible boards:
30

We want to…
• minimize the amount of hardware we use
• utilize the hardware as much as possible
• share hardware between projects
Nice to have:
• reuse the existing hardware
▸ Developer computers (nightly builds)
▸ Manual QA labs
▸ …
31
Issue #2. Hardware is Expensive!
M – x000$
L –x0000$
M - http://www.xilinx.com/univ/xupv5-lx110t.htm
L - HAPS-64, www.synopsys.com

Access should be dispatched by Jenkins
Builds utilize the hardware during a single phase only
Scheduled availability of nodes
Teams fight for resources
Clean environment (again)
32
Issue #3. Hardware is Shared

Resource Management
33
Dispatching parallel access:
• Lockable Resources plugin is
integrated with Pipeline
• Throttle Concurrent Builds
plugin is not integrated,
license limitations may be a
problem
Restricting access:
• Job Restrictions Plugin
• Node access schedule
echo 'Starting'
node('board1') {
stage('Build')
git url: 'http://github.com/myorg/myproject.git'
withEnv(["PATH+GCC=${tool ’gcc4'}/bin"]) {
sh 'make clean build'
}
}
stage(’Run tests')
lock(env.NODE_NAME + "-board1") {
// Run something on the board
// ...
}
}

Tools…
34

1. Local tool installation on a node
2. External Tool Managers (CLI)
3. Jenkins-based solutions
-Via plugins (if available)
-Generic approaches: Custom Tools,
EnvInject, etc.
-Limited availability in Pipeline
4. Virtualization
Tool usage approaches

Advantages:
•Quick setup if you have ready infrastructure
•Load balancing on physical hardware: vCenter, Docker Swarm
•Works well with configuration as code (e.g. Puppet)
•Clean environment
•Dozens of plugins in Jenkins
Disadvantages:
•Poor scalability when you have many tool combinations
•Complicated for BIG tools (common case for EDAs)
Virtualization for tools

Docker Plugin
Docker Pipeline Plugin
Docker Custom Build Environment Plugin
…
Yet Another Docker Plugin
Example. Docker plugins

Example. Docker-based tools and Pipeline
Docker Pipeline Plugin:
• You can share workspaces
• External Workspace Manager
• https://github.com/jenkinsci/external-
workspace-manager-plugin
• You can create build chains

You do NOT need to use a single node

Example
https://github.com/jenkinsci/external-workspace-manager-plugin

Jenkins Pipelines are Complex
How to reuse them?

Pipeline Shared Libraries
•Libs are located in SCM
•Versioning
•Definition for scopes:
▸Jenkins system-wide
▸Folder
▸Job

Pipeline Library Definition [1/2]

Pipeline Library Definition [2/2]

Using Pipeline Libraries

More examples:
https://github.com/jenkinsci/pipeline-
examples

Pipeline. What’s next?
• Stability / Fault tolerance
• UI/UX, Blue Ocean
• Plugin integrations
• Syntax improvements
• (?) Freestyle => Pipeline Converter

Blue Ocean
https://jenkins.io/projects/blueocean/
Ø New UI for Jenkins
Ø New UX for common Jenkins users
Ø Deep integration with Pipeline

Blue Ocean

Pipeline. Dev tools
Available:
•Minimal IDE: Auto-completion
•Minimal static analysis
•Documentation browser for steps
•Library manager
Missing:
•Debugger
•Real-time code deployment from IDE
•Real-time approval of Script Security
•Freestyle => Pipeline Converter

Summary. Embedded projects & Pipeline
SO…
• We successfully adopted Jenkins Pipeline in several projects
▸E.g. LibreCores CI (http://librecores.org/)
• We got advantage in user experience: Templates, Blue
Ocean, Pipeline Model Definitions
LESSONS LEARNT
• Pipeline greatly simplifies task parallelization and failover
within a single build
• Pipeline saves much time on automation system maintenance
• Pipeline is a programming languages, high barrier to entry
• Pipeline still lacks Dev Tools and integrations, but it evolves

There are many articles about Jenkins in
the Embedded area
•Most of them describe the “classic
approaches” based on
Freestyle jobs
Embedded solutions page:
•https://jenkins.io/solutions/embedded/
Blogposts:
•https://jenkins.io/node/
54
More Info Embedded Solutions Page
…

Jenkins project:
• Website: https://jenkins.io
• Embedded Solutions page: https://jenkins.io/solutions/embedded/
Pipeline:
• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md
• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md
• Examples: https://github.com/jenkinsci/pipeline-examples
CloudBees website:
• Website: https://www.cloudbees.com
• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform
55
Useful Links

Thank You!
Where to find me:
E-mail: onenashev@cloudbees.com
GitHub: oleg-nenashev
Twitter: @oleg_nenashev
Want to get more from Jenkins?
go.cloudbees.com

The Butler is still young – applying modern Jenkins features to the Embedded area

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The Butler is still young – applying modern Jenkins features to the Embedded area

Similar to The Butler is still young – applying modern Jenkins features to the Embedded area (20)

Recently uploaded

Recently uploaded (20)

The Butler is still young – applying modern Jenkins features to the Embedded area