By Sarah Woodall, NXP Semiconductors LPCXpresso is a multi-platform IDE for developers of embedded software to run on NXP Semiconductor's ARM-based microcontrollers. NXP needs to test that the debugger can execute programs on numerous different development boards that connect to the USB ports of host computers. Besides building a complex software product, the Jenkins installation drives an automated test farm consisting of home-built software-controlled USB switches ("cows") that control a huge array of combinations of test board, debug probe and host platform. This talk will give a tour of the NXP farm, including video of the cows in action, and will describe the features of Jenkins that are used to make it work, with particular emphasis on dynamic selection of combinations within matrix jobs, parameterized triggers and the Summary Display plugin. Finally, plans to migrate to the new Workflow plugin will be discussed. NXP believes the Workflow plugin will simplify the structure and make it more maintainable.

1. The Famous Cows of Cambridge
A non-standard use-case for Jenkins
Sarah Woodall, NXP Semiconductors

2. #jenkinsconf
Footer
What this talk is about
• How (and why) we use Jenkins to test our software
automatically using a farm of custom hardware
• The plugins and features of Jenkins that help
• Improvements we would love to see
• Why we think the Workflow Plugin will be important
2

3. #jenkinsconf
Setting the scene

4. #jenkinsconf
Some of the world’s most photographed cows
4
Photo by Alex Brown
(originally posted to Flickr
as Cows and King's)
[CC BY 2.0 (http://
creativecommons.org/
licenses/by/2.0)], via
Wikimedia Commons

5. #jenkinsconf
The product that our Jenkins builds and tests
A software
development
toolchain based
on Eclipse and
GCC/GDB
• Designed for ease of use
with NXP’s ARM-based
LPC microcontrollers
• Hosted on Windows, Mac,
Linux (separate installers)

6. #jenkinsconf
LPCXpresso in normal use
6
Host computer
running
LPCXpresso
Debug probe
USB
Development board
Target MCU
Stub

7. #jenkinsconf
LPCXpresso’s components
Open-source IDE built and tested by the Eclipse Foundation
Public releases three times a year
GNU tools (GCC compiler, GDB debugger, ...) built and tested by ARM
Public releases four times a year
Project wizards; Debug perspective; MCU definitions;
support for advanced features such as Trace, etc ...
Written locally: our responsibility to build and test
Debug stubs: let GDB talk to NXP MCUs on target boards via USB
Many target/probe/host platform combinations => many stubs!
Written locally: our responsibility to build and test
Target software: C Libraries; LPC-Open related components; ...
Written locally: our responsibility to build and test

8. #jenkinsconf
Testing LPCXpresso’s
debug stubs
8

9. #jenkinsconf
Manual testing will not get us very far
9

10. #jenkinsconf
Debug stubs: the scale of the testing problem
This matrix is only a subset
• New targets and debug probes
are constantly being added
• We have to run the same tests
over again for each host platform

11. #jenkinsconf
It’s not enough to run the tests just once
Why test frequently?
• It’s much easier to find problems straight away
• Developers see what is wrong while their work
is still fresh in their minds
Why test everything, everywhere?
• Many of the stubs share common code:
change one – break the others!
• Stubs work with the host OS, so they behave
differently on each platform
11

12. #jenkinsconf
The solution: automate testing with Jenkins
• Jenkins builds the three LPCXpresso installers
– (How it does that is a whole talk for another day)
• A set of host machines are set up as Jenkins slaves
– Windows 7, Windows 8.1, Ubuntu, Fedora, Mac, …
• TestInstall (a matrix job)
– Uses the Copy Artifact plugin to get the right installer
– Runs it silently and checks the result
12

13. #jenkinsconf
Test a wide range of target/probe combinations
• Using a big matrix job, each testing slave can
– use every kind of debug probe to communicate
with target boards
– execute a set of standard images compiled for
each kind of target hardware
13

14. #jenkinsconf
The snag: how to power-cycle the test boards?
• In normal use, nobody would expect to have multiple
boards and debug probes attached to one computer
– They are not designed to work like that!
• Many of them have to be switched off and on again
as well as being booted before use
• How can we simulate this from Jenkins?
14

15. #jenkinsconf
The answer: our test farm has COWs
15
COW =
“Collection
of Wires”

16. #jenkinsconf
The test farm (much simplified)
16
Jenkins master
TestWindows TestMac TestUbuntu ... many similar Jenkins slaves
Ermintrude Daisy Poppy Willow
USB
probe
+
target
... many similar cows
... and very many more
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
USB

17. #jenkinsconf
Part of the Cambridge herd
17

18. #jenkinsconf
An expatriate cow
18
Our Jenkins master drives this laptop in California as a (rather slow) slave

19. #jenkinsconf
How are our cows connected today?
• Cows are physically
plugged in to particular
test machines
• We run a special job to
tell Jenkins whenever a
cow has been moved
• All the test jobs begin by
copying over the artifact
that this job created
19

20. #jenkinsconf
Summary Display shows test coverage available
The result of the
ConfigureTestFarm
job shows us
where the gaps in
our testing are

21. #jenkinsconf
Test results overview shown in Jenkins
Results table
created with the
Jenkins Summary
Display Plugin

22. #jenkinsconf
Tests are defined using Robot Framework
• Open source; easily extended using Python
• Works on all our supported platforms
• Ready-made output display formatted in HTML
• Easy to integrate with Jenkins
22
robotframework.org

23. #jenkinsconf
Links in our results table lead to Robot’s reports

24. #jenkinsconf
Drilling down further shows more detail

25. #jenkinsconf
A cow in action
25

26. #jenkinsconf
A cow driving a development board
26

27. #jenkinsconf
More automation
Testing other aspects of LPCXpresso

28. #jenkinsconf
Importing and compiling 2000 templates
28

29. #jenkinsconf
Uploading, downloading, …
• Jenkins uploads our build artifacts to a cloud server
for easy access by colleagues in remote locations
• A shared cloud slave enables communication with a
distant group’s Jenkins instance
– Their Jenkins uploads zipped packages as artifacts
– Our Jenkins fetches them, unpacks them and runs
them through our test farm, reporting results by email
29

30. #jenkinsconf
The plugins and features of
Jenkins that help us most

31. #jenkinsconf
Our favourite plugins
• Exclusion
• Conditional buildstep
• EnvInject
• Parameterized trigger
• Matrix combinations
• Promoted builds
• Copy artifact
• Node/label parameters
• Summary display
• Description setter
• Build timeout
• Green balls
31

32. #jenkinsconf
Things we do with build parameters
• Check out from a branch of the source
• Name a separate exclusion resource on each slave
• Allow interactive input to Groovy scripts
• Select matrix combinations dynamically
32

33. #jenkinsconf
Selecting matrix combinations dynamically
33
This is part of the
configuration for a job
called LaunchTestMatrix

34. #jenkinsconf
Matrix combinations example
This is generated automatically by a script
COMBINATIONS=(PROBE=="LPC-Link"&&(TARGET=="LPC2138"||
TARGET=="LPC11U14_201"||TARGET=="LPC1114_301"||
TARGET=="LPC1549"))||(PROBE=="RedProbe
+"&&(TARGET=="LPC2368"||TARGET=="LPC1768"))||(PROBE=="LPC-
Link2"&&(TARGET=="LPC4337"||TARGET=="LPC1768"||
TARGET=="LPC4330_SPIFI_1M_64K"||TARGET=="LPC1227_301"||
TARGET=="LPC1830_NGX_SPIFI"||TARGET=="LPC4370_SPIFI"||
TARGET=="LPC1549"||TARGET=="LPC11U68"||
TARGET=="LPC54102J512"||TARGET=="LPC810"))||(PROBE=="RDB-
Link"&&(TARGET=="LPC1768"))||(PROBE=="CMSIS-
DAP"&&(TARGET=="LPC11U68"||TARGET=="LPC824"||
TARGET=="LPC1549"||TARGET=="LPC4370_SPIFI"||
TARGET=="LPC54102J512"||TARGET=="LPC1768"))
34

35. #jenkinsconf
Exclusion to manage shared resources
The whole point of the cows is to have only one target
board/probe combination switched on at one time.
• What if more than one job is running on the slave?
Our solution: use the Jenkins Exclusion Plugin
• Define a resource called ${SLAVE}_BOARDS
• Claim it before running each test
35

36. #jenkinsconf
Our Jenkins wish-list

37. #jenkinsconf
We do love Jenkins, but it would be nice if …
• You could select one configuration in a matrix job
when copying artifacts from a permalink
• There was proper support for source-code branches
• The Cross-platform shell plugin worked better
• Matrix jobs were first-class citizens
37

38. #jenkinsconf
Automated is good,
but scripted is better!

39. #jenkinsconf
Our job structure is quite complicated
Visualization created with the
Jenkins Dependency Graph Plugin
39

40. #jenkinsconf
We think the Workflow plugin will help
• Allows complicated job structures to be expressed
concisely as scripts
• Easier to read, to write and to manage
But
• We shall need good support for matrix jobs
40

41. #jenkinsconf
What is special about our
use-case for Jenkins

42. #jenkinsconf
Our world is a matrix world
• Our test farm is a sparse multi-dimensional matrix
• The hardware is reconfigurable, so the Jenkins
model of it has to be flexible, too
• We need build parameters and combination filters to
manage this complexity
• The proper functioning of matrix jobs is essential
42

43. #jenkinsconf
• www.lpcware.com/lpcxpresso
• sarah.woodall@nxp.com
• sarah@woodall.org.uk
Questions?

44. London, UK 23-24 June