To introduce a robust, secure and reliable platform for the industrial environments is a key challenge. Therefore, running with the industrial-grade Linux distribution to fulfill the requirements mentioned above is imperative. The Linux distribution includes the Linux kernel and user space. Based on this testing design, the distribution will be built, deployed and tested in the device under automatic test by using continuous integration development practice to withstand the harsh industrial environments. In this presentation, SZ Lin will introduce how the industrial-grade Linux distribution is built, deployed and tested without human intervention, and review the test scope in both Linux kernel and user space. In addition, he will also address the design architecture of 24/7 long-term automated testing in all device under test with each release of new update.

1. Building, Deploying and Testing
an Industrial Linux Platform
SZ Lin (林上智)
MXcore, Software Supervisor
06/01 2017
Open Source Summit Japan 2017

2. Confidential
About Me
SZ LIN (林上智)
 Software Engineer at Moxa
• Industrial Grade Linux Distribution
 Debian Developer
 Blog - https://szlin.me
2

3. Confidential
Industrial Linux Platform
3

4. Confidential
Industrial/ Harsh Environments
Including smart rail, smart grid, intelligent transportation,
factory automation, oil & gas, marine, and more
Confidential4

5. Confidential
Smart
Grid
Smart
Rail
Smart
Oil Field
Smart
Factory
Smart
Transportation
Smart
Marine
Application
Device
Industrial
Routers
Industrial
Ethernet
Industrial
Wireless
LAN
Network Infrastructure
Confidential
Embedded Computers
Industrial Computing
Industrial Linux Platform
Serial
Connectivity
I/O
Connectivity
Video
Connectivity
Edge Connectivity
5

6. Confidential
Smart
Grid
Smart
Rail
Smart
Oil Field
Smart
Factory
Smart
Transportation
Smart
Marine
Application
Device
Industrial
Routers
Industrial
Ethernet
Industrial
Wireless
LAN
Network Infrastructure
Confidential
Embedded Computers
Industrial Computing
Industrial Linux Platform
Serial
Connectivity
I/O
Connectivity
Video
Connectivity
Edge Connectivity
6
HardwareSoftware

7. Confidential
Bug fixes
Security
fixes
New userspace
program backports
New kernel
features backports
Industrial Linux Distribution
Over 10+ years
Maintenance release
7

8. Confidential
Industrial
Linux
Distribution
CI
CDLT
8
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test

9. Confidential9
Continuous Integration
and
Continuous Delivery/ Deployment

10. Confidential
Industrial
Linux
Distribution
CI
CDLT
10
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test

11. Confidential
Master
Develop
Feature ..
Merge
Pull Request
Fork
Local
Branch
Patches
Notification
Internal/ External
Developers
CI/ CD Automatic
Release Pipeline
Maintainer
Approval
Pass
Pass
Y
Y N
N
11

12. Confidential
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4321
Release
12

13. Confidential
Code
Continuous
Integration
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4321
Release
13

14. Confidential
Server
Slave
node
Webhooks
Patches
Internal/ External
Developers
Local
Branch
Pull Request
Slave
node
Slave
node
Static Program
Analysis
 Coding style
 OWASP [2]
 Infer [3]
 Sonarqube [4]
 …
14

15. Confidential
Test Cases Management - Jenkins
15
Static
analysis
#1
Static
analysis
#2
…
Static
analysis
#n

16. Confidential
Image: https://c1.staticflickr.com/5/4030/4438139050_04604b4908.jpg
16

17. Confidential
Distributed Compiler
• Software
– Icecream/ IceCC was created by SUSE based on distcc
[5][6]
• Improve performance of compile jobs in parallel
• Add dynamic scheduler of the compilation jobs
• Support multiple platform
• Support cross compiling
• Hardware - for each node
– SSD
– Large capacity memory
– Gigabit LAN
17

18. Confidential
Client Client Client
ICECC - Distributed CompilerServer
Patches
Internal/ External Developers
Local
Branch
Pull Request
3. Send
log & image
18
Webhooks
1. Send command
2. Get source

19. Confidential
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4321
Release
Code
Continuous
Integration
Continuous Delivery
19

20. Confidential
Continuous Delivery – LAVA [7][8]
20

21. 21
Server
Master
Worker WorkerWorker
1. Send job file via XML-RPC
5. Deployment via TFTP
2. Dispatch job via ZMQ
3. Download image
via curl or wget
4. Boot up via Ethernet remote I/O
DUT Clusters
Dynamic Program
Analysis
● gcov [9]
● valgrind [10]
● profiling tools [11]
● ...
Platform Test
● LTP [12]
● Security testing
[13]
● Kselftest [17]
● ...
6. Send test command

22. Confidential
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4321
Release
Continuous Delivery Continuous
Deployment
Code
Continuous
Integration
22

23. Confidential
Master
Develop
Merge
Notification
CI/ CD Automatic
Release Pipeline
Maintainer
Approval
Pass
Pass
Y
Y N
N
23
Image
Deployment
24/ 7 Long-term
Platform Test

24. Confidential
Long-term Test
24

25. Confidential
Industrial
Linux
Distribution
CI
CDLT
25
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test

26. Confidential
Notification
Test Results
26
* Test cases are managed by LAVA

27. Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [18]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [1]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
27
Endurance test
Compatibility test
…

28. Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [18]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [1]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
28
Fuzz testing
[14][15][16]
…

29. Confidential
24/ 7 Long-term Platform Test
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [1]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
29
Power failure test
Reboot test
…

30. Confidential
24/ 7 Long-term Platform Test
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
30
Daily test for CVE
…

31. Confidential
CI/ CD/ LT are
concepts of software engineering
instead of
tools or procedures
31

32. Confidential32
Collaboration with
upstream developer
Bootloader test integration
Performance/ Unit test
integration
Test cases framework
enhancement
Keep eyes on Fuego
Future Work

33. Confidential33

34. © Moxa Inc. All rights reserved.
34
Thank You

35. Confidential
References
[1]https://en.wikipedia.org/wiki/Reliability,_availability_and_servic
eability
[2] https://wiki.jenkins-ci.org/display/JENKINS/Plugins
[3] http://fbinfer.com/
[4] https://www.sonarqube.org/
[5] https://github.com/icecc
[6] https://www.slideshare.net/szlin/distributed-compiler-icecc
[7] https://www.linaro.org/initiatives/lava/
[8] http://elinux.org/images/3/35/LAVA_Project_Update.pdf
[9] https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
[10] http://valgrind.org/
[11] https://perf.wiki.kernel.org/index.php/Main_Page
35

36. Confidential
References
[12] http://linux-test-project.github.io/
[13] http://www.openvas.org/
[14] https://github.com/google/syzkaller
[15] http://codemonkey.org.uk/projects/trinity/
[16] https://github.com/google/oss-fuzz
[17] https://kselftest.wiki.kernel.org
[18] https://en.wikipedia.org/wiki/Robustness_(computer_science)
36

37. Confidential
Meet Our Team
Elvis Yao
Software engineer
at Moxa System.
Fero Zhou
Software engineer
at Moxa System.
Gavin Lai
Software engineer at
Moxa Networking.
37