To introduce a robust, secure and reliable platform for the industrial environments is a key challenge; moreover, the platform needs to survive for a long time (more than 10+ years). There are many good solutions aiming to meet these requirements, such as LTSI (Long Term Support Initiative) and CIP (Civil Infrastructure Platform). However, it still needs a high amount of maintenance and development costs in handling SoC/ hardware board in-house patch, non-upstream driver and keep source code consistent with different SoC and platform afterwards. In this presentation, SZ Lin will introduce how to operate long-term maintenance model of embedded industrial Linux distribution. In addition, he will also address the building, deploying and testing architecture and workflow for producing a robust, secure and reliable platform.

1. Long-term Maintenance Model of
Embedded Industrial Linux
Distribution
LINUXCON + CONTAINERCON + CLOUDOPEN CHINA 2017
SZ Lin (林上智)
MXcore, Software Supervisor
06/19 2017

2. Confidential
About Me
SZ LIN (林上智)
 Software Engineer at Moxa
• Industrial-grade Linux Distribution
 Debian Developer
 Blog - https://szlin.me
2

3. Confidential
Industrial/ Harsh Environments
Including smart rail, smart grid, intelligent transportation,
factory automation, oil & gas, marine, and more
Confidential3

4. Confidential
Smart
Grid
Smart
Rail
Smart
Oil Field
Smart
Factory
Smart
Transportation
Smart
Marine
Application
Device
Industrial
Routers
Industrial
Ethernet
Industrial
Wireless
LAN
Network Infrastructure
Confidential
Embedded Computers
Industrial Computing
Industrial Linux Platform
Serial
Connectivity
I/O
Connectivity
Video
Connectivity
Edge Connectivity
4

5. Confidential
Smart
Grid
Smart
Rail
Smart
Oil Field
Smart
Factory
Smart
Transportation
Smart
Marine
Application
Device
Industrial
Routers
Industrial
Ethernet
Industrial
Wireless
LAN
Network Infrastructure
Confidential
Embedded Computers
Industrial Computing
Industrial Linux Platform
Serial
Connectivity
I/O
Connectivity
Video
Connectivity
Edge Connectivity
5
HardwareSoftware

6. Confidential
Bug fixes
Security
fixes
New userspace
program backports
New kernel
features backports
Industrial Linux Distribution
Over 10+ years
Maintenance release
6

7. Confidential
Maintenance PhaseDevelopment Phase
Hardware
Survey
Regular
Update
Kernel/ RFS
Preparation
4321
CI/ CD/ LT
7

8. Confidential
Development Phase
Take action from the very beginning
8

9. Confidential
Development Phase
Hardware
Survey
Regular
Update
4321
CI/ CD/ LT
Kernel/ RFS
Preparation
9

10. Confidential
Why We Need Hardware Survey
Image:
https://upload.wikimedia.org/wikipedia/commons/5/5b/Linux_kernel_map.png
10

11. Confidential
Hardware Survey – Key Components
Short LifecycleLong Lifecycle (> 10 years)
SoC
e.g. ARM, MIPS
Non-volatile memory
e.g. NAND flash, eMMC,
SD, SSD
DRAMRTC
Watchdog PMIC
Cellular
module
Wi-Fi
module
PHY
Switch
module
UART
module WPAN
module
External LAN controller
11

12. Confidential
Hardware Survey - Long-term Supply
Action Pros Cons
Build your own hardware
components
• Long-term support
• Create hardware innovation
Not cost-effective
Sign long-term contract
with hardware component
provider
• Long-term support Follow
manufacturers’
regulations
Do nothing Choose what you want Massive resource
cost
12

13. Confidential
Development Phase
Hardware
Survey
Regular
Update
4321
CI/ CD/ LT
Kernel/ RFS
Preparation
13

14. Confidential
Kernel/ RFS Preparation
1
Choose proper RFS (Root filesystem)
Stable root filesystem
2
Unified kernel
All SoC is using the same version of kernel
Choose proper kernel
Long-term maintenance kernel version
3
Apply patches
Build custom kernel
4
14

15. Confidential
Choose Proper Kernel
Long-term maintenance kernel version
15

16. Confidential
SoC Board Support Package Kernel
• Kernel version depends on SoC vendors
– Well made but not well maintained
• Contain lots of in-house patches
– Errata patches
– Specific feature patches
– …
• Different SoC might use different versions of
kernel
• The lifetime is unsure
16

17. Confidential
LTS: Long Term Stable Kernel [1]
Extend software uptime for stable kernel
• Only accept bug fixes and security fixes
img: https://www.kernel.org/category/releases.html
17

18. Confidential
LTSI: Long Term Support Initiative [2]
• Linux Foundation collaborative project
– Based on LTS
– Add another chance to include further patches on top
of LTS
– Auto Test framework
– Same lifetime with LTS (yearly release and 2 years life
time)
18

19. Confidential
CIP (Civil Infrastructure Platform) [3]
• Linux Foundation collaborative project
– Support kernel and core package
– Auto Test framework
– Maintenance period
• 10 years and more (10-20 years)
19

20. Confidential
img: https://wiki.linuxfoundation.org/civilinfrastructureplatform/cipconferences
20

21. Confidential
Comparison Table
Version
Maintenance
Period
(years)
Features
Latest
version
Root
filesystem
Maintainer
SoC
BSP
kernel
? Bug fixes ? N
SoC vendor
kernel team
LTS
kernel
2 ~ ?
• Bug fixes
• Security fixes
4.9 N Kernel.org
LTSI
kernel
LTS + 2
• Bug fixes
• Security fixes
• Specific features
• New features
4.1 N LTSI
CIP
kernel
10 +
• Bug fixes
• Security fixes
• Specific features
• New features
4.4
Y
(Project X)
CIP
21

22. Confidential
Choose proper RFS
(Root filesystem)
Stable root filesystem
22

23. Confidential
img: http://lmelinux.net/wp-content/uploads/sites/2/2014/07/linux_distros_2013.jpg
23

24. Confidential
Good system security [5]
Everything is open
Usually, fixed packages are uploaded
within a few days
Stability
unstable → testing → stable
Scalability
Server, Desktop,
Laptop, Embedded devices
Long term support
5 more years by Debian-LTS project
(i386, amd64, armel and armhf)
Multiple architectures
alpha, amd64, armel, armhf,
aarch64, hppa, i386, ia64, mips,
mipsel, powerpc, s390, and spar
Why We Choose Debian [4]
Incredible amounts
of software
Debian comes with over 43000
different pieces
of software with free
24

25. Confidential
Unified kernel
All SoC is using the same version of kernel
25

26. Confidential
Unified Kernel
• Kernel supports multi-platform
• Centralized topology
– Use the same kernel version
• Long-term maintenance
– Apply patches through a single entry point
• R&D resources focus
– Modify Kernel configuration
– Edit Device tree
– Prepare SoC-related patches
26

27. Confidential27

28. Confidential
Apply Patches
Build custom kernel
28

29. Confidential29

30. Confidential30

31. Confidential31

32. Confidential

33. Confidential
How to Find Out In-house Patch By
Using Your Own Custom Kernel
• Get hardware component errata/ datasheet
• Get hardware component official driver if
possible
– It always surprises you
• Refer to mainline kernel
– It might be fixed already
• Time to backport
• Discuss the issue in subsystem mailing list or
forum
– Make the patch
33

34. Confidential
Maintenance Phase
Regular update and long-term testing
34

35. Confidential
Maintenance Phase
Hardware
Survey
Regular
Update
Kernel/ RFS
Preparation
4321
CI/ CD/LT
35

36. Confidential
Regular Update Mechanism
1
Well-testing
Stable, stable and stable
2
Backported patches
Include device drivers and new kernel features
Upstream first
Do not create in-house patch after fixing bug
3
Minimal modification
Apply the patches only what we need
4
Bug fixes
New feature patches
Security fixes
New device drivers
Backported patches
…
36

37. Confidential
Backported Patches in Maintenance
Phase
• Try not to modify *ANY API* (if possible…)
• Try not rename/ remove any member of structure
– “Add” behavior is also risky
• Regression test
• Unit test
• Do not apply lots of commits
– Handle it carefully
• git cherry-pick
• git blame
• git bisect
• …
37

38. Confidential
Maintenance Phase
Hardware
Survey
Regular
Update
Kernel/ RFS
Preparation
4321
CI/ CD/LT [13]
More info: Building, Deploying and Testing an Industrial Linux Platform
Open Source Summit Japan 2017 [13]
38

39. Confidential
Industrial
Linux
Distribution
CI
CDLT
Ecosystems for Industrial Linux Distribution
CI : Continuous Integration
CD : Continuous Delivery/ Deployment
LT : Long-term Test
39

40. Confidential
Master
Develop
Feature ..
Merge
Pull Request
Fork
Local
Branch
Patches
Notification
Internal/ External
Developers
CI/ CD Automatic
Release Pipeline
Maintainer
Approval
Pass
Pass
Y
Y N
N
40

41. Confidential
CI/ CD Automatic Release Pipeline
Building Testing
Deploying
4321
Release
Continuous Delivery Continuous
Deployment
Code
Continuous
Integration
41

42. Confidential
Master
Develop
Merge
Notification
CI/ CD Automatic
Release Pipeline
Maintainer
Approval
Pass
Pass
Y
Y N
N
42
Image
Deployment
24/ 7 Long-term
Platform Test

43. Confidential
Notification
Test Results
43
* Test cases are managed by LAVA

44. Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [12]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [11]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
44
Endurance test
Compatibility test
…

45. Confidential
24/ 7 Long-term Platform Test
Robustness
Robustness is the ability
of a computer system to
cope with errors during
execution and cope with
erroneous input [12]
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [11]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
Longevity
Long-term support at
least 10 years life cycle
with bug fixes, new
features and new
hardware components
45
Fuzz testing
[14][15][16]
…

46. Confidential
24/ 7 Long-term Platform Test
Reliability
Reliability is enhanced
by features that help to
avoid, detect and repair
hardware faults [11]
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
46
Power failure test
Reboot test
…

47. Confidential
24/ 7 Long-term Platform Test
Security
Quick response in
resolving CVE/
vulnerabilities and
attacks in platform
47
Daily test for CVE [25]
…

48. Confidential
Tools
Continuous Integration Jenkins [17]
Continuous Delivery/ Deployment LAVA 2 [18]
Test Case Management • Jenkins
• LAVA 2
Version Control Git with gitlab [19]
Static Program Analysis • Coding style
• OWASP [17]
• Infer [20]
• Sonarqube [21]
Dynamic Program Analysis • Gcov [22]
• Valgrind [23]
• Profiling tools [24]
Security Testing OpenVAS [25]
Fuzzing Testing • Syzkaller [14]
• Trinity [15]
• OSS-fuzz [16]
48

49. Confidential
CI/ CD/ LT are
concepts of software engineering
instead of
tools or procedures
49

50. Confidential50
Community
Collaboration
Upstream FirstPreparedness
Planning
24/7 Testing
Stable, stable and stable
Conclusion

51. Confidential51

52. © Moxa Inc. All rights reserved.
Thank You

53. Confidential
References
[1] https://www.kernel.org
[2] http://ltsi.linuxfoundation.org
[3] https://www.cip-project.org/
[4] https://www.debian.org/intro/why_debian.en.html
[5] https://www.debian.org/security/index.en.html
[6] http://ck.kolivas.org/patches/
[7] https://www.kernel.org/pub/linux/kernel/projects/rt/
[8] http://elinux.org/images/2/2e/MiniDebianConfJapan-
Yoshi.pdf
[9] https://openwrt.org/
[10] https://backports.wiki.kernel.org/index.php/Main_Page
[11]https://en.wikipedia.org/wiki/Reliability,_availability_and_
serviceability
53

54. Confidential
References
[12] https://en.wikipedia.org/wiki/Robustness_(computer_science)
[13] http://events.linuxfoundation.org/sites/events/files/slides/Build
ing%2C%20Deploying%20and%20Testing%20an%20Industrial%20Linux%
20Platform.pdf
[14] https://github.com/google/syzkaller
[15] http://codemonkey.org.uk/projects/trinity/
[16] https://github.com/google/oss-fuzz
[17] https://wiki.jenkins-ci.org/display/JENKINS/Plugins
[18] https://www.linaro.org/initiatives/lava/
[19] https://about.gitlab.com/
[20] http://fbinfer.com/
[21] https://www.sonarqube.org/
[22] https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
[23] http://valgrind.org/
[24] https://perf.wiki.kernel.org/index.php/Main_Page
54

55. Confidential
References
[25] http://www.openvas.org/
[26] http://events.linuxfoundation.org/sites/events/files/slides/2017-02-
22_CIP-ELC-r7.pdf
[27] http://events.linuxfoundation.org/sites/events/files/slides/PRE-trunk-
ELCE-LongTermMaintenance.pdf
[28] http://events.linuxfoundation.org/sites/events/files/slides/Efficient_ke
rnel_backporting.pdf
[29] http://events.linuxfoundation.org/sites/events/files/slides/LTSI-
ELC2017-final.pdf
[30] https://lwn.net/Articles/700530/
[31] Special thanks to Bruce Chen and Roy Hsu
55