Slides from my talk at NDC Techtown 2019. Abstract: Software updates has for a long time been a mess, consisting of “homegrown” solutions specific to a certain project and there was very little re-usage between projects and very little collaboration in our community to solve these complex problems. Luckily for us that time is over and the community around this topic has grown over last couple years and still is growing as the demand increases with the growth of IoT and OTA firmware updates (which introduces even more complexity). There are now well established open-source solutions that have been “battle tested” that we can collaborate on to make the complexity of software updates manageable. We are heading for a time where a quality Board Support Package should provide an software update implementation because it really should be solved at this level instead of handing this over to application developers which have limited knowledge of low lever architecture on a embedded device. In this talk Mirza will present some of the challenges of doing software updates on embedded system. He will also present the available open-source projects that can be used to solving these challenges. Projects such as mender.io, SWupdate, RAUC and more.

1. Mirza Krak
Embedded Linux Solutions Architect
Mender.io
NDC Techtown 2019
The ultimate guide to software updates on embedded Linux devices

2. Session Overview
● Introduction
● Embedded Linux overview and use-case
● Software update methods
● Existing solutions
Goal: Present an overview of software update techniques used
in embedded Linux and present existing solutions for software
updates

3. About Me
Mirza Krak
○ 8 years in embedded Linux
○ Background in automotive
○ Board Support Package development
○ Linux kernel contributor
Mender.io
○ Over-the-air update solution for embedded Linux
○ Open source (Apache 2.0 License)
○ Remote deployment management (server)
○ Under active development

4. Embedded Linux

5. Embedded Linux

6. Embedded Linux - areas of usage
● Transportation
○ Public transport
○ Fleet management
○ Tracking
○ Predictive maintenance
● Infrastructure
○ Power and water grids
○ Metropolitan areas
● Buildings
○ Monitoring
○ Energy optimization
○ Access control
○ Security
● Automotive
○ Infotainment
○ Telematics and connectivity
● “Things”
○ Appliances, Routers, TV’s...

7. Internet of Things (IoT)
The Internet of things (IoT) is the extension of
Internet connectivity into physical devices
and everyday objects. Embedded with
electronics, Internet connectivity, and other
forms of hardware (such as sensors), these
devices can communicate and interact with
others over the Internet, and they can be
remotely monitored and controlled
Source: WikipediaIt means taking all the things in the world and
connecting them to the internet

8. The embedded environment
● Remote
○ Expensive to reach physically
● Long expected lifetime
○ 5 - 10 years
● Unreliable power
○ Battery
○ Suddenly unplugged
● Unreliable network
○ Intermittent connectivity
○ Low bandwidth
○ Insecure
What can
go wrong?

9. Must be updatable
There will be bugs, vulnerabilities
… and new features
… after the device is deployed to the field

10. Cost savings by using open source platforms
Embedded System Stack
Cost
10%
30%
60%
Product
Differentiation
Use open source here to:
Lower layers are expensive and
provides no differentiation
HMI
Applications
Middleware
OTA update
Operating System
Board Support
Package
Hardware
● Lower cost on non-differentiating
components
● Shorten time-to-market
● Reallocate development to differentiation:
○ Applications and HMI

11. Software update on-site
● No connectivity
● Technician
○ USB flash drive
● Does not scale

12. Software update Over-The-Air (OTA)
● No easy access to the device
● Deployment management server
○ Traceability
○ Status reports
○ Current versions
○ Controlled roll-outs

13. Anatomy of embedded Linux
Bootloader: system initialization code starting with
the reset vector.
● Initialize and scrub RAM
● Setup power rails and clocks
● Load the “rest” of the system
Bootloader
Linux kernel
Device Tree (ARM)
Root filesystem
Kernel: core operating system functionality
● Resource management
● Process control
● Device drivers
Device Tree: hardware description (ARM)
Root filesystem: all files, executables, data, etc for the
system

14. Anatomy of embedded Linux (Image)
Linux kernel
Distribution
/
Applications
Update Client
Devicetree
File system image

15. Requirements (minimal)
● Able to update all components
○ Normally unsafe to update bootloader
● Never render the device unusable (brick)
○ Fail-safe / roll-back
● Atomic updates
○ No partial install
● Integrity check
● Signed images
○ Trusted images
● Compatibility check

16. Alternative approaches
● Image/block based updates
○ Easy to implement, test, verify and maintain
○ Can be complemented with binary delta
● Incremental atomic image upgrade mechanism
○ Complexity
● Containers
○ Run applications in containers on device
● Package managers (dpkg, dnf, opkg)
○ Not designed for embedded use-case
○ Not atomic
○ Hard to maintain

17. Asymmetric Image updates
● Android (pre N)
● Fail-safe
● Downsides
○ Downtime
○ Intermediate
storage
Bootloader
Persistent data
Device
OS A (active) Recovery OS
Main OS

18. Symmetric Image updates
● Android (post N)
● Seamless updates
● Fail-safe
● Roll-back
● Downside
○ Storage overhead
Bootloader
Persistent data
Device
OS A (active) OS B (inactive)OS A (active)

19. 1. In-place (deb,
rpm)
2. Asymmetric 3. Symmetric 4. Containers
Atomic (for
rollback)
Consistent fleet
Workflow
integration
Bandwidth (*) (*)
Downtime
Comparison of installer strategies
(*) Can be mitigated with compression or with “binary delta” transfers

20. Ecosystem

21. Categories
● Frameworks
○ SWUpdate (client)
○ RAUC (client)
○ libostree (infrastructure)
○ Hawkbit (standalone server)
● OTA Solutions
○ Mender.io
○ Updatehub.io
● OTA Solutions (SWUpdate + Hawkbit)
○ FullMetalUpdate
○ Update Factory
● Full stack (includes OTA)
○ foundries.io (libostree and containers)
○ balena.io (former resin.io, containers)
○ Torizon (based on foundries.io)
○ ….
● Proprietary (Automotive)
○ ….

22. SWUpdate
“SWUpdate is a Linux Update agent with the goal to provide an efficient and safe
way to update an embedded system”
● http://sbabic.github.io/swupdate/
● C & GPLv2
● Update agent on device
○ Integration with Hawkbit
● Integrated web server for “local updates”
● Update types:
○ Symmetric/Asymmetric Image Updates (rdiff)
○ In-place (files, tar.gz etc..)
○ Proxy firmware
● Cryptographic signing and verification of updates
● Yocto & Buildroot integration

23. RAUC
“RAUC controls the update process on embedded Linux systems. It is both a target
application that runs as an update client and a host/target tool that allows you to
create, inspect and modify installation artifacts.”
● https://www.rauc.io/
● C & Python
● License: LGPL-2.1
● Update agent on device
○ Integration with Hawkbit
● Integrated web server for “local updates”
● Update types:
○ Symmetric/Asymmetric Image Updates (casync)
○ In-place (files, tar.gz etc..)
● Cryptographic signing and verification of updates
● Yocto, Buildroot and PTXDist Integration

24. Hawkbit
“Eclipse hawkBit™ is a domain independent back-end framework for rolling out
software updates to constrained edge devices as well as more powerful controllers and
gateways connected to IP based networking infrastructure.”
● https://www.eclipse.org/hawkbit/
● Java
● Eclipse Public License 1.0
● Standalone server
○ Client integrations are provided by users
○ SWUpdate & RAUC are example of clients
● Eclipse Foundation project

25. Hawkbit - features at glance
● A device and software repository.
● Artifact content delivery.
● Software update and roll out management.
● Reporting and monitoring.
● Interfaces:
○ for direct device control.
○ for IoT solutions or applications to manage the
repository and the roll outs.
○ for device management federation (i.e. indirect
device control)
○ and a user interface to operators to manage and
run the roll outs.
● More on features here

26. Torizon
“Torizon is a new Linux-based software platform that simplifies the process of developing and maintaining embedded
software. It allows you to configure the system for your use case quickly and easily, so you can focus on application
development instead of Linux builds.”

27. Mender
Mender’s prime directive: robust and secure OTA updates
● Apache 2.0 & Golang
● Asymmetric images updates (binary delta coming soon)
○ Can be extended using the Update Module
framework
● Integrity checks to avoid corruption
● Device groupings for controlled rollout management
● Secure TLS communication between server/device
● Signed images
● Yocto, Buildroot, Debian, Ubuntu, Raspbian, OpenWRT

28. Mender
● Singular complete solution:
○ Fully integrated on-target client and web-based management server
○ Ability to update both your kernel and applications in one mechanism
● Mender supports both managed OTA updates and standalone deployments.
● Interface to adapt the update flow to application-specific use cases, e.g. user confirmation
● Microservices API-based architecture allows simple adaptation to custom requirements.
● Low system overhead

29. Mender and the value of an OSS community
● Mender is the only end-to-end open source OTA manager (both management server and client)
○ No vendor lock-in
○ Permissive Apache 2.0 license
○ A healthy peer review mechanism with more eyes on the code to increase its security
● Less reliance on tribal knowledge; less expensive than a homegrown system
○ Homegrown systems have unpredictable scope, which leads to more cost and time
○ Reliance on tribal knowledge: if a key person leaves, could cause serious issues
● Mender Hub is the only online community dedicated to enable OTA on any device and OS
○ Get started quickly
○ 40+ board integrations

30. Mender Hub
A community-driven
repository to enable OTA
updates with Mender on
every board and OS!

31. Thank You!
Q&A
@KrakMirza
mirza.krak@northern.tech
https://www.linkedin.com/in/mirzakrak/
@mender_io
https://mender.io
https://docs.mender.io/2.0/getting-started
https://hub.mender.io