Different upgrade and update strategies exist when it comes to embedded Linux system. If at development time none of these strategies have been chosen, adding them afterwards can be tedious task. Even harder it gets when the system is already deployed in the field and only accessible via a 3G connection. This talk is a developer experience of putting in place exactly that. Giving a return of experience on one way of doing it on a system running Barebox and a Yocto-based distribution. Patrick Boettcher

1. Remote-update adventures
RAUC, Yocto, Barebox
Embedded Recipes 2019
2019-09-24 – Paris Story, Paris
Patrick Boettcher YAISE

2. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
About me
●
Kernel developer since 2004 – (media drivers)
●
Embedded C++, C (mostly Linux and RTOS),
Python, compilers and testing frameworks
●
Freelancer with YAISE

3. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Overview
1- Intro / Starting position
2- Remote update-strategy
3- Remote upgrade-strategy

4. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
The project
●
industrialized LORA-gateway (LoRaWan
with 3G/4G
●
special housing and connectors
●
selected Phytec phyCORE-i.MX 6UL
– 512MB RAM
– 512MB NAND
– ARM Cortex A7 – up to 900MHz
●
Phytec provides a customizable Yocto distribution

5. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Phytec’s Yocto Distro
●
2018
●
based on morty (yocto 2.2)
●
using barebox as bootloader
●
device-tree well handled (barebox merges with
dtb and hands all of it to the kernel)
●
no “standard” system-updade mechanism in this
release

6. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
System configuration
●
barebox (with built-in-device-tree)
●
barebox-env
●
kernel
●
dtb
●
rootfs-partition (~490MB )
●
system is accessible via remote-ssh-tunnel (initiated
by the system, via the WAN connection)

7. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
What happened?
●
June 2018: first 100 devices have been delivered with the
application-layer running nicely
●
September 2018: a software update requires update of the
base-system (kernel and libc)– of course
– tried to do it with opkg/ipk – not satisfying (to be retried)
– decision to study upgrade possibilities
– learned a lot during #er2018 when discussing with Marek
●
October 2018: Decision-time (customer pays):
– 1st define general update strategy
– 2nd define strategy to update the deployed 100 systems

8. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Overview
1- Intro / Starting position
2- Remote update-strategy
3- Remote upgrade-strategy

9. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Basic facts after study - 1
Stick close to what exists and
do not re-invent the wheel, but what does exist?
●
Partitioning (UBI) and filesystem (UBIFS) cannot be
modified from Linux’s user-space running on UBI
●
Putting processes “somewhere else” to change UBI is
complex; barebox can of course do it
●
Barebox has bootchooser
●
Barebox has “state”-partition

10. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Basic facts after study - 2
●
Barebox has a filesystem inside its environment
●
Barebox is self-updatable
●
Barebox’ environment can be accessed from
Linux’s user-space
●
Robust Auto-Update Controller (RAUC) with signed
bundles exists and works
●
Newer versions of Phytec’s Distro use all this, but
do not support Morty

11. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
New partitioning (schema!)
●
barebox (with built-in-device-tree)
●
barebox-env
●
rescue-initramfs (46M)
●
rescue-dtb (512K)
●
system0/1: kernel (9M), dtb (512K), rootfs (210M)

12. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Reconfiguring barebox
●
Add bootchooser-functionality (default: 3 attempts per
system, then rescue)
●
Add state-functionality to device-tree (stored in
eeprom)
●
Add format and flash-scripts to defaultenv
●
Add boot-scripts to defaultenv
●
Build barebox-target-tools to rootfs
●
barebox_2017.04.0-phy3.bbappend / state-eeprom.dts

13. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
RAUC - bundle
●
Add meta-rauc
– provides “bundle-class”
– provides user-space tools and config-base
●
In your layer add a bundle-file and a .bbappend
for certs and config
●
bitbake <bundle-file-target> meli-bundle.bb

14. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
RAUC – bootchooser - demo

15. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Overview
1- Intro / Starting position
2- Remote update-strategy
3- Remote upgrade-strategy

16. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Blind upgrade strategy
●
How to convert a remote system from a single-
system-installation to dual-installation?
●
Simple:
– Copy new-barebox, kernel, dtb, rootfs to the rootfs (scp)
– Instrument barebox from userspace (bareboxenv)
– Reboot
– Meditation during 3 minutes
– ssh-reverse-tunnel is apprearing

17. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Blind time- 1
●
user-space
– injects a script to barebox which is executed at boot
– reboot
●
in old-barebox:
– script: mount rootfs, update-barebox if image present
– reset

18. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Blind time- 2
●
in new-barebox:
– script mounts rootfs, copies images if present
– copy some specific config-files to backup-dir in barebox-env
– repartitioning
– flash rootfs, kernel, dtb
– resets state, bootchooser
– self-destruction
– reset
●
in new-linux
– first-boot: config-files are extracted from backup-dir barebox-env
– everything works – ssh-tunnel appears

19. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Success
●
All remotely deployed devices have been
upgraded successfully to date

20. 2019-09-24 Remote-update adventures - RAUC, Yocto, Barebox
Oh well
●
Rescue system had no functionality – the idea
was: we’ll get to it later
– 25 systems failed over to rescue system because of
power-supply-variation during kernel boot
●
SSL certificate was valid for one year only. No
certificate infrastructure present for this project
– Update installs a certificate with 100 years validity

21. Questions?
@PatBoeFra
https://github.com/pboettch
https://yai.se/
p@yai.se