Introducing resinOS: An Operating System Tailored for Containers and Built for the Embedded World
•Download as PPTX, PDF•
5 likes•17,909 views
This presentation, from the Embedded Linux Conference Europe in October 2016, discusses how resinOS was built, highlights some of its key features, and shares a roadmap for future development and contribution. resinOS is the latest open-source tool built by resin.io to enable the future of hardware with the tools of modern software. resinOS is a simple yet powerful operating system that brings standard Docker containers to embedded devices and works on a wide variety of device types and architectures. resinOS was born from the team’s experience deploying embedded containers across device types and has been battle-tested in production environments. You can download resinOS at https://resinos.io
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Introducing resinOS: An Operating System Tailored for Containers and Built for the Embedded World
Similar to Introducing resinOS: An Operating System Tailored for Containers and Built for the Embedded World (20)
More from Balena
More from Balena (8)
Recently uploaded
Recently uploaded (20)
Introducing resinOS: An Operating System Tailored for Containers and Built for the Embedded World
- 1. Introducing resinOS An Operating System Tailored for Containers and Built for the Embedded WorldAndrei Gherzan / Petros Angelatos October 2016
- 2. Andrei Gherzan ● Lead engineer of resinOS ● Maintainer of meta-raspberrypi, meta-chip About us Petros Angelatos Founder / CTO Ported Docker to ARM
- 4. Be the embedded OS of choice for containers in IoT Create a community around containers for IoT Modern security features Minimal footprint Production ready Mission
- 5. Started 4 years ago Modern devops practices to the embedded world Naturally leaned towards containers Ported Docker to ARMv6 Ported Docker to ARMv5 Fixes upstreamed History - resin.io
- 6. Needed an OS for our platform Tried a modified Arch Tried a modified TinyCore Both had important shortcomings History - resinOS
- 7. Started in January 2014 as internal project Used Yocto as a base Open sourced in July 2015 Currently under very active development It’s been running in production for 2.5 years History - resinOS
- 8. Architecture
- 9. Why we chose yocto Minimal Low footprint Build system allows for easy patching Board vendors usually supply Yocto BSP Easier device support Yocto
- 10. Yocto layer architecture poky meta-oe meta-resin resin-<board-name> board BSP submodules One repo per board Submodules for dependent layers Tried repo tool Tried monolithic repo Each board can move independently https://github.com/resin-os?query=resin-
- 11. meta-resin meta-resin meta-resin-common Jethro overlayer Fido overlayer Daisy overlayer Main resinOS layer Automatic aufs patching BSP independent kernel configuration Can prepopulate docker images Kernel headers for out-of-tree module development https://github.com/resin-os/meta-resin
- 12. Environment defined in a Dockerfile Predictable host configuration Docker image artifacts You can use the OS as a container resin/resinos:<version>-<board> Build system https://github.com/resin-os/resin-yocto-scripts
- 13. Separate rootfs and root state We know exactly which services write to disk Dual root partition data partition auto-expands on first boot Partition layout rootA databoot rootB state
- 14. Forced us to investigate all writes Configuration stored in state partition Network configuration Random seed Clock at shutdown Some state is stored in tmpfs DHCP leases Limited logs Read-only root
- 15. Cleaner separation OTA updates are much easier Enables diff based updates We can’t leave state behind Read-only root
- 16. Compartmentalisation of failures Device can survive data partition corruption Most I/O activity happens in there Root partition is never written to while in use We strive to do atomic operations everywhere Reliability
- 17. Runtime ResinOS Userspace User Application Language Packages Language Runtime OS packages Base Image Container Engine (Docker) Linux Kernel + Kernel Modules CONTAINER CONTAINER(S)
- 19. Leverage a lot of systemd features Adjusting OOM score for critical services Running services in separate mount namespaces Very easy dependency management NTP Socket activation for SSH Saves RAM since ssh is running only when needed Systemd
- 20. DNS is hard dnsmasq Integration of Docker with host’s dnsmasq NetworkManager Excellent D-Bus API ModemManager Excellent D-Bus API Lots of documentation Networking
- 21. AUFS driver Allows support for NAND based devices Currently on docker 1.10.3 Backported stability patches Journald logging driver Avoids SD card wear Seccomp enabled Docker
- 22. All logs end up in journald In RAM 8MB buffer by default Configurable log persistence Journald allows for structured logs Container logs are annotated with metadata Easy to send logs to a central location to store and process Log management
- 23. Features
- 24. Some boards have internal storage Image for these boards is a flasher Automatic copying to internal storage Feedback through LEDs Two stage flashing
- 25. So many options It’s one of our biggest focus areas resinhup is our current approach Takes advantage of dual root partition Validates everything before changing the state It’s still experimental Host OS updates https://github.com/resin-os/resinhup/
- 26. ● Used by CoreOS, ChromiumOS, Ubuntu Snappy Brillo, Mender.io But wastes a lot of space We’re experimenting with more advanced approaches ostree docker Dual root partition method
- 27. Integration with docker It uses docker to pull the OS image It then unpacks and applies it Leveraging important docker features Signed images Programmatic API for fetching Open question: can unify containers and host? ResinHUP https://github.com/resin-os/resinhup/
- 28. Automatic emulated testing ● We support virtual QEMU boards ● Automated basic testing on every PR ○ Booting ○ Networking ● Integrated with our Jenkins https://github.com/resin-io/autohat
- 29. Automatic hardware testing ● Manual testing doesn’t scale ○ Currently 22 boards ● We built a board that instruments boards ○ GPIO ○ Provisioning ○ SD muxing ○ Wifi testing https://github.com/resin-io/autohat-rig
- 30. ARM64 ● Coming soon ARMv6 ● RPI Zero ● RPI model 1 A+ ARMv5 ● TS7700 Device support ARMv7 ● Raspberry Pi 2 ● Raspberry Pi 3 ● Samsung Artik 5 ● SamsungArtik 10 ● Beaglebone Black ● Beaglebone Green ● Beaglebone Green Wireless ● Odroid C1/C1+ ● Odroid XU4 ● SolidRun Hummingboard i2 ● Boundary Devices Nitrogen6x ● Parallella Board ● VIA 820 board ● Zynq zc702 ● TS4900 single and Quad X86_32 ● Intel Edison X86_64 ● Intel NUC
- 31. Device support ● Easy to add new boards ● Meta-resin handles ○ Userspace ○ Image generation ○ Kernel configuration
- 33. ● How do you.. ○ Configure network credentials? ○ Provision a device? ○ Develop on the board? ○ Get logs? Development tools
- 34. ● Development images have ○ Open SSH server ○ Docker socket exposed over TCP ○ mDNS exposed metadata ● Device is at <hostname>.local Development mode
- 35. ● Image configuration ● Wifi credentials ● Hostname ● Persistent logging Resin Device Toolbox $ rdt configure ~/Downloads/resinos-dev.img ? Network SSID super_wifi ? Network Key super_secure_password ? Do you want to set advanced settings? Yes ? Device Hostname resin ? Do you want to enable persistent logging? no Done!
- 36. ● Automatically detects removable storage ● Won’t wipe your drive! ● Validates after writing Resin Device Toolbox $ sudo rdt flash ~/Downloads/resinos-dev.img ? Select drive /dev/disk3 (7.9 GB) - STORAGE DEVICE ? This will erase the selected drive. Are you sure? Yes Flashing [========================] 100% eta 0s Validating [========================] 100% eta 0s
- 37. Docker development Finds device in local network Continously syncs code into the container Rebuilds when necessary Resin Device Toolbox $ rdt push --source . * Building.. - Stopping and Removing any previous 'myapp' container - Removing any existing container images for 'myapp' - Building new 'myapp' image
- 38. ● More than 500 images for each supported device type ● Debian, Fedora, Alpine ● Nodejs, python, golang, Java ● Follow docker conventions Base Images https://github.com/resin-io-library/base-images
- 39. Future
- 40. ● Roadmap includes.. ○ Compressed RAM ○ Docker 1.12 ○ Hardware watchdog integration ○ Secure Boot ○ ramoops integration ○ ... ● We interested in your thoughts ● There is lots of room for innovation Future
- 41. Open source ● Website - https://resinos.io/ ● Github - https://github.com/resin-os ● Gitter - https://gitter.im/resin-os/chat ● Apache 2 Licence
- 42. Questions?
Editor's Notes
- Ποιοι είμαστε Γιατί κάνουμε το resin Πως δουλεύει το resin
- We will upload the slides The links next to the github logo link to the repo for what is mentioned in the slide
- We believe containers give some unique capabilities when applied to IoT Developers use very familiar tools They don’t have to reboot for every iteration If their container completely crashes the OS is still there to recover But also have some unique challenges A container in the cloud interacts with its environment by requiring computation, memory, networking, storage All of these are very well abstracted and have no different if they are virtual or not You can’t have a virtual GPIO Containers in IoT have to interface with hardware sometimes Devices can be connected with to very poor networks or via 3G, bandwidth matters We’re also aiming at making the OS developer friendly and providing the tools that allow you to build great applications We’ve made a lot of progress during the past years and now we’re making an official release as a standalone project
- We were managing a fleet of 200 devices across London Most of the tooling felt primitive. It was a hard problem We wanted to bring more modern tools to the embedded world Bridge yocto world with docker/cloud world Started with `git push` workflow, only supported nodejs projects
- Most distributions are not optimised for remote management and embedded devices We had a very early prototype which sometimes it would start being slow Turned out filesystem indexing was kicking off in the background We had to manually build tooling around creating images, installing software Most distributions support a limited set of architectures
- resinOS is the result of our efforts to have a reliable system in production Has been running on thousands of devices We moved the development to the open a bit more than a year ago We’re now making it a proper open source project
- Yocto had drawbacks, namely steep learning curve, resource-hungry builds Benefits outweight them by far We can disable compile-time features that we don’t use Very big community in the embedded space Provides the right tools for distribution maintainer
- For each board we support there is a GH repo with a well-defined structure We manage yocto dependencies by using git submodules We can move each board independently through versions without blocking the development of meta-resin Normally each such repo has no code, but sometimes some glue code is put there in a non-submoduled layer
- This is where the bulk of the work is happening We need AUFS for docker - we’ll talk about it later on why we chose this backend Meta-resin auto-detects if the kernel provided by the bsp has aufs support (like beaglebone) and automatically applies patches if not Meta-resin automatically extends the kernel config provided by the BSP to enable the right features for container to run Prepopulating docker images means easier mass-production/provisioning
- We use a docker container for the build env yocto runs in Well defined versions of host dependencies One of the artifacts of the build process is a docker image containing the userspace and boot files We are actually pushing this image in dockerhub for every release This can be used as a base for a container or for testing It will also become relevant in the talk when we talk about OS updates
- Meta-resin creates this layout by default Data partition is where all containers and container state is kept State partition is where partition state is kept We generally try to keep the boot partition as a FAT partition This allows our development tools to read and write to it cross-platform
- We chose to have a read-only root partition to make failures much less likely During board boot we bind-mount a lot of relevant directories from the state partition to the appropriate locations Another approach, used by OpenWRT, would be having a read-only root (squashfs in their case) and using overlayfs on top. We didn’t want to be dependent on a union filesystem for the rootfs Some POSIX semantics are lost You can still get random mutations of the rootfs that will bite you later when you update
- The goal of the OS is to be fully updatable but also flexible. If all the application code was living in the host then a complex host OS update would be required for every change. This is where containers come in. Our userspace is designed to do the absolute necessary things to bring up the board and connect to the internet Using this distinction we can have very fast reboot-less updates for containers But we can still update the host with traditional methods but much more infrequently
- The initial versions of our OS were based on sysvinit But we quickly found ourselves having a hard time managing dependencies Sometimes restarting a service wouldn’t kill all its processes Each program was managing its own logs We plan to leverage it a lot more in the future
- Networking outside the data center is a wild place Glibc only supports max 3 nameservers In some networks DHCP dns is broken In some networks google DNS is broken In some networks DNS server take forever to reply In some networks DNS servers reply with the wrong data D-Bus can be easily exposed to containers Have also tried connman We had to patch it to convince it to not do NTP Documentation buried into repo and examples in test suite We didn’t have a good way of persisting a config file describing 3G connections ModemManager gets us that In general much more developer friendly
- Overlay needs 3.18+ and uses a lot of inodes Overlay2 needs 4.0+ BTRFS is not stable enough, especially for low storage devices Devicemapper doesn’t work on NAND Zfs… yeah Docker’s default logging driver will keep appending your stdout into a very big JSON file on disk We configure docker to send logs to journald Meta-resin also enables seccomp in the kernel and docker uses a default profile for all containers
- We don’t allow anything to write its own logs One can easily send the stream of journald logs to a remote server If configured, ResinOS can store logs in the
- Yocto by default produces the host OS image but sometimes this needs to be written to internal storage Dependening on the board this might involve a few steps ResinOS can create a “flasher” type image that copies itself into internal storage and shuts down the board
- There are so many options for OS updates. I think there are 4 talks on this subject in this ELCE Resinhup is how are addressing our current production needs It’s still experimental and tested in a very carefully selected set of customers
- Host OS updates is a very hot topic and is definitely
- One important feature we added recently is integration with docker Pulling down an fs image using a tarball felt wrong when we had docker Currently we’re using docker only for pulling an image and then copy it to the rootfs Docker has a lot of nice features that can be leveraged. Signed images for example
- Automate all possible ways of booting a board ( SD card / DFU / Network booting? ). * Emulate the peripherals that could be connected to the SBC ( GPIO/SPI/I2C/Bluetooth). * All external hardware used to run the tests should be on USB to enable for portability of the entire rig. * Simulate and test different network conditions under which a SBC operates. * Accompanying software components should have an easy DSL/way of writing and extending tests, so that peripheral abstractions can be re-used. * Accompanying software component should enable Continuous Integration with Jenkins * All custom hardware designs should be in KiCad to allow for easy contribution as the project is open source.
- Automate all possible ways of booting a board ( SD card / DFU / Network booting? ). * Emulate the peripherals that could be connected to the SBC ( GPIO/SPI/I2C/Bluetooth). * All external hardware used to run the tests should be on USB to enable for portability of the entire rig. * Simulate and test different network conditions under which a SBC operates. * Accompanying software components should have an easy DSL/way of writing and extending tests, so that peripheral abstractions can be re-used. * Accompanying software component should enable Continuous Integration with Jenkins * All custom hardware designs should be in KiCad to allow for easy contribution as the project is open source.
- 20 boards
- Normally you have to connect with ethernet/serial cable Configuration is tricky Flashing
- All of our code is open source and our goal is to build a community around this project We’ll be more than happy to chat with you on our gitter channel
- Meetup info