This webinar is for software developers and product leaders interested in using open standards to build extensible autonomous or robotic systems.

1. Improving User Experience with Ubiquitous QuickBoot
Jeff Tranter, ICS
Andrew McNaughton, Ubiquitous AI

2. 2
Towards End-User Extensibility in Marine Robotics
Charles Cross
CTO & Co-Founder
charles@missionrobotics.us

3. The Problem
Technology in the field of marine robotics is currently extremely fragmented.
Unlike in many other fields of robotics, such as unmanned aerial and ground
systems, there are few, if any, vehicle platforms, software frameworks, and
commonly used standards upon which to build.
The result?
● Excessive reinventing of the wheel
● “Build it from scratch” or “Settle for less” decision-making
● Higher development costs and barriers to entry
● Products and systems that do not interoperate
● Slower pace of innovation and a 10-20 year lag behind similar fields
3

4. Our Team
Highly cross-functional with extensive experience in marine, aerospace, and ground systems
4
Brian Grau
CEO
Mechanical Engineer and
production specialist who
has been building and
shipping ROV systems
around the world for the
last ten years.
Charles Cross
CTO
Walt Holm
Director of Engineering
Software Engineer
specializing in robotics with
an emphasis on creating
modular, extensible, and
connected systems.
Electrical Engineer with a
diverse background solving
instrumentation challenges
in aerospace, marine, and
medical fields.
John Clauss
Operations
Lifetime of marine
operations experience,
including extensive
archaeological work,
particularly in the Lake
Tahoe area.

5. The Mission
Unify and Expand Marine Technology
Interoperability and extensibility can be achieved through the use of open standards and the creation of electronics and software
solutions that are inherently designed with these goals in mind.
Empower Domain Experts, Fuel Innovation
We want companies and individuals to spend their time answering important questions and solving unique problems, not fighting with
their tools or otherwise having to waste their time and talent solving previously solved problems.
Remove Distance as a Barrier
By leveraging advances in both autonomy and connectivity, we seek to move past the traditional models of one-to-one and
many-to-one operations, instead opting for a future where one operator can direct many systems, even those halfway around the
world.
5

6. ROV and AUV Systems as a Case-Study
ROV (Remotely Operated Vehicle):
● Traditional name within the industry for what you might call a manually piloted underwater drone
● Many size-classes and specifications, from backpackable observation-class to ship-bound work-class
● Within the last 10 years, great strides in the creation of low-cost vehicles (under $10K)
AUV (Autonomous Underwater Vehicle):
● Often long, torpedo-shaped, in form factor
● Typically used for long range surveys
● Generally expensive, with few “low-cost” solutions
6
Riptide™ AUV WHOI’s REMUS AUV
Trident™ Underwater Drone Customized BlueROV™ WHOI’s Jason ROV

7. Past Lessons Learned
Over time, one of the biggest lessons learned working in this space is that there is
no one-size-fits-all solution when it comes to the many tasks that underwater
vehicles perform.
For any particular job you may need:
● A different vehicle type (ROV vs AUV)
● A different thruster configuration/vehicle layout
● Different actuator and sensor payloads
● Tethered vs untethered system
● Operation in turbid vs clear water
● Large carrying/lifting capacity
● Zero to many cameras arranged in various ways
● Hot-swappable power/batteries or power-from-topside
● Etc...
7

8. Who are the Users?
Three customer segments that we are working to serve:
● Developers
● Create novel technology and focus on innovation
● New vehicle concepts
● New sensor, actuator, and communication products
● New software capabilities (computer vision, ML, etc)
● Target specific problem domains (i.e. coral reef restoration, seafloor mapping, etc.)
● System Integrators
● Integrate existing off-the-shelf products into bespoke, ready-to-go vehicles
● Often target specific customers, industries, and geographic regions
● Understand their needs very well!
● Are often not in the business of electronics or software development
● End-Users
● Own and/or use the end vehicle/robot to perform a task or service (inspection, salvage, etc.)
● Likely to extend and upgrade their systems by purchasing COTS products
8

9. “How do I…”
Both in our discussions with customers and potential customers, as well within
popular forums and communities, we hear often hear the same questions posed:
“How do I…”
● … add more cameras?
● … add a new sensor/actuator to the vehicle?
● … create a new sensor/actuator that can be integrated?
● … deploy custom software to the vehicle?
● … modify the pilot application to do or show X?
● … synchronize the recorded data in time?
● … use a different microcontroller/embedded computer?
● … display data on multiple monitors and machines?
● … communicate with the MCU from the app processor?
● … control X programmatically?
9

10. Some Underlying Causes
Some of the reasons these questions come up time and again include…
● Lack of shared comm/API standards across the ecosystem
● Historical constraints and limitations from project origins
● Extensibility and developer friendliness are not core user stories
● Existing solutions are designed around manual 1-to-1 operations
● Communication and power distribution underwater is tough
Unfortunately, the answer users often get back is:
“Everything is open source, so you should be able to read and modify it to suit your needs”
There is no consistent, general approach to adding new capabilities, because the systems are not
modular or flexible enough. Also, though open source, there is a steep learning curve!
10

11. Where does that leave us?
The results of these issues include...
● Vendors creating walled gardens
● “All of our products work together”
● Often black boxed and still provide no path for user development
● Higher cost tier
● Adopters of open source stacks trudging forward
● Hacks and workarounds to extend and improve systems
● Reqs and conventions of non-marine systems a lasting constraint
● Improvements outside the conventions cannot be upstreamed
● Technology becoming fragmented
● Everyone develops in isolation, taping systems together orthogonally
● Ex. Sonar systems communicating over entirely separate lines with different apps
11

12. A Path Forward
We believe that one of the most effective solutions to the aforementioned issues
is to adopt open standards and frameworks that have become common across
the rest of the robotics industry.
12
● DDS (Data Distribution Service)
● An open middleware standard for real-time, publish-subscribe,
peer-to-peer communication.
● ROS2 (Robot Operating System)
● From ros.org: “...a flexible framework for writing robot
software. It is a collection of tools, libraries, and conventions
that aim to simplify the task of creating complex and robust
robot behavior across a wide variety of robotic platforms.”
● Built on top of DDS

13. DDS Quick Summary
● Completely peer-to-peer
● No server/broker required for communication between applications/machines
● Supports dynamic discovery of peers within a system/network (no need to specify endpoints!)
● Topic-oriented communication
● A topic consists of a name and a type description
● Datawriters publish data samples to topics
● Datareaders subscribe to topics
● Datawriters and Datareaders are loosely coupled - Datawriters don’t care who or how many subscribe
● Quality of Service
● Applications must specify not only what data is available, but also how it is to be transmitted
● Datawriters and Datareaders must agree upon Quality of Service (QOS) policy to be used
● QOS policies specify aspects like reliability, historical caching, late-joiner behavior, and more
● Communication fully specified by combination of Data Types, Topics, and QOS
● Software written by two or more parties can interoperate with well-defined behavior
● Encourages modular system design and discourages vendor lock
13

14. DDS Quick Summary
14

15. ROS2 Quick Summary
● Introduces a layer of abstraction around the middleware
● Simplifies development for users
● Reduces DDS-specific learning curve by providing rational defaults and patterns for common use-cases
● Provides utilities, libraries, and data types common to many robotics use-cases
● Scheduling
● Multi-threading
● Data recording and logging
● Coordinate systems and transformations
● Simulation tools
● Common data types (geometry, point clouds, images, GPS, time, common sensor data, and much more)
● Command line utilities for introspection and debugging
● Etc…
● Allows users to package and share their software components
● Nodes - individual applications that can be composed to form complex behaviors and systems
● Libraries - reusable library elements that can be used across ROS projects
● Messages - Data type specifications used between nodes
15

16. Common Robotic System Architecture Example
16
● One or more microcontrollers
○ Often running an RTOS
○ Real-time tasks
○ Flexible I/O interfaces
○ Sensor/actuator control
○ Sub-MCUs
● One primary App Processor
○ Often running Linux
○ Soft real-time tasks
○ Compute intensive tasks
○ High-bandwidth I/O
○ System coordination
● One or more remote C&C devices
○ Laptop, mobile device, etc.
○ Control & monitoring tasks

17. Putting it all Together - The MR Approach
Hardware Development Approach
● Drop-in replacement for commercially available vehicle platforms/ecosystems
● Designed for ROVs and AUVs, but adaptable to other types of systems
● Dramatically improved image quality, latency, and increased camera count
● Improved processing performance (support CV/ML and other intensive tasks)
● Additional space in enclosure for accessories
● Improved overall ease of use; reduced manual labor
● Expanded I/O and flexible power architecture to support add-ons
● Enabling of onboard data and telemetry recording
17

18. Putting it all Together - The MR Approach
Software Development Approach
● Provide a system that works out of the box with little to no configuration
● Support existing components within the ecosystem (sensors, cameras, etc)
● Build out complete OS distribution that takes care of common needs
○ Application scheduling
○ Recording & Playback
○ Software and Firmware Updates
○ Plug-and-play discovery/management of common devices (like cameras)
○ Fault and system resource monitoring
○ Remote access
○ Configuration Management (OS, Network, Vehicle settings, User Software settings, etc)
● Implement all external interfaces and APIs using DDS and ROS2
● Provide an SDK and complete development environment that allows day-1 deployments
● Create a flexible GUI application that supports user extensibility
18

19. Hardware Overview
Features:
● Integrated motherboard
● Recent generation of MCU
● Support for various App Processor models
● New cameras
● Superior low-light performance
● 130ms glass-to-glass latency standard config
● 70ms glass-to-glass ultralow-latency mode
● High-speed onboard storage
● Additional space for user hardware
● Additional connectorized I/O interfaces
● Flexible power management architecture
● 3.3V, 5V, and 12V buses
● Switchable rails for failsafing/load-shedding
● RTC for tracking real-world time
● Low-power warning system
19

20. 20

21. 21

22. 22

23. Software Overview
Features:
● Docker-based embedded Linux platform
● Read-only Host OS running application containers
● Main container running Mission Robotics OS
● Additional containers dedicated to private User software
● OTA updates and remote fleet management capabilities
● DDS and ROS2 communication
● Vehicle and Client software implement both standalone DDS and ROS2 APIs
● mrSDK* available to simplify and accelerate development of user applications (both vehicle and client)
● Board/driver support for latest Nvidia platforms, designed for autonomous vehicles
● Hardware acceleration, computer vision, and machine learning features
● Complete set of services providing standard functionality
● Can be disabled by users if they wish to implement their own
23
*placeholder name

24. Developer Tools
24
● Docker based dev & deployment environment
○ Simplifies building, testing, and deploying software
○ Isolation of components from different vendors
● CI/CD
○ Full build, test, and deploy pipeline ready on day one
● CLI tools
● Example codebase
○ Python & C++ DDS & ROS2 starter applications
○ Arduino & STM32F example code for MCU apps
● Remote vehicle connectivity and monitoring

25. 25
MicroROS / DDS XRCE
ROS2 / DDS

26. Options for Extensibility
26
There are five key paths to extend the system:
● Certain plug-and-play features are already built-in
● V4L2 and other supported cameras are automatically detected and brought up (configurable)
● MCUs that implement supported protocols are automatically bridged to the DDS bus
● Explicitly supported devices with mrOS* drivers automatically brought up
● Create and deploy custom ROS2/DDS applications to the vehicle user containers
● Or any applications/protocols, if desired
● Create and deploy custom ROS2/DDS topside applications that interact via the public API
● Create plugins for the Qt piloting application
● Create standalone MCU + sensor/actuator devices that are tunneled to both the DDS bus and Qt app
*placeholder name

27. “How do I…”
“… Add more cameras?”
Two main approaches:
● If the camera is already supported by mrOS, plug it in and configure if needed
● Camera Supervisor will automatically bring driver up for camera
● Camera data will be made available over ROS2/DDS
● Qt App will automatically discover new camera and allow streaming/control
● If the camera is not already supported, user can do one of two things:
● Write a plugin library for the Camera Supervisor to allow it to bring the camera up
● Must implement way to find/access the device, acquire frames, change settings
● Write a standalone ROS2/DDS application that handles the camera on their own
● By following the established type/topic API conventions, it should just work
27

28. 28

29. “How do I…”
“... create a new sensor/actuator that can be integrated?”
If the device can be controlled using I/O available on App Processor, write a new ROS2/DDS node.
If more I/O is required, user can use an additional MCU, such as an Arduino:
● Write an Arduino sketch that can interact with the sensor/actuator
● Add some code that implements the public API over the serial interface
● We provide a ready to go library and examples to make this easy
● Two API paths:
● Use MicroROS/DDS XRCE to implement API directly on the existing databus
● Use a simplified JSON-based API that will be forwarded directly to the Qt app
As an example, let’s consider the user adding a custom light module via Arduino...
29

30. 30
● Arduino sketch sends messages to App Processor
○ Messages conform to a simple JSON-based API
■ Contains unique message name, fields, field types, and field values
● MCU supervisor running on App Processor detects user MCU
○ A bridge service is created for each user MCU
● Bridge directly forwards the messages to Qt App on special DDS topic
○ Also creates an “input” topic for messages to travel back to user’s MCU

31. “How do I…”
“... modify the pilot application to do or show X?”
● Introspect the fields and convert them into a dynamic model representing the “backend” for the device
● Backend contains a property map for input values and output values
● Backend provides hints about the data it contains to choose an appropriate display widget
● Signals emitted upon value changes received
● Use the model as a data source/sink for frontend elements
● User can configure the Pilot activity or any customizable screen to adjust the layout
● The new data sources/sinks will be shown, along with “suggested display elements” based on hints
● App is pre-packaged with a number of common components: labels, dials, sliders, buttons, etc.
● Upon instantiating the display component based on configuration, inputs and outputs will be connected
31

32. Custom User Components
While the app comes pre-packaged with common components, we want to
provide a way for the user to create an entirely new widget/component that can be
created during runtime without the need to compile anything and without any prior
knowledge of the user’s data types built into the app.
● User creates a plugin containing:
● One or more QML file that encompasses a “Component”
● A metadata file that describes the signals/slots of the component and any accepted hints
● This plugin is loaded via a utility in the Qt app
● Or simply place the plugin folder into a specified location where the app will find it
● The new component is made available within the layout config
● At this point, user can drop the new component into various screens
● The signals/slots will be automatically connected, allowing real-time bidirectional updates
32

33. Resources
● ROS2 Resources and Documentation
● https://docs.ros.org/en/galactic/
● https://micro.ros.org/
● DDS Tutorials and Learning Materials
● http://www.laas.fr/files/SLides-A_Corsaro.pdf
● https://www.dre.vanderbilt.edu/~schmidt/PDF/dds-sos.pdf
● https://www.youtube.com/watch?v=GGqcrccWfeE
● ROS2, micro-ROS, and DDS in aerial drone systems (PX4/Dronecode)
● https://www.youtube.com/watch?v=lZ8crGI16qA
33

34. How to Add Actuator/Sensor Data and Associated
UI to a Closed-Source Qt/QML App?
34
Qt App
“Ground Station”
(closed source)
Actuators
Sensors
Vehicle
QML User Interface
User-Defined Controls
User-Defined
Environment
Telemetry
Commands

35. What is Qt?
35
Application development framework (not just user interface)
Two user interface frameworks: widgets (desktop) and QML (touch)
Open Source
Cross platform (Windows, Linux, Mac, Android, iOS...)
“Write once, compile anywhere”
Triple licenced: GPL, LGPL, and commercial (recommended)
C++ but other language bindings exist e.g. Python
Actively developed by The Qt Company with services provided by ICS (ics.com)

36. Ways to Extend a Closed-Source Qt App
QPluginLoader
QJSEngine (formerly Qt Script)
Dynamic QML
Loader
Binding
Connections
QQmlPropertyMap (dynamic properties)
36

37. Loader Loads QML User Interface at Run Time
37

38. Defining C++ Properties for QML at Compile Time
38

39. PROPERTY Super Macro
39

40. QQmlPropertyMap Defines Properties at Run Time
40

41. Sample App
41

42. Object.h
42

43. Object.cpp
43

44. main.cpp
44

45. Main.qml
45

46. qml/Boolean.qml
46

47. qml/Number.qml
47

48. Thank you!
48
Any questions?