Presentation at 16th IEEE International Conference on Software Testing, Verification and Validation (ICST): An Empirical Characterization of Software Bugs in Open-Source Cyber-Physical Systems. Journal of Systems & Software (JSS).

1. J1 presentation:
“An Empirical Characterization of Software Bugs in Open-Source
Cyber-Physical Systems” - Journal of Systems & Software (JSS)
17 April, 2023 - Ireland - ICST 2023
Sebastiano Panichella
Zurich University of Applied Sciences
https://spanichella.github.io/
ICST 2023:
16th IEEE International Conference on Software
Testing, Veri
fi
cation and Validation (ICST)
Fiorella Zampetti Ritu Kapur Massimiliano Di Penta
University of Sannio

2. Outline
2
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.

3. “Emerging Cyber-physical Systems (CPS) will play a crucial role in the quality of
life of European citizens and the future of the European economy”
Context
• CPS relevant sectors:
• Healthcare
• Automotive
• Water Monitoring
• Railway
• Manufacturing
• Avionics
• etc.
MEDICAL DELIVERY
FOOD DELIVERY
• Avionics
3

4. FOOD DELIVERY

5. Background
First aerodynamic
fl
ight on another planet. Landed with Perseverance rover on 18 February 2021
SPACE EXPLORATION

6. 6
UAVs
“But do we have, today UAVs, that would autonomously
map the disaster area at the Fukushima nuclear power
plant or spot the location of people stranded and isolated
after such disaster?”
Fukushima disaster
Unmanned Aerial Vehicles (UAVs) - a specific case of “CPSs”
Problem Statement (1)

7. • -
• Our (Software Engineering) view of DevOps and AI for IoT systems:
• DevOps and Continuous Delivery (CD): Whats is it?
• Present, Challenges, and Opportunities
• Relevant Research Questions
• Arti
fi
cial Intelligence (AI) and Testing Automation:
• Present, Challenges, and Opportunities
• User-oriented Testing Automation
• Relevant Research Questions
“We all recognize the relevance and capacity of contemporary cyber-
physical systems for building the future of our society, but ongoing research
in the
fi
eld is also clearly failing in making the right countermeasures to
avoid that CPS usage a
ff
ects human being safety”. In
“Self-driving Uber kills Arizona
woman in first fatal crash involving
pedestrian”
“Swiss Post drone
crashes in Zurich
Problem Statement (2)
“A simple software update was
the direct cause of the fatal
crashes of the Boeing 737”
7

8. • -
• Our (Software Engineering) view of DevOps and AI for IoT systems:
• DevOps and Continuous Delivery (CD): Whats is it?
• Present, Challenges, and Opportunities
• Relevant Research Questions
• Arti
fi
cial Intelligence (AI) and Testing Automation:
• Present, Challenges, and Opportunities
• User-oriented Testing Automation
• Relevant Research Questions
“Self-driving Uber kills Arizona
woman in first fatal crash involving
pedestrian”
Challenge
“A simple software update was
the direct cause of the fatal
crashes of the Boeing 737”
Observability, testability, and predictability of the behavior of emerging
CPS is highly limited and, unfortunately, their usage in the real world can
lead to fatal crashes sometimes tragically involving also humans
8

9. Sebastiano Panichella Sajad Khatiri
Christian Birchler
COSMOS:
DevOps for Complex Cyber-physical Systems
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS https://lnkd.in/eUVeaYaz

10. 10
What types of bugs occur in open-source CPSs?
CPS
Bugs

11. 11
Famous Software Failure in a CPS: The Case of Ariane 5

12. 12
Bugs and Failures in CPS
CPS Bug is
“a
fl
aw in the hardware (not properly
handled by the software), or an incorrect
interaction between the software and
hardware components leading to a CPS
misbehavior’’
A CPS bug can manifest as a CPS failure,
which makes a CPS unable to deliver its
required functionality or not ful
fi
lling some
non-functional properties
Properties

13. 13
Bugs in the PX4 Project
https://github.com/PX4/PX4-Autopilot/issues/8980
Px4 Issue 8980: Unsuccessful
fl
ight
“ Autopilot receiving noisy sensor-data…“

14. 14
Bugs in the OpenPilot Project
Openpilot Issue 2103: A CAN bus error
https://github.com/commaai/openpilot/issues/2103
“ Software update on unsupported hardware devices…“

15. 15
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta,
Sebastiano Panichella: An Empirical Characterization
of Software Bugs in Open-Source Cyber-Physical
Systems. Journal of Systems & Software (JSS).
What types of bugs occur in Open-source CPSs?
CPS
Bugs
1,151 closed issues sampled from
14 open-source CPS projects
CPS bugs taxonomy comprises
8 di
ff
erent high-level categories

16. 16
What types of bugs occur in Open-source CPSs?
Process for designing a taxonomy of bugs occurring in CPSs
from GitHub (Arduino, drones, robotics, automotive, etc.)

17. 17
Analyzed Projects
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta, Sebastiano Panichella: An Empirical Characterization of Software Bugs in Open-Source Cyber-Physical Systems. Journal of Systems & Software (JSS).
14 open-source CPS projects CPS Domains Issues
Closed
Issues
Bug-related
Issues

18. 18
What types of bugs occur in Open-source CPSs?
33% of the
bugs are
CPS-speci
fi
c
7
6
5
4
3
2
1
8
Grouped into
8 high-level categories
CPS Bug taxonomy:
22 different root causes

19. 19
Hardware Bugs in Open-source CPSs
Energy Faulty Sensors
Hardware failure
Hardware
Not Supported/Compatible
1
Memory

20. 20
Hardware Bugs in Open-source CPSs
Energy
Faulty Sensors
Hardware failure
1
Bug #21033 in openpilot points out the
presence of a CAN bus error on a
speci
fi
c device (i.e., Rav4 Prime).
Main Findings:
Hardware-speci
fi
c bugs are peculiar to our taxonomy, and, unsurprisingly, all of
them are CPS-speci
fi
c.
Recognizing (and simulating) hardware failures has paramount importance in V&V.
Also, developers should take particular care of hardware compatibility, especially
for CPSs targeting multiple devices.
The interaction with the hardware makes particularly crucial the analysis of non-
functional properties such as performance, memory, and energy consumption.

21. 21
Network & Interface Bugs in Open-source CPSs

22. 22
Network & Interface Bugs in Open-source CPSs
Bug #4302 in Arduino, where there is a memory leak while
doing repeated connections to a server, causing the loss of around
8KB for each connection.
Bug #6546 in PX4-AutoPilot that has been inherited from the third-
party library being used while interfacing with GPS (dealing with GPS
‘‘jamming’’ that has already been reported as an issue in the library aimed at supporting
the Intel Aero Platform)

23. 23
Network & Interface Bugs in Open-source CPSs
Main Findings:
Networking plays a paramount role for CPSs
and can be the origin of bugs.
The CPS infrastructure should include
network monitors and V&V techniques may
contemplate CPS misbehavior caused by
network-speci
fi
c aspects.
Main Findings:
Interfacing bugs are challenging for
developers coping with CPSs, and testing
e
ff
orts should focus on them.

24. 24
What about Safety Bugs of CPSs?
CPS
Bugs

25. 25
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?
CPS Safety Related Issues of UAVs
Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano
Panichella: Automated Identification and Qualitative Characterization of Safety Concerns Reported in
UAV Software Platforms. Transactions on Software Engineering and Methodology.

26. 26
Co-occurrences
of hazard
categories
and accident
categories
Hazard Accident
Hazard categories and
corresponding occurrences in our
dataset of 273 safety-related
issues and pull requests.
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?

27. 27
DevOps Challenges for Dealing with CPS Bugs and Complexity
Interview-based methodology
Interviews’
transcripts
Card Sorting
Early feedback from
COSMOS partners
Bad (and good)
practices,
Challenges,
Barriers,
Mitigation
Analysis Triangulation
Validation outside COSMOS
(survey questionnaire)
Pull Requests (PRs) Mining
20 CPS related projects

28. 28
DevOps Challenges for Dealing with CPS Bugs and Complexity
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.
Finding Overview:

29. 29
DevOps Challenges for Dealing with CPS Bugs and Complexity
Finding Overview:
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

30. 30
DevOps Challenges for Dealing with CPS Bugs and Complexity
Finding Overview:
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

31. 31
Challenges vs. Mitigation Strategies
Mitigation Stategies Mitigation Stategies
Challenges Challenges

32. Pipeline analysis tools
Examples of rules:
Missing stable branch
Arbitrary skip of steps/stages
Build time aging
Inappropriate cache handling
Different outcome on different
targets
Build stages not properly ordered

33. Example: Different outcome on different targets
Consistently different build outcome on different targets
linux-aarch64
linux-x86_64
macos-x86_64
macos-aarch64
windows-x86_64
✘
✔︎✔︎✔︎✔︎
commit1
linux-aarch64
linux-x86_64
macos-x86_64
macos-aarch64
windows-x86_64
✘
✔︎✔︎✔︎✔︎
commit2
……
linux-aarch64
linux-x86_64
macos-x86_64
macos-aarch64
windows-x86_64
✘
✔︎✔︎✔︎✔︎
commitN
✘ ✘ ✘
✘
✔
✔
Targets

34. Summary
34
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.

35. Thanks for the Attention!
• Any Questions?
J1 presentation:
“An Empirical Characterization of Software
Bugs in Open-Source Cyber-Physical Systems”
- Journal of Systems & Software (JSS)
20 April, 2023 - Ireland - ICST 2023
Sebastiano Panichella
Zurich University of Applied Sciences
https://spanichella.github.io/