Slides of my presentation at the A-mobile 2020 workshop, co-located with the international conference on Automated Software Engineering (ASE 2020). Implementation of the framework: https://github.com/S2-group/android-runner Paper: http://www.ivanomalavolta.com/files/papers/A_Mobile_2020.pdf

1. Ivano Malavolta - https://github.com/S2-group/android-runner
A Framework for the Automatic Execution of
Measurement-based Experiments on Android Devices
Ivano Malavolta, Eoin Grua, Cheng-Yu Lam, Randy de Vries, Franky Tan, Eric
Zielinski, Michael Peters, Luuk Kaandorp

2. Ivano Malavolta - https://github.com/S2-group/android-runner
Run-time measurement of Android apps is difficult
● It requires large boilerplate code
● Measurement tools must be carefully setup
● Various empirical best practices are scattered across the literature
● Experiments can take several hours of sheer execution time
● Existing experimental pipelines are
○ Ad-hoc for one specific experiment
○ Tailored to one specific quality attribute (e.g., energy efficiency)
→ Poor replicability + many research groups reinventing the wheel

3. Ivano Malavolta - https://github.com/S2-group/android-runner
Android Runner
A framework to automatically execute measurement-based
experiments on native and web apps
Android
Runner
Intuition - users define their experiment in a descriptive fashion, and then its
execution is fully automatic, customizable, and replicable
Practitioner
Researcher
Researcher
Conducting empirical studies on Android apps
Developing new run-time profilers for Android devices
Measuring the quality of their apps at run-time

4. Ivano Malavolta - https://github.com/S2-group/android-runner
Design drivers of Android Runner
Experiments executed without any interaction from the researcherAutomation
Incremental
experiments
Customizability
Usability
Profiler
independence
Experiments
replicability
Pause/resume mechanism
Experiments defined in a descriptive fashion
Possibility to add your own business logic or automated testing tool
Either hardware or software profilers, support for multiple profilers
in parallel
Possibility to easily execute an already-performed experiment

5. Ivano Malavolta - https://github.com/S2-group/android-runner
Overview of Android Runner

6. Ivano Malavolta - https://github.com/S2-group/android-runner
Experiment orchestrator - input
Experiment
configuration

7. Ivano Malavolta - https://github.com/S2-group/android-runner
Experiment orchestrator - tasks
1. Perform diagnostic checks
○ User input, e.g., all subjects’ APKs are present
○ Environment, e.g., check if external tools like ADB are properly configured
2. Setup the devices via the Devices manager
3. Bring up the profilers specified in the experiment configuration via the Plugin
handler
4. Setup the plan of the runs via the Progress manager
5. Execute the runs according to the sequence defined in the plan
6. Clean up the devices via the Devices manager
○ e.g., uninstall all subjects in case of native experiment
7. Aggregate the collected measures

8. Ivano Malavolta - https://github.com/S2-group/android-runner
External scripts
Optional Python scripts for executing custom behaviour at
specific points of the experiment
before_experiment Before the first run of the experiment
before_run Before every run of the experiment
after_launch After the current subject is launched, but before measurement starts
interaction During the measurement window (e.g., usage scenarios)
before_close Before the current subject is closed
after_run After every run completes
after_experiment After the last run of the experiment

9. Ivano Malavolta - https://github.com/S2-group/android-runner
Examples of uses of external scripts
● Replay previously recorded usage scenarios (interaction)
○ Python
○ Monkerunner JSON format
● Bring up a local web proxy for recording all traffic generated by the subjects
(before_experiment)
● Programmatically pass a login screen (interaction)
● Throttle the network connection to simulate slower/faster networks

10. Ivano Malavolta - https://github.com/S2-group/android-runner
Devices manager
Layer of abstraction on the low-level operations on the devices
● Diagnostic checks on the ADB connection to each device
● Programmatically enable/disable USB charging
● List/install/uninstall apps on the devices
● Launch/stop/force-terminate processes on the device
● Read/write files on the device
Devices

11. Ivano Malavolta - https://github.com/S2-group/android-runner
Progress manager
● Build the plan of all the runs of the experiment
● Shuffle the plan, if requested by the user
● Provide the plan to the orchestrator
● Persist the current status of the plan during the experiment
ID Subject Device RunIndex
run1 com.facebook.katana lg5x 1
run2 com.twitter.android lg5x 16
... ... ... ...

12. Ivano Malavolta - https://github.com/S2-group/android-runner
Plugin handler
● Profilers as third-party plug-ins
● Extension point for plugin developers exposing:
○ the Devices manager
○ the current subject being executed
○ ...
● Each plugin follows the same lifecycle

13. Ivano Malavolta - https://github.com/S2-group/android-runner
Available plugins
Di Nucci et al. Software-based energy
profiling of android apps: Simple,
efficient and reliable? SANER 2017.
Luis Cruz. Tools and Techniques for
Energy-Efficient Mobile Application
Development. PhD thesis, 2019.
Hecht et al. An empirical study of the
performance impacts of android
code smells. MOBILESoft 2016.

14. Ivano Malavolta - https://github.com/S2-group/android-runner
Outputs
●
○ Produced by the plugins
○ The format is plugin-dependent
■ Higher degree of freedom for plugin developers
■ Usually CSV files
■ The convention is to always save the measures in the file system → higher replicability
○ Based on the logcat Android tool
○ Contains all system messages during all runs
○ Allows future inspections of problematic/interesting runs
Measures
Logs

15. Ivano Malavolta - https://github.com/S2-group/android-runner
Generic benchmarking apps https://github.com/S2-group/
android-apps-benchmark
App Description
Baseline No functionality (idle)
Accelerometer Listens for accelerometer data
Ambient light sensor listens for changes in the ambient light sensor
Camera (x3) Takes a picture from the front camera every 15, 10, or 5 seconds
CPU (x3) Computes a large factorial number every 3, 2, or 1 seconds
Display Plays a video at full brightness
GPS (x3) Listens for location updates every 3, 2, or 1 seconds
Gravity sensor Listens for continuous updates of the gravity sensor
Gyroscope Listens for continuous updates of the gyroscope
Local storage (x3) Writes 1 MB of data to local storage every 3, 2, or 1 seconds
Magnetic field sensor Listens for continuous updates of the magnetic field sensor
Microphone Records audio from microphone
Networking (x3) Sends an HTTP Get requests every 3, 2, or 1 seconds
Room database (x3) Writes 1 MB of data via the Room Android library every 3, 2, or 1 seconds
Speaker Plays audio file at half volume
Each app continuously
stresses the same
component until it is killed
Fully reusable, even
independently of Android
Runner

16. Ivano Malavolta - https://github.com/S2-group/android-runner
Benchmarking Android Runner (excerpt - batterystats)
c

17. Ivano Malavolta - https://github.com/S2-group/android-runner
Benchmarking Android Runner (excerpt - batterystats)
The older device (Nexus 5X) tends to consume more energy
● Not for the camera app
Low coefficient of variation (<5% for 22 over 27 apps)
Camera, display, and GPS (high frequency) tend to consume more energy
All network-related apps tend to have higher variation

18. Ivano Malavolta - https://github.com/S2-group/android-runner
Android Runner as a learning platform
Android
Runner
Green Lab Final projects
● Green Lab = Master course on
empirical software engineering
for energy efficient software
● Students use Android Runner
as black-box tool for their own
experiments
● Students go deeper on
run-time profiling
○ Plugins for new profilers
○ Improve Android Runner
● Community of learners
● Discover and fix bugs
● Android Runner always up to date
● Learn the basics of OSS development

19. Ivano Malavolta - https://github.com/S2-group/android-runner
Conclusions
https://github.com/S2-group/android-runner