The document discusses the Telemetry performance testing framework in Chrome. It provides an overview of Telemetry and its goals. The methodology section describes running a smoothness benchmark case as an example. Key concepts in Telemetry like benchmarks, measurements, page sets and options are explained. The document details how these concepts are connected in code and outlines the overall workflow when running a benchmark, from starting the browser to collecting and outputting results. It describes where results are located and how the performance metrics in results are generated and interpreted.
3. Overview
What is Telemetry?
Telemetry is Chrome's performance testing framework. It allows you to perform arbitrary actions on a set of web
pages and report metrics about it. - http://www.chromium.org/developers/telemetry
Supported Platform
Target: ChromeOS (did not try Android though the official doc mentioned)
Host: Linux (did not try others though the official doc mentioned)
Goal
Get familiar with code layout and structures
Understand the control and data flow about Telemetry performance test framework
Know how the result is collected, calculated and reported
Non-goal
WPR
Dev Tools Remote Debugging Protocol
Every thing(class) else not in example case
Write a new case – will cover in another slides, soon…
3
17. page_runner
Setup result according to --
output-format
• csv_page_measurement_results.py
Find browser according to --
browser
• browser_finder to get suitable browser
executable in target
• cros_interface.py to ssh, find
‘/opt/google/chrome/chrome’
Prepare resource according to
page set
• Check page set present
• Check wpr options
• Check page achieve present, download if
version change
Start WPR server
• Webpagereplay.py
• Get wpr server port #
Setup ssh reverse tunnel for
target http(s) request/response
• cros_forwarder.py
Start browser
• ssh with dbus-send command, options: --
no-proxy-server,--host-resolver-
rules=MAP * 127.0.0.1,EXCLUDE
localhost,--testing-fixed-http-port=59219,-
-testing-fixed-https-port=59220, --
remote-debugging-port=59221
Setup ssh forward tunnel for
remote debug port
• -L47259:127.0.0.1:59221
Run measurement for page
• Measurement start/end will call
smoothness_controller.py to start/stop
browser trace
• Start/stop was send via websocket
through debug port
• Browser trace will collected about 5
seconds
• inspector_backend will help to execute
simple javascript to generate start/stop
marker
When page run complete(in this
case, 5s), compute the
performance data
• fps
• Janks
17
18. Results:
- Where is it?
- What is the output?
- How does the data generated?
- How to interpret those data?
19. Where is it? What is the output?
In our example, the result simply displayed in stdout. You can
specify –o option to send output to designate file
The output looks as following
19
page_name
frame_times
(ms)
jank (ms)
mean_frame_time
(ms)
mean_pixels_appro
ximated (percent)
mostly_smooth
(score)
http://mudcu.be/labs/JS1k/
BreathingGalaxies.html
17.26396552 149.406 17.264- -
http://runway.countlesspro
jects.com/prototype/perfor
mance_test.html
40.03739837 2732.9738 40.037- -
http://ie.microsoft.com/te
stdrive/Performance/FishIE
Tank/Default.html
16.6890301 34.4974 16.689- -
http://ie.microsoft.com/te
stdrive/Performance/SpeedR
eading/Default.html
31.9674359 646.008 31.967- -
20. How does the data generated?
Raw data send from Chrome browser via debug port
Processed by
telemetry/telemetry/web_perf/metrics/smoothness.py
frame_times: arithmetic mean of frame_time sequence
Jank: discrepancy of frame_times sequence
mean_frame_time: arithmetic mean of frame_time sequence, round 3
mean_pixels_approximated: not available
mostly_smooth: if 95% frame time < 19ms (1000ms/60) then 1, otherwise 0
20
21. How to interpret those data?
frame_times/ mean_frame_time
1/FPS
jank
smoothness
score
quality
21
23. Questions
What is the overhead on *target* if using telemetry?
What is the memory consumption on *host*, over time?
Can a benchmark run with multiple measurement? Why?
How many metrics currently supported on ChromeOS?
Do we try all of them?
WPR package contains multiple application data, how to
update data? Update one application will affect others?
…