We have developed technology that allows a wide range of graphical interfaces to be streamed efficiently over wide area networks. This is an enabling technology that enables remote graphics akin to the way MPEG compression enables video streaming.
3. Exporting Graphics
Graphics can be exported from any of the four
levels of the graphics stack
– Application level
– Toolkit level
– Rendering level
– Bitmap level
Ascender Technology Ltd Remote Rendering
5. Application Level (1)
This is the normal way Linux/Android/Iphone
runs apps.
The application itself is exported and run locally.
Normally applications either come installed with
the system or are added later (e.g. Android
Market).
Applications can run locally without using any
network bandwidth.
Ascender Technology Ltd Remote Rendering
6. Bitmap Level (4)
Common approach to exporting graphics (VNC,
Chromoting).
Works well for any mix of graphics applications.
Simple to implement.
Heavy user of network bandwidth.
Amenable to compression.
Computationally intensive.
Ascender Technology Ltd Remote Rendering
7. Toolkit Level (2)
It is technically very complex. Android, to date,
has 15 different toolkit API variants.
Every application can extend the toolkit with
custom widgets (subclasses of android.view.View).
Clearly impossible.
Ascender Technology Ltd Remote Rendering
8. Toolkit Level (2)
It is technically very complex. Android, to date,
has 15 different toolkit API variants.
Every application can extend the toolkit with
custom widgets (subclasses of android.view.View).
Clearly impossible.
This talk will show that effectively exporting
graphics at the toolkit level and even the
application level is in fact possible.
Ascender Technology Ltd Remote Rendering
9. Android Rendering Level
This work exports graphics at the rendering level.
In Android there are a number of rendering
interfaces that can be used:
– Skia graphics
– OpenGL ES 1.1 or OpenGL ES 2.0
– Android.view.View
We shall see that in effect toolkit and application
level graphics can be exported.
Ascender Technology Ltd Remote Rendering
12. Software Rendering Level (Skia)
The rendering level is the graphics layer that
actually “colors” the pixels in the bitmap.
The Skia Graphics Engine is a compact open
source graphics library written in C++.
It was originally developed by Skia Inc., which
was subsequently acquired by Google, who
released the software as open source.
Now known as Skia, it is currently used in Google
Chrome, Chrome OS, Chromium OS, and Android.
Ascender Technology Ltd Remote Rendering
13. Rendering Level (Skia)
Skia is a complete 2D graphic library for drawing
Text, Geometries, and Images. Features include:
− 3x3 matrices w/ perspective
− Antialiasing, transparency, filters
− Shaders, xfermodes, maskfilters,
patheffects
Ascender Technology Ltd Remote Rendering
14. Rendering Level (Skia)
Each Skia call has two components: the primitive
being drawing (SkRect, SkPath, etc.) and
color/style attributes (SkPaint).
For example:
canvas.save();
canvas.rotate(45);
canvas.drawRect(rect, paint);
canvas.drawText(“abc”, 3, x, y, paint);
canvas.restore();
Ascender Technology Ltd Remote Rendering
18. Android Remote Rendering
We can simply modify the Skia library to export
graphics by adding a small stub to Android to
serialize and send a RPC-like rendering stream.
The local client can be on any system (Linux, MS
Windows, Mac OS).
The prototype developed was on Linux-X11.
Only 17 Skia routines had to be modified to
provide remote Android graphics.
Ascender Technology Ltd Remote Rendering
19. Round Trip Latencies
The US east-west coast round trip is about 70 ms.
Just one round trip delay per frame would reduce
the possible frame rate to 14 FPS.
Potentially every Skia call returns a result. Some
results are critical for “Measure Passes”.
In order to obtain measurements, we run a Skia
renderer in parallel on the remote machine.
We can then send the rendering stream at full
network bandwidth, via a simplex link.
Ascender Technology Ltd Remote Rendering
21. Compression
Android apps have a frame rate of 30-60 FPS.
For every frame the complete rendering sequence
is sent. Compression is a good idea.
Ascender Technology Ltd Remote Rendering
22. MPEG Compression
Video streams, photographically taken or
photorealistically synthesized, can be compressed
with MPEG standard codecs.
Even though no specific technique used in MPEG
compression is applicable for the compression of
our problem domain (rendering streams), some of
the assumptions about how the data sets are
generated are similar and the compression model
design is similar.
Ascender Technology Ltd Remote Rendering
23. MPEG Stream Assumptions
The material consists of a large number of
sequential images (frames).
The target of the video images is the human visual
system and consists of moving images.
The subject matter is a product of our everyday
visual world and not some random series of
random images.
Apparent smooth motion depends on visual
continuity between frames.
Ascender Technology Ltd Remote Rendering
24. MPEG Compression Techniques
The conversion of RGB images to YUV and sub-
sampling is motivated by the color physiological
opponency theory model of human vision.
Within each frame, the accuracy of spacial changes
with shorter wavelengths are less important than
the accuracy of longer wavelengths.
Inter-frame compression is based on finding
motion vectors of the current frame based on
previous and subsequent frames.
Ascender Technology Ltd Remote Rendering
25. Rendering Stream Assumptions
The rendered material consists of a large number of
sequential images (frames).
The target of the GUI images is the human visual
system.
The subject matter while not being a product of our
everyday visual world is modeled on the visual
contexts of this world.
The GUI images are generated frame after frame
by repeated invocation of GUI procedures.
Ascender Technology Ltd Remote Rendering
26. Repeated Invocation of GUI
Procedures
In practice, the number of unique sequences of
rendering functions in execution paths taken within
the code are bounded. This is because the rendering
commands are generated by a fixed number of GUI
functions and an application running a bounded
amount of code.
The execution paths can be incrementally learned
and entered into a procedure dictionary as the
rendering commands are streamed.
Ascender Technology Ltd Remote Rendering
27. Rendering Compression Techniques
Even if the sequences of rendering functions
themselves are in the dictionary, the data
arguments associated with these functions might be
quite different from one another. Therefore, we
keep a dictionary of the data arguments previously
encountered for this particular sequence.
As a rendering procedural sequence with
associated data is encountered, the data sequence
dictionary for this procedural sequence is searched
for the closest match to the current data sequence.
Ascender Technology Ltd Remote Rendering
30. Structured Procedural Compression
A careful examination of the rendering stream
generated by the Android GUI reveals additional
structural information that can be used to improve
the data model.
The rendering stream has balanced save()-restore()
pairs within each frame of the rendering stream.
Each save() is found at the beginning of a GUI
function and a restore() is found at the end of each
GUI function.
Ascender Technology Ltd Remote Rendering
31. Structured Procedural Compression
This information can be used to reverse engineer
individual GUI and application procedures.
It will also reveal the call-graph of these procedures.
Using this information, the rendering code
dictionary becomes a rendering procedure
dictionary.
The call-graph data is best embedded in the per-
procedure data dictionary.
DEMO
Ascender Technology Ltd Remote Rendering
32. Structured Procedural Compression
The server (encoder) constructs a “procedure
dictionary” that is identical to the dictionary that is
constructed by the client (decoder). Similar of
LZW compression.
The procedures in this dictionary, surprisingly, are
reversed engineered toolkit and application level
routines.
We in effect have ended up exporting graphics at
the toolkit-application level.
Ascender Technology Ltd Remote Rendering
33. Structured Procedural Compression
Statistics
Our compression algorithm was tested on the
rendering trace of a 60 frame sequence.
There were 13702 rendering commands for an
average of 228 rendering commands per frame.
Of the 13702 rendering commands, there were
2691 functions (save/restore pairs).
Of these, only 47 were unique. This gives a
compression rate of 1.75% (about 1:57).
Ascender Technology Ltd Remote Rendering
34. Functional Compression
47 Unique Functions From 2691 Total
Ascender Technology Ltd Remote Rendering
35. Structured Procedural Compression
Statistics
Of the 13702 rendering commands only 354 had
completely unique data parameter sets and 203 had
data sets which are partially different.
Only these 557 data sets must be transmitted,
thereby giving data compression of 4.06% (about
1:25).
If the partially different data sets are differentially
transmitted, a data compression of 3.3% (about
1:30) is obtained.
Ascender Technology Ltd Remote Rendering
36. Functional Compression
557 Unique Data Args From 13702 Total
Ascender Technology Ltd Remote Rendering
37. Structured Procedural Compression
Entropy Encoding
For the last stage of compression, general
techniques such as run length encoding (RLE) and
entropy encoding (Huffman or Arithmetic) are
used to produce a minimum bit count
representation of the compressed material.
This additional phase should be expected to reduce
the number of bits by a factor of 2-3
Ascender Technology Ltd Remote Rendering
38. ICS GUI Rendering
Under ICS remote rendering can be done in any of
three different interfaces:
– Software rendering – Skia
– Hardware rendering – OpenGL ES 2.0
– Abstract rendering – Canvas class
All three interfaces can use procedural
compression.
The OpenGL rendering stream is much more
verbose (5x) than the equivalent Skia stream.
Ascender Technology Ltd Remote Rendering
39. Uses
So what is all this useful for:
− Cloud computing, remote app server
− App library, subscription model
− App demos
− Remote enterprise applications
− Set-top boxes
Ascender Technology Ltd Remote Rendering
40. Cloud computing
Any Android app can be run on the server.
It is not necessary that the remote server and the
local device have the same architecture, i.e. an
ARM server can provide services for an Intel
device.
For example: You might use Remote Rendering to
provide Android apps to Chrome OS.
Ascender Technology Ltd Remote Rendering
41. App Library / Subscription Model
Currently, apps are loaded into the local device -
either installed at the time of purchase or added
subsequently.
If efficient remote execution of apps can be
supported, then instead of software purchases, a
subscription model becomes possible.
A fixed monthly fee would entitle the subscriber to
access a large library of applications.
Ascender Technology Ltd Remote Rendering
42. App Demos
In a purchase/rental model, apps can be demoed
remotely, prior to purchase.
If the user of the device finds the app to his/her
liking, it can then be bought.
Ascender Technology Ltd Remote Rendering
43. Remote Enterprise Applications
Applications may benefit from running within the
enterprise's data centers, with the obvious benefits
of scalability, security and maintainability.
Enterprise workers typically migrates through
various computing environments (mobile-fixed-
mobile) during a typical day.
Having the application running on a server will
allow the running applications to migrate to
various clients seamlessly.
Ascender Technology Ltd Remote Rendering
44. Set-top Boxes
Many set-top boxes cannot practically run Android
since the set-top box might not have sufficient
resources or might not contain an ARM processor.
A local client application that only performs
remote rendering takes limited resources and needs
no long term persistent state.
Set-top boxes typically don't have complex local
installations. Installing Android apps locally would
present a support challenge to the operator.
Ascender Technology Ltd Remote Rendering