http://fr.droidcon.com/2014/agenda/ http://fr.droidcon.com/2014/agenda/detail?title=Framing+the+Canvas This talk will guide you through a topic that is often ignored: the Canvas API. Even if there's only scarce documentation for it, the Canvas API is at the heart of everything that shows a UI on Android. You'll need to tame this powerful and mysterious creature if you want to create great custom views and brilliant, lightweight UIs for your apps. What is a Canvas? How do you Paint? Can you draw Paths? What is Skia, and who invited it anyway? What actually happens during a drawing pass? What are Shaders? What are Filters? Is it true that text is basically impossible to measure correctly? What shape does a Shape have if noone's painting it? These and other questions will get an answer during this session. Well, some of them will not, probably. Connect the dots in the spotty documentation the Android team has assembled, and become a true pixel pusher! Speaker: Sebastiano Poggi, Novoda Sebastiano Poggi is a Android Software Craftsman at Novoda in London. A self-taught and strong OSS believer who loves in beautiful and simple UX and UI, where “less is more”, he spends a big chunk of his spare time reading and writing about Android development. He also creates and maintains some FOSS apps and libraries, that span from the fun little project to more serious (and, perhaps, useful) work. He worked some time in AKQA as a Senior Software Engineer. Before moving to London, he also worked at i’m Spa (an Italian smartwatch startup) for two years, messing with the Android platform and handling their Developer Relations. He's been toying with the Android SDK since 2010.

1. Framing the Canvas
Get your Paint together, follow the Path and master Shaders

2. Sebastiano Poggi
Android Software Craftsman à Londres
Insultez / Stalkez moi sur:
@seebrock3r
+SebastianoPoggi
rock3r
I don’t know about you, but when I work with the Canvas API, I spend most of my time looking like this…

3. CC-BY-NC Joe Benjamin on Flickr
Yes, let’s say that is my confused, or WTF, face. And again, yes, I had another guy looking puzzled at my side the whole time. But that’s a different story.
!
The point is, if you want to do amazing UI on Android you have to master the Canvas API, with all its quirks and secrets. Most of the time you have to look
at code to understand what’s going on, but that only gives you a narrow view. What you miss is the big picture, and I hope you’ll be leaving this room with
a better understanding of what does what when we talk about drawing stuff on screen.

4. Paint on the Canvas
CC-BY-NC-ND anna gutermuth on Flickr
So, let’s start from the basics. How many of you have actually ever created custom Views or crafted some custom drawing logic? The rest of you probably
have never used the Canvas API.

5. Canvas
• Base for any Android UI
• Drawing on a buffer
• Hardware or Software
• API: Canvas, Paint, …
CC-BY Cara St.Hilaire on Flickr
The Canvas API is used in Android to draw all the UI. The Canvas is the interface between your code and the underlying drawing buffer, that is then shown
on screen. That buffer is either backed by HW or SW.
!
The name is really helpful to understand what the role of the Canvas is. But as in real life, a Canvas alone isn’t enough to paint. You also need other tools
to draw colour, shapes and text onto your Canvas. These tools take the name of Canvas APIs.

6. WELL GOOGLE, IF WE COULD HAVE
SOME DOCUMENTATION EVERY NOW AND THEN
YEAH, THAT’D BE GREAT
Of course, it’s vastly under documented at best. Hey, it’s a Google thing.

7. We should love Skia
• Underneath Canvas
• Native code
• 2D graphics rendering toolkit
• Basic painting
• Advanced features
walcor on DeviantArt
Canvas is basically a Java wrapper around Skia, a 2D rendering toolkit also used in Chrome and Chrome OS. Skia is implemented in native code and
accessed by the wrapper through JNI.
!
Skia is very powerful. It can handle all the usual drawing operations, including blitting bitmaps, drawing primitive shapes, rendering text (using the
FreeText engine). It can also do complex operations, such as change the painting behaviour using Shaders, different transfer modes, matricial
manipulation, etc.

8. …but Skia kinda sucks
• Text is a pain
• Weird quirks
• E.g., no proper arcs
• No documentation
Eugenio Marletti on Medium
Not everything that glitters is gold. Skia falls short in some areas, especially when it comes to text (you can’t change the kerning; there’s no reliable way to
exactly measure text). Some functions are simply badly implemented. For example, try drawing an arc using the drawArc method. On top of it, there’s
almost no documentation available for both the Java wrapper and the C++ implementation, besides some really generic and high level infos. Good luck
with that!

9. Google
OK, Canvas is still cool
• Draw using HWUI
• Subset of Skia
• Honeycomb and later
• With limitations (of course)
• Revert to software buffer
On the plus side, Canvas APIs can be used to draw on HW accelerated surfaces since Honeycomb — using HWUI.
!
HWUI is a subset of Skia that is accelerated on hardware using OpenGL ES 2. Trivia? Trivia! Given that OpenGL ES 2 cannot be emulated in SW on
Android, that is the reason why devices running Android 3+ are required to have an OpenGL ES 2.
!
From now on we’re going to just talk about Skia, as HWUI for what we care about is basically the same thing.
!
Since life is never that easy, of course HWUI comes with quite a list of caveats. Some things are simply not available on HW accelerated layers. MaskFilters,
for example. Or some XferModes. Or drawing hinted/subpixel text. You’ll need to explicitly set your Canvas to use software rendering if you want to use
those! But you then lose HW speed improvements on that Canvas.

10. Unsupported ops on HW
Google
As you can see, there are quite a few operations that are not supported under HW acceleration. If you try to use them in an HW-accelerated Canvas, they
will map out to no-ops. These tables come from the Android Developers website; trust me, they’re not exhaustive. Things are really slowly improving, but
there are still quite a few use cases not covered under HW acceleration.

11. Good read
Hardware Acceleration
on Android Developers website
http://goo.gl/HgV8D

12. Using the Canvas
• Transforms using matrices and Cameras
• Views must use it on the UI thread
• Or go with a SurfaceView
• Canvas is stateful
Canvas can be transformed in a variety of ways. You can scale, translate, rotate your canvases using the Canvas methods. All the transformations resolve
into matrix transforms (yay, linear algebra!). This means you should brush off your maths skills if you don’t want to shoot yourself in the foot here;
concatenating matrix transformations can quickly lead to grief, suffering and pain if you’re not careful.
!
Views must only perform operations on the Canvas on the UI thread — but you shouldn’t be doing anything UI-related anywhere else anyway. If for some
reason you want to draw on a Canvas off of the UI thread, use a SurfaceView, which provides asynchronous drawing callbacks.
!
Lastly, remember: each Canvas has a state. It can be your best ally in avoid random and random-like drawing behaviours.

13. State of the art
1.Call save() — or saveToCount()
2.Do your stuff
3.Restore original state: restore()
— or restoreToCount()
So, how to deal with transformations on a Canvas? In your onDraw() method, you just have to remember to do everything this way:
1. Save the Canvas state before doing anything by calling save(). Use saveToCount() if you want to do more than one save/restore, or if you don’t trust
subclasses to balance off their save/restores.
2. Do everything you need: transform, draw, set paints, push pixels.
3. Before leaving onDraw remember to call restore(), or random stuff might — or rather, will — happen. Which you probably don’t want.

14. Seb’s top tip
Here’s a Seb top tip: how to draw a bunch of stuff on a canvas with a non fully opaque alpha. It’s not as trivial as you might think to achieve in some cases.
But then it is if you use this trick.

15. Seb’s top tip
Alpha composite overlaps, the framework way
1.int state = saveLayerAlpha(…);
2.Do your stuff
3.Blit back: restoreToCount(state);
If you need to paint overlapping objects, you can’t use the Paint alpha property, because they’ll add up and you’ll end up with less transparent areas.
Instead what you want to do is the same thing the framework does when you set a View’s alpha.
1. Save the current Canvas state using saveLayerAlpha(). This method is like save() but takes an alpha value and it also allocates an offscreen buffer. All
Canvas calls from now on will happen on that buffer.
2. Paint whatever you want to paint, without caring for the alpha.
3. When you do a balancing restore() call, the contents of the buffer get blitted back on the Canvas with the alpha you set.

16. Surfacing ideas
CC-BY-NC-SA arbyreed on Flickr
This is all nice and well, but how does what you paint get on the screen?

17. Surfaces and flingers
• Draw on surfaces
• Buffers
• SurfaceFlinger
• SW or HW backing
CC-BY-NC-SA Nicolas Hoizey on Flickr
When you talk about drawing on a Canvas, you’re actually drawing on a Surface. A surface is basically a buffer, as we already called them when talking
about Skia.
!
Surfaces are composited together to be shown on screen by the SurfaceFlinger, which manages all the offscreen buffers and coordinates them to be
shown on screen.
!
Surfaces can be backed in hardware, in which case they are also said to be HW composed, or in software. HW composed surfaces are managed by the
instance of SurfaceFlinger that is backed by an HardwareComposer. HWComposer is a HAL for a memory buffer hosted outside of system memory, usually
on a 2D blitter, that can also be the GPU.
!
When the HWComp cannot handle a surface (because it might not have enough memory available, because it’s been asked not to, etc) then a surface can
be backed in SW. That means it resides on system memory, and is managed by an instance of SurfaceFlinger that is backed by OpenGL ES 1.

18. Good read
AOSP Graphics
on AOSP Source website
http://goo.gl/i5iVwH
The graphics subsystem in Android is incredibly complex and a slide is barely enough to name the main components. I strongly recommend you to read
the fairly recent documentation on the AOSP source website. It’s long and can be quite hard for a Java dev to understand it from time to time, but it’s
definitely worth it if you want to understand what’s going on under the hood.
!
Besides, it’s one of the very few examples of good documentation in AOSP, so might as well use what we get.

19. Layers
• Offscreen buffers
• Every Canvas has one
• Can be blitted with a Paint
• Different types of layer
CC-BY-NC-SA Doug88888 on Flickr
A Layer is basically an offscreen buffer that you can paint onto. Does this remind you of anything? Yep, they’re basically mapped to surfaces.
!
Every Canvas has a backing layer, and you can create more layers as needed, to use as buffers, as we’ve just seen in the previous slide.
!
Views can have an offscreen layer, and use it to achieve some special effects when compositing back the result. In particular, it’s interesting that you can
use any Paint object to have the buffer blitted back on the Canvas. You can use this Paint to apply Shaders, for example to tweak the brightness/hue/…,
or some other color transformation.
!
There are different layer types, which we’ll now see.

20. View Layer types (3.0+)
• NONE: no buffer
• SOFTWARE: bitmap buffer
• HARDWARE: HW-backed texture
• Can improve performances (e.g., animations)
On Android 3.0+ (API 11+) Google has introduced the setLayerType() method and thus different types of layers. All Canvases before then were backed by
software layers.
!
This allows you to specify the backing layer type for a View’s Canvas:
- The NONE flag indicates that the Canvas doesn’t have any backing layer (buffer), either SW or HW
- The SOFTWARE flag indicates that the Canvas is backed by an in-memory drawing buffer. This is useful to use features not available with HW
acceleration
- The HARDWARE flag indicates that the Canvas is backed by an hardware texture buffer on the GPU (where available)
!
Using offscreen buffers can help improve performance. In the case of HW textures, they’re kept around until the invalidate() method is called, so the Views
can be redrawn without calling onDraw again until needed. In the case of SW bitmaps, they are used as a cache by the View class, to achieve a similar
result.
!
Besides that, using offscreen HW buffers can also help saving time when blitting with some kind of effects. For example, if you use setAlpha() on a View
that has a backing layer, the framework will not have to call onDraw again to change the alpha; it will simply blit with a different Alpha level. This is

21. CC-BY Brandon Morse on Flickr
Pushing pixels
What happens under the hood in Skia then, from the moment you issue a draw command to the point where you see the results on screen? Let’s take a
look at the Skia pipeline.

22. Canvas API
Paint
Color Matrix
Shape
NinePatch
Drawable Bitmap
Path
Camera
Movie
Picture
Interpolator SurfaceTexture
Ty p e f a c e
The Canvas API has got a lot of things in it. This is just a small part of all the classes you will have to deal with. And yet, we barely have the time to cover
one, which is arguably the most important one:

23. Canvas API
Drawable
Shape
Path
Color
Bitmap
Camera
Movie
Picture
Interpolator SurfaceTexture
Ty p e f a c e
Matrix NinePatch Paint
Paint.

24. Just deal with it.
!
Also, hello Tim! This gif’s for ya.

25. Paint it (black)
• Knows how to colour the
pixels
• All drawing calls need a Paint
• Handles text as well
• TextPaint subclass
CC-BY-NC-ND Mark Chadwick on Flickr
Paint is the object that holds the informations about how to paint something on a Canvas. Actually, all drawing operations require that you pass a Paint
along. It’s really the intuitive idea of the paint that you use to draw something in real life: if you have a canvas and a brush but no paint, there’s no way
you’re ever going to be able to draw anything.
!
Paint holds informations about the usual basic properties about how to draw: the fill color, the stroke (color, width, caps, etc), the alpha level, etc. It also
holds informations about the antialiasing to apply, the bitmap filtering (off by default) and the text hinting. Of course not all of Paint’s properties are useful
for all kinds of drawing operations.
!
Paint knows also how to measure and draw text. It holds informations on the Typeface to use, the text size, style, scaling ratio, etc. It offers methods to
measure text size (all of which do a pretty bad job at it, and each of them has a nuanced different meaning from the others, but that’s really confused and
difficult to understand), and there’s even a text-specific subclass, TextPaint, that holds more informations about the text should be measured and painted.
!
The Paint object you pass to a drawing call is used throughout the painting pipeline.

26. Skia pipeline
Adapted from Laurence Gonsalves’
post on Xenomachina
The rendering pipeline for drawing calls in Skia is composed of four main stages: path generation, rasterization, shading and transfer. This is a simplified
view of the general pipeline; it’s worth noting that there are several shortcuts and optimisations that can change the way this works.
!
The first stage, path generation, computes a Path that contains the region that will be painted. This path is then fed into the Rasterization step, which can
happen in two different ways.
!
If the user has specified a Rasterizer, then that’s going to take the whole step onto itself and generate the final alpha mask. If not, the Path is trivially
rasterised using a scan conversion (including semi-transparent pixels on anti-aliased edges).
!
The next step (again, this might even happen in parallel to the previous steps) is Shading, which determines the colour that the pixel will be painted with.
!
This outputs an image that is then used in conjunction with the mask in the last step, Transfer. In this step the image and the mask are blended and
painted on the destination Canvas.

27. Good read
Android's 2D Canvas Rendering Pipeline
by Laurence Gonsalves on Xenomachina
http://goo.gl/4W5R0Z
If you want to read in more detail how the Skia pipeline works, here’s an excellent article that I only found out recently. Of course I found it after I had
spent an inordinate amount of time on doing my own research, because OF COURSE. So frustrating! But no, seriously, read it. It’s full of super interesting
stuff.

28. Effective painting
• Steps have two phases each
• “Two-pass” strategy
• Effects modify steps output
• Second passes default to
identity
CC-BY-NC-SA ClintJCL on Flickr
Each step output, as you could see in the flowchart, can be modified using the so-called Effects. Each step has generally two effects that you can apply, in
a sort of “two-pass” strategy. Think of it as a first pass to achieve an intermediate result, and a second pass to refine it.
!
There is an exception, which is…? Yes, the Rasterisation pass. If you assign a Rasterizer to your Paint, it will bypass the built-in rasterisation logic and also
any MaskEffect you might have set. Fun. Even better, undocumented fun! Luckily enough you’ll probably never need to use a Rasterizer anyway, especially
considering there only is one available, LayerRasterizer, which implements a matrioska of shaders.
!
In the pipeline, if you don’t specify any custom effect, Skia will use its defaults. Usually, “second passes” effects default to the identity function, which
means, they output what they receive as input, unaltered.

29. Mask filters
• Rasterization phase
• Affect alpha mask
• Not HW accelerated
• Blur and Emboss
P R S T
Mask filters allow you to affect the alpha mask of the paint, before it is colorized and drawn. The alpha mask is, in fact, an ALPHA_8 bitmap that is
generated by rasterising the Path resulting from the first step. Remember that the MaskFilters will be bypassed if you use a Rasterizer!
!
In case for some reason you didn’t know what an Alpha channel is, but you still ended up in this session, it’s the channel that contains per-pixel information
about the transparency. A value of 0 (conventionally depicted as black) means fully transparent, and a value of 255 (white as per conventions) means fully
opaque.
!
The image shows how the Blur filter affects the alpha mask for a circle. The top half is drawn on a software layer, the bottom part on a HW layer.
!
If you were wondering, YES, this means MaskFilters aren’t available on HW layers. You can set the layer to use a SOFTWARE buffer to get access to the
MaskFilters, but remember that applying them can be pretty expensive, performance-wise.
!
There’s another built-in Mask filter: Emboss. Nobody ever cared about embossing stuff for, like, ever.

30. Good read
Blurring Images (series)
by Mark Allison on Styling Android
http://goo.gl/zZVs2V
There are other ways to blur something. RenderScript is surely the best one, as it offers a Blur intrinsic. Remember that using RenderScript involves
marshalling data through JNI, as it would if you were using native code, and that can be a bottleneck. Also, since we’re talking about alpha masks here,
probably using a mask drawable is more efficient.
!
In case you’re interested on how to use Renderscript to blur out stuff, you can find an in-depth series of articles on Mark Allison’s Styling Android blog.
And then get him a beer. He deserves it.

31. Shaders
• Shading phase
• Similar to OpenGL shaders
• Non programmable
• TileMode
CC-BY-NC-P R S T SA Andreas Köberle on Flickr
Shaders are used in the Shading phase (duh!) to determine the colour of each pixel that will be drawn.
!
Conceptually, a Skia Shader is kinda similar to an OpenGL shader (but not really). There’s a huge difference, which is, in Skia shaders aren’t programmable
— not on Java, anyway. This means they’re basically immutable, which means that to alter a Paint’s Shader you must create a new one. Yay GC!
!
Good news is, this is supposed to change in future versions of Android.

32. minSdkLevel ‘over9000’
Bad news is, see you in a few years’ time before we can actually use them. Go, minSdkLevel ‘over9000’!

33. Shaders
• Shading phase
• Similar to OpenGL shaders
• Non programmable
• TileMode
CC-BY-NC-P R S T SA Andreas Köberle on Flickr
Shaders aren’t affected by the Paint mask. If drawing outside of the shader bounds, you can use Shader.TileMode to specify if you want to CLAMP,
MIRROR or REPEAT the shader contents outside of the clipping mask.
!
CLAMP is usually the most efficient; if your bitmaps have power-of-two sizes, then…

34. …everything is awesome. Pretty much.

35. A Shady bunch
Shader
BitmapShader
ComposeShader
*Gradient
Use a Bitmap as texture when
painting
Combine two different shaders and
mix them with a Xfermode
Paint using a Linear, Radial or
Sweep gradient
P R S T
There are three main types of Shaders:
- The BitmapShader is a Shader that loads a bitmap as a texture and uses it to paint the pixels
- The Gradient shaders (LinearGradient, RadialGradient and SweepGradient) paint the pixels using the specified gradient
- The ComposeShader allows you to combine two shaders together when painting, specifying the XferMode to use to mix them.
!
Note that in HW accelerated canvases there is a very limited support for ComposeShaders (can’t nest them, can’t use two shaders of the same type).

36. Color Filters
• Adjust colours after Shaders
• Uniform transformation for all pixels
ColorFilter
ColorMatrixColorFilter
LightingColorFilter
PorterDuffColorFilter
Apply a 4x5 colour matrix transform
Multiply SRC colours by a colour, and
then add a second colour to them
Apply a colour to the SRC colour
using a Porter-Duff mode
P R S T
Color filters are used to modify the paint colours after the Shaders have set them. They are the second phase of the Shading step.
!
ColorFilters aren’t function of the pixel coordinates, which means, they apply the same transformation to all pixels.
!
There are three types of ColorFilters:
- The ColorMatrixColorFilter transforms the colors in each pixel by using a 4x5 ColorMatrix. This can be used for changing the Brightness/Saturation/Hue,
to convert between RGB and YUV color spaces, etc. You can also concatenate matrices to achieve composite transforms.
- The LightingColorFilter allows you to premultiply all pixels’ colors by a specified color, and then offset the result by a second color. All values are limited
within the [0, 255] range
- The PorterDuffColorFilter applies a single color to every pixel, mixing it with the source color using the specified Porter-Duff mode

37. Transfer modes
• Second step of Transfer phase
• Blend SRC image onto DST through mask
Xfermode
AvoidXfermode
PixelXorXfermode
PorterDuffXfermode
Draw (or don’t draw) pixels based on
the “distance” from a reference colour
Exclusive-OR source and destination
pixels. Drops the alpha channel!
Blends the source and destination
colours using a Porter-Duff mode
P R S T
Transfer modes (Xfermodes for the friends) are the second phase of the Transfer step, and the last of the pipeline.
!
Transfer modes are used to determine the way the shaded source image (SRC) is blended with the destination (DST), through the mask.
!
There are three Xfermodes in Skia:
- The AvoidXfermode draws, or doesn’t draw, pixels that are within a certain colorimetric distance (tolerance) from the specified key color.
- The PixelXorXfermode XORs all the pixels of the source and destination. Since this Xfermode doesn’t follow premultiplied conventions, the resulting
pixels will always hava a solid alpha (255).
- The PorterDuffXfermode mixing the destination with the source pixel colors using the specified Porter-Duff mode.

38. Q&A
pipeline outline
code?
onDraw jdoc — canvas — views

39. We’re hiring!
join@novoda.com
If you like what you’ve seen and you’d want to be working on awesome Android stuff then drop us an email! We don’t generally bite.