iOS and Android accessibility APIs (AccessU 2017)

iOS and Android
accessibility APIs
#AccessU2017
Austin, Texas
18 May 2017 (Happy #GAAD!)
Jon Gibbins
@dotjay

Introduction
• Jon Gibbins (@dotjay)
– Accessibility consultant at Dig Inclusion
– Web developer since 1999
– Assistive technologist since 2002
– Mobile specialist since 2012
– Passionate about accessibility – why?
2

Introduction
Photo credits: LG, Gould, Larson,
DiC, Apple
3

Why accessibility?
5
Photo credit: Jon Gibbins with thanks to Drake Music Project

Empathy
Most of us have a connection to accessibility
6

Empathy
is about understanding people
7

Inclusion
is about understanding people and the
barriers that they face
8

What we will learn
• AT on mobile devices in brief
• iOS UIAccessibility
• Android accessibility API
And if there’s time…
• Some remediation techniques
• Testing API on mobile platforms
10

Questions are encouraged
– If you have a question, or want to know
more, please put your hand up
11

Mobile features
In brief…
12

Mobile features
iOS
13
Settings > General > Accessibility

Mobile features
Android
14
Settings > Accessibility

Mobile screen readers
Two main interaction
methods
1. Explore by touch
– Drag finger over screen
– Items under your finger are
described by screen reader
– Tap with a second finger or
double tap to open/activate
2. Gesture navigation
16

Mobile screen readers
Two main interaction
methods
1. Explore by touch
2. Gesture navigation
– Swipe right/left moves focus
to next/previous content in
sequence
– Items are described by
screen reader as focus
moves
– Double tap to open/activate
17

VoiceOver
1. Triple click the Home
key to activate
2. Dial to open the Rotor
3. Swipe up/down to
navigate parts
4. Swipe right/left for
next/previous content
18

TalkBack
1. Activate using Power +
2 fingers shortcut
2. Swipe right/left for
next/previous content
3. Swipe up/down to
switch modes
4. Use Global/Local Menu
for other features
20

• Enable “Accessibility shortcut” in
Settings
• Long press volume to
suspend/resume (4.2+)
• Swipe down and right (L-shape) for
Global Context Menu
• Swipe down and left (backwards L) for
Local Context Menu
• “Read from top” is in the Global
Context Menu
TalkBack tips
21

Accessibility API
fundamentals

A dialog takes place between an application and assistive technology.
26
Accessibility theory
Accessibility API fundamentals

• We know what assistive technology is,
but how does it work?
• Accessibility APIs
– Present interfaces as information rather
than purely visual
– Allow interfaces to be changed by AT
– Provide a common vocabulary
27

• Accessible Object Properties
– User interface is represented as a hierarchy of
accessible objects
– Each object has a variety of properties, such as:
• name: Defines a label. (“Hi, what’s your name?)
• role: Defines the behavior. (“So, what do you do?”)
• state: Defines the current condition. (“How are you?”)
28

• Accessible Events
– Accessibility APIs notify assistive
technologies of changes by broadcasting
events.
29

• Example: a checkbox in a desktop
application
– role: check box
– name: Open new windows in a new tab instead
– state: checked
30
Accessible object properties

• Examples in HTML
– role:
<img alt="Can of soda" src="soda.jpg">
<label for="last">Last name</label>
<input type="text" id="last" name="last" value="Bloggs">
– name:
<img alt="Can of soda" src="soda.jpg">
31

• Examples in HTML
– state:
<input type="text" id="last" name="last" value="Bloggs"
disabled="disabled">
– value:
32

• An accessible name is required and
identifies an object. It is short and does
not necessarily describe the object.
<label for="date">Event date</label>
<input type="text" id="date" name="date">
33

• An accessible description is optional
text that provides additional information
about an object.
<label for="date">Event date</label>
<input type="text" id="date" name="date”
aria-describedby="instructions">
<p id="instructions">Must be in mm/dd/yyyy
format.</p>
34

The iOS Accessibility API:
UIAccessibility overview

source: Accessibility for iOS by Chris Fleizach, WWDC 2012 (47 mins)
https://developer.apple.com/videos/wwdc/2012/?id=210
The iOS Accessibility API
UIAccessibility
36

• UIKit mostly “just works”
– A lot of accessibility comes for free if you use
these UI components
• Much of the work involved is in setting labels
– accessibilityLabel
• Also look out for:
– isAccessibilityElement
– accessibilityTraits
– accessibilityHint
• This is about 80-90% of the work involved in
implementing accessibility on iOS
UIAccessibility
37

• Back to our common API vocabulary:
– role = traits
– state = traits (again‽ multitasking superstar!)
– name = labels
– value = value
– desc = hint
UIAccessibility
38

UIAccessibility
isAccessibilityElement
accessibilityLabel = name
accessibilityHint = description
accessibilityValue = value
accessibilityLanguage
accessibilityTraits = role and state
accessibilityFrame
accessibilityPath
accessibilityActivationPoint
accessibilityElementsHidden
accessibilityViewIsModal
accessibilityNavigationStyle
shouldGroupAccessibilityChildren
UIAccessibility
39

UIAccessibilityTraits
UIAccessibilityTraitNone;
UIAccessibilityTraitButton;
UIAccessibilityTraitLink;
UIAccessibilityTraitSearchField;
UIAccessibilityTraitImage;
UIAccessibilityTraitSelected;
UIAccessibilityTraitPlaysSound;
UIAccessibilityTraitKeyboardKey;
UIAccessibilityTraitStaticText;
UIAccessibility
40

UIAccessibilityTraits (continued)
UIAccessibilityTraitSummaryElement;
UIAccessibilityTraitNotEnabled;
UIAccessibilityTraitUpdatesFrequently;
UIAccessibilityTraitStartsMediaSession;
UIAccessibilityTraitAdjustable;
UIAccessibilityTraitAllowsDirectInteraction;
UIAccessibilityTraitCausesPageTurn;
UIAccessibilityTraitHeader;
UIAccessibility
41

UIAccessibilityTraits = role and state
let myCustomButton = UIView()
myCustomButton.isAccessibilityElement = true
myCustomButton.frame = self.view.frame
myCustomButton.accessibilityTraits
= UIAccessibilityTraitButton
myCustomButton.accessibilityLabel = "Press me!"
Best approach?
UIAccessibility
42

UIAccessibilityTraits = role and state
let myButton = UIButton()
myButton.accessibilityLabel = "Press me!"
Use UIKit!
UIAccessibility
43

accessibilityFrame
– Method implementation
– VoiceOver polls the API for accessibilityFrame info
• Why? Frames can change, e.g. on orientation
– Important when merging two elements or creating invisible
prompts (such as container headings)
UIAccessibility
44

UIAccessibilityIsVoiceOverRunning
– Allows developers to detect is VoiceOver is running.
UIAccessibilityVoiceOverStatusChanged
– Allows developers to detect when VoiceOver is enabled.
• When is this appropriate?
– e.g. performance vs restart app
UIAccessibility
45

Labels
• accessible name
– Describe the meaning or purpose of the interface
element
– Front-load important information
– Use commas to break up information when announced
by TalkBack
• android:contentDescription in XML layout
files
• setContentDescription() to dynamically
update descriptions

Labels
• Assign contentDescription to all user interface
components, e.g.
– ImageButton
– ImageView
– CheckBox
<ImageButton
android:id="@+id/add_note_button"
android:src="@drawable/add_note"
android:contentDescription="@string/add_note"/>
• Decorative images should not have descriptions
– android:contentDescription="@null"

EditText fields
• android:labelFor
= best approach
Most like HTML / ARIA (since API level 17, Jelly Bean MR1)
• android:hint
= not ideal accessibility
Previously, best practice to provide an attribute instead of a content
description. When the field is filled, TalkBack reads the entered content
to the user, instead of the hint text.
• android:contentDescription
= terrible accessibility
Only read before text is typed into the field. Once the field is filled, the
contentDescription is never read again

Roles
• Use built-in UI components / Material
Design for role and state:
– Button
– Text field
– Checkbox + setChecked(boolean)
– Radio button
– Switches / Toggle button
– Spinner
– Pickers
– Sliders

Focus
Two focus concepts on Android:
1. Accessibility Focus (or "traversal”)
The linear order in which elements receive focus
when using TalkBack gesture navigation.
2. Input Focus
The order interactive widgets (text fields, buttons,
etc) receive focus when navigating using a D-pad
(spatial directional navigation) or keyboard (linear
Tab navigation).

Accessibility Focus
Enabling accessibility
android:importantForAccessibility (API 16, Jelly Bean)
Like aria-hidden (in reverse)…
• auto (default)
The accessibility service determines whether the view is important for
accessibility. Recommended for most cases.
• yes
The view is important for accessibility. Useful if a view does not appear
to be exposed to TalkBack for any reason.
• no
The view is not important for accessibility. Useful if a view is announced
by TalkBack but you want to hide it from TalkBack users.
• noHideDescendants
The view is not important for accessibility, nor are any of its descendant
views.

Accessibility Focus
Traversal order (API 22+)
android:accessibilityTraversalAfter
Sets the next view to receive focus when using
TalkBack gesture navigation.
android:accessibilityTraversalBefore
Sets the previous view to receive focus when using
TalkBack gesture navigation.
Note: API 22 is Android 5.1, Lollipop MR1.

Input Focus
Enabling focus
android:focusable
Like tabindex in HTML…
If a view is not focusable by default:
• android:focusable="true"
• setFocusable("true")
Check state or request focus
• isFocusable( )
• requestFocus( )

Input Focus
Focus order for Tab navigation (API 1+)
android:nextFocusForward
Sets the next selectable view to receive focus
when the user navigates, for example, by pressing
Tab on the keyboard.

Input Focus
Focus order for directional navigation (API 1+)
android:nextFocusLeft / Right / Up / Down
Sets the next view to take focus when the user
navigates using the arrow keys on a keyboard or
directional controller.

Focus
• How do I move TalkBack focus?
targetView.sendAccessibilityEvent(
AccessibilityEvent.TYPE_VIEW_FOCUSED
)

Other
• How do I get TalkBack to speak?
View.announceForAccessibility(
"Say something! Say something!
Anything…");

Other
• How do I detect TalkBack?
– AccessibilityManager.isEnabled() (API 4)
– isScreenReaderActive (old, don’t use)

• If focus does not move to a new screen, likely that a
screen changed notification has not been fired
iOS remediation tips
66

• If focus does not move to a new screen, likely that a
screen changed notification has not been fired
• If focus does not move to a modal, likely that a layout
changed notification has not been fired
67

• Separating visual text from what is announced
– e.g. accessible date format announcements
68

• Add hidden content
– e.g. landmark names
– hidden heading solution
69

– hidden heading solution
• Arbitrary announcements
– e.g. dynamic messages
70

• …
• Create VoiceOver specific behaviours
– Demo: Stocks.app chart (UIAccessibilityContainer)
– Demo: Mail.app swipe options (accessibilityCustomActions)
71

• …
• Create VoiceOver specific behaviours
– Demo: Stocks.app chart (UIAccessibilityContainer)
– Demo: Mail.app swipe options (accessibilityCustomActions)
• Reorder focusable elements
– Accessibility Elements array (again UIAccessibilityContainer)
72

Android
• name =
android:contentDescription or
setContentDescription()
• role and state = built-in controls
– Button
– Text field
– Checkbox ( e.g. setChecked(boolean) )
– Radio button
– Toggle button
– Spinner
– Pickers

Android
– LinearLayout with a contentDescription and
android:importantForAccessibility="yes"
– Hidden heading solution (no visible focus)
<TextView
android:layout_width="1dp"
android:layout_height="1dp"
android:contentDescription="My heading"/>

Android
• Custom AccessibilityAction
– Added to Local Context Menu
• When all is broken, override accessibility tree
– AccessibilityNode*

iOS testing
• Accessibility Inspector
– Check exposed accessibility properties
– Part of Xcode, but does not require Apple
Developer Program subscription to use
– Xcode > Open Developer Tool > Accessibility Inspector
– Older: General > Accessibility > Accessibility Inspector
78

iOS testing
• New Accessibility Inspector in Xcode 8
– Launched September 2016
– Use to identify potential accessibility issues
– Inspect app on attached iOS device
– Not a replacement for manual testing
– Automated accessibility audits
– Interactive inspection modes
– Accessibility settings live preview
Xcode > Open Developer Tool > Accessibility Inspector
79

iOS testing
• New Accessibility Inspector
•Select target
• Inspect tool
• Inspection
• Debug and analyze accessibility state
83

iOS testing
•Audit
• Find and report accessibility issues, and
suggests fixes
• Element has no description
• Potentially inaccessible element
• Image has no label
• Image name used in description
• Does not check contrast / color blindness
84

iOS testing
•Settings
• Test impact of accessibility setting change
85

iOS testing tips
• Sup' up your Rotor
– Speed up your testing
– In Rotor settings, include the setting for "Speech
Rate”
– Use Rotor to tweak speech rate on the fly
– I use around 75%
86

iOS testing tips
• Sup' up your Rotor
– You often don't have access to the code base, so
you can't inspect what is going on
– Switch the character mode and listen to the label
spelt out
– You should be able to ascertain if something is
human readable text or a piece of code, an ID of
some kind
– Typical for images to be "named" its file name.
87

Android testing
• Accessibility Scanner
– Launched March 2016
– Any Android app
– Identifies potential issues
– Suggests solutions
• larger text
• color choices
• navigational tools
• more
89

Android testing
• Accessibility Scanner
– Download
– Enable in Settings > Accessibility
– Open app to test
– Start scan
– Long-pressing to move screens
90

Thank you!
• Slides available later
• I’m @dotjay on Twitter
• Feedback gratefully received!
Copyright 2017 Jon Gibbins/Dig Inclusion
Contributions from The Paciello Group used with permission
© Jon Gibbins 93

iOS Accessibility
• iOS Accessibility Programming Guide
• Apple Developer videos on accessibility
– Accessibility for iOS – get started by watching Chris
Fleizach at WWDC 2012 (47 mins)
• Usability of iPad Apps and Websites by
Nielsen Norman Group
• Cheat sheet for learning the gestures used
on VoiceOver for iOS
95

Android accessibility
• Android Emulator
http://developer.android.com/tools/help/emulator.html
• Using the Android Emulator
http://developer.android.com/tools/devices/emulator.html
• Android Accessibility home page
http://developer.android.com/guide/topics/ui/accessibility/index.html
• Making Android Applications Accessible
http://developer.android.com/guide/topics/ui/accessibility/apps.html
• Android Accessibility Developer Checklist
http://developer.android.com/guide/topics/ui/accessibility/checklist.html
• Android Accessibility Testing Checklist
http://developer.android.com/tools/testing/testing_accessibility.html
• Using Android Lint (video, 1 min 45 secs)
http://www.youtube.com/watch?v=OtwCe-YlD5k
• Enabling TalkBack (video, 1 min 27 secs)
http://www.youtube.com/watch?v=VY6CZo7gADE
96

iOS and Android accessibility APIs (AccessU 2017)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to iOS and Android accessibility APIs (AccessU 2017)

Similar to iOS and Android accessibility APIs (AccessU 2017) (20)

More from Jon Gibbins

More from Jon Gibbins (8)

iOS and Android accessibility APIs (AccessU 2017)