Voice Interface Design: Skills, Actions, Tips for Building Voice Apps

© 2017 Versay Solutions
Voice User Interface
Design:
Skills, Actions, And The Future
Crispin Reedy, Versay Solutions
@crispinTX crispinreedy.com
#BigD17

Voice User Interface
Design:
Skills, Actions, And The Future
Disclaimer: This session was
NOT sponsored by Dominos

• Voice User Interface Designer
• 15+ years in the field
• Former coder; got interested in UX
• President of the Association for Voice
Interaction Design
• Consultant for Versay Solutions
@crispinTX
crispinreedy.com

Session Description
• Amazon Skills for Alexa, Google Actions for Home
– Should your company build a conversational
voice interface for one of these systems, and if
so, how?
• What are the differences between a voice user
interface and other types of UIs?
• What types of skills does a VUI designer need?
• What are some best practices for these VUIs?
• You’ll walk away with answers to the questions
“If, Why, and How” you might choose to explore
this interesting new area of design.

Easy Answer To #1
• If your company is involved in home
automation:
• Mostly likely Yes, and Yesterday
• Although how you do it will depend on your
platform
• More on that later!
• Everyone else
• Let’s keep talking!

Basic Terms

Terms & Technologies
•Speech Recognition
•Natural Language Understanding
•Voice Verification (Biometrics)
•Text to Speech

Speech Recognition “ASR”
“See the cat.”

Natural Language Understanding
•Extracting meaning from natural text
“Hello, yes,
I’d like to
pay my
water bill.
Can you
help me with
that?
Intent =
BillPay
Entity
(Bill Type) =
Water

Voice Verification
“My voice is
my password.”
“Authenticated.
Welcome, Mr.
Smith.”
✓

Speech Recognition
• Hands-free command /
control
• Dictation
• Input text
• Small form factor
device, etc.
Text To Speech
• Output text dynamically
• Respond to input
• Useful when no
display is available
Natural Language
Understanding
• Necessary for all
language-based input
• Extract meaning
• Parse large volumes of
text
Voice Verification
• Security

ASR
Application
Data
• Sign-In
• Interaction
• Request
• Action
• Meaning
• Access Data
• Output
TTS
NLU
Voice
prints
Verifi-
cation

Speech Technology
Today

Speech Agents, Apps, and APIs
Speech Agents:
• Amazon Alexa
• Echo, Dot, Echo Show
• Google Assistant
• Pixel, Android, Google Home, iPhone app
• Apple’s Siri
• iPhone, iPad, MacOS (Sierra), AppleTV
• Microsoft’s Cortana
• Windows 10, Windows Phone, Xbox, iPhone app
• Samsung’s Bixby
• Galaxy S8, Family Hub 2.0 Fridge

Speech Agents can be extended with
“Voice Apps”
• Alexa Skills
• Google Actions
• SiriKit
• Cortana SDK

Agent capabilities and apps are somewhat
determined by:
• Platform: Device
• Screen, keyboard, phone, mics, etc.
• Environment: Web site, apps that interact with
the agent
• Ecosystem: Underlying connections, technical
partnerships

Platforms

Environment
Google “Actions” or
“Apps”
• Curated
• Direct vs.
Conversational
Siri - Works via apps
Order Uber Order Lyft

APIs: Allow you access to the underlying
technology
• Amazon
• AVS (Alexa Voice Service) Create an “Alexa” on your
own device
• Amazon Lex, Amazon Polly
• Google
• Cloud Speech API
• API.ai
• Apple
• Apple Speech Framework
• Microsoft
• Bing Speech API
Ecobee Smart
Thermostat

Use Cases

Use Case “Bakeoff” from Tech Insider
•Travel
•Email
•Messaging
•Sports
•Music
•Weather
•Calendar
•Social
• Translation
• Basic tasks
• General knowledge
• Personality
http://www.businessinsider.com/siri-vs-google-assistant-cortana-alexa-2016-11/

Use Case “Bakeoff” from Tech Insider
• “wildly finicky when it comes to phrasing.”
• “Each assistant still feels like a fragile, thinly veiled
web of loosely connected services — because that's
what they are.”
• “incredibly uncomfortable to speak to an inanimate
thing in public.”
• “In Google Assistant's case, normalizing the need to
call on a brand ("OK Google") whenever you need a
hand is Orwellian.”
• “None of these things
are at a place I could
comfortably call
"good.””

Personal Assistant vs. Home Assistant
The Google Pixel XL.
Hollis Johnson/Business Insider
Google.com

Personal Assistant vs. Home Assistant

Getting Specific With
Alexa

“Layers” of Alexa
•Alexa Native Capabilities
•Alexa Skills
•Alexa Voice Services

“Layers” of Alexa
• Alexa Native Capabilities
• Come out of the box
• Require Alexa wake word (can be changed)
• Alexa Skills
• Alexa’s “Extensions” or “Add-Ons”
• Designed for and deployed on Echo Device
• Skills must be downloaded to Echo
• Require Alexa wake word + Skill name
• Alexa Voice Services
• Add Alexa voice control to your own device

Alexa “Native” Capabilities
Alexa, what’s 3 + 5?
Alexa, set an alarm
for 3 am.
Alexa, set a thirty
second timer.
Alexa, what’s the
weather?
Note:
Mix of TTS &
Pre-Recorded
Audio
Note:
“Hint”

Design Considerations
•Proactive “Hints”
• Similar to “Hover Help” or “Tool Tip”
• But less avoidable!
• Pro: Can teach user about other capabilities
• Con: Can be annoying!
• Guideline: If used, be sparing
• Develop rules for when and how frequently to
offer

© 2017 Versay Solutions Amazon.com
Native & Skill
Skill
Skill
Skill
Native & Skill
Alexa Skills

Alexa Skills
Amazon.comAmazon.com

Amazon.com

• Invoking Skills:
• Alexa, open Oprah Magazine
• Alexa, order a pizza from Domino’s
• Alexa, ask Cook Reference what’s the
safe temperature for chicken
• Syntax:
Open <skill>
Ask <skill> for (about, to, with, etc.)
<action>
Ask <skill> <question>
Also: Search, Tell, Talk to, Launch, Start,
Resume, Run, Load, Begin
Oprah Magazine

•Skills can be “installed” on the
fly
•If the user knows the name of
the skill
•Skills that require account
information will need extra
steps
Cook Reference
Domino’s

Alexa App + Linking

•Managing access to skills may become
difficult or confusing.

•Attention (or lack of attention!) to
technical details can become “deal-
killing” part of overall experience
Domino’s

Really?
Dominos.com

No
Dominos.com

• Confirmation
• What’s the phone number?
• 214-555-1235
• You said 214-555-1235. Is that correct?
• Yes
• Note: System confirmed the phone number but
not the address
• Was the address really correct?

• “Would you like to place your Easy
Order, reorder your most recent
order, or start a new order?”
• If I’m not logged into my account on
the Alexa app, options 1 and 2 don’t
make much sense.
• “Would you like” is ambiguous – could
be used for Yes / No questions or for
multi-item questions
• First part of the sentence runs into the
choices
• Reuse of the word “order” just seems
odd (but may be unavoidable).
• Could have used more pauses (SSML)
Domino’s

Design Considerations: SSML
• Speech Synthesis Markup Language
• Can control the way your TTS playback sounds
• Very important if your output is mostly TTS
• Which is true of all most platforms
• Should be supported by all types of TTS
engine
• Amazon has platform specific options
• Plan on using it to fine tune your audio output

New Prompts & SSML Examples
• Note: TTS Samples with SSML created
with Amazon Polly, not Alexa
• “You can: Place your easy order.
Reorder your most recent order. Or,
start a *new* order.”
• You can: <break time="500ms"/>Place your easy order,
<break time="500ms"/> Reorder your <emphasis
level="moderate">most recent</emphasis> order, <break
time="500ms"/> Or, start a <emphasis
level="strong">new</emphasis> order.
• Placing an order, great! Choose from:
My easy order. My most recent. Or,
start a *new* order.
• <speak>Placing an order. <prosody
pitch="high">Great!</prosody> Choose from: My easy
order. My most recent. Or, start a <emphasis> <prosody
pitch="high">new</prosody> </emphasis>
order.</speak>
Domino’s

Still Trying To Order That Pizza
• Start of the interaction has changed!
• Probably due to login
• “Would you like to place an order, or
track an order?”
• What just happened!!!?
•System was expecting me to say
“Start a new order” and I only said
“New Order.”
Domino’s

• Make sure your input grammar covers all
possible logical utterances (what user can say)
• Don’t leave this stuff up to the programmers!
• Provide examples of coverage
• Coverage should match prompts
• Use some kind of markup to show coverage
• [] optional
• () grouping
• | or
• “Would you like to place your Easy Order,
reorder your most recent order, or start a new
order?”
• [place] [my | an] Easy Order
• [reorder] [my] most recent [order]
• [start a] new [order]

•Reprompts:
• What do you do when you didn’t
understand what the caller said?
• Probably don’t want to say “Sorry”
• This can be annoying
• But you CAN rephrase the prompt to make it
different
• Using the same prompt gives the user a sense
that something has gone wrong

Pizza Pizza Pizza
• Hey you didn’t really need to
explain about the phone number
since I saved it but OK….
• Address has been saved to profile,
great!
• And then boom
Domino’s

With Speech, you need to spend
a lot more time thinking about
what happens when things go
wrong.

I Didn’t Really Want to Order Pizza
But By Now I Am Hungry
And So Is Somebody Else
• Note “Easy Order” and Credit Card
cannot be set up on the website
unless you’re actually placing an order.
• Give people enough time to talk!
• There’s that grammar coverage issue
again
• Bell pepper = Green pepper
• What synonyms is your user likely to say?
• At some point couldn’t you just give me a list?
• Notice how they screwed up the article +
the item “… adding a parmesan bread
twists”
Meow
Domino’s

• Confirm and correct
• “Do you want to add anything else?”
• “Yes, I want to add peppers.”
• Disambiguation
• “Olives”
• “Ok, we have two kinds of olives. Black olives, or
green olives.”
• A Voice User Interface design is a time-based
interface
• As a designer concerned with user experience
you’re going to be involved in things (such as
pauses) which may not occur to you

How Did Google Home Do?
•“OK Google, Order Dominos”
• “There are stores at….”
• Had to go find the right “App Name” online
•“OK Google, Talk to Dominos”
• “You can link to your Domino’s account…”
• Had a terrible time finding the “Google
Apps.”

•Menu worked!
• System did not recognize “Ham” (Should
offer list of ingredients)
• System became very laggy

• Edited for time
• Original was 3:35
• This is 2:15
• Use of “Dom” persona and male voice
• “Hand off”
• Playback of address:
• Alexa: “Eighty seven twenty three”
• Google: “Eight thousand seven hundred twenty
three”
• Same issue with “twists”
• “Your day just got cheesier”

•Discoverability
• “OK Google, Order Dominos”
•Persona
• Google Home has more control over the
voice
• Branding considerations – “Dom” name and
male TTS
•Playback of Dynamic Data
• Attention to detail – don’t trust the platform
to do it the way you want it

Maintaining State:
•Between dialogs
• “Who is Seth McFarlane?”
• “Seth McFarlane is…”
• “When’s his birthday?”
• “I’m not sure what you’re talking about.”
•From session to session
Oprah Magazine

Home Automation
•Onboarding issues are very similar to
“Skills,” but there is an additional layer of
complexity
• Companies are working to improve the
experience
• After setup, you get a lot of bang for the
buck

“Computer,
turn on the
library lights”

TP Link

Design Considerations: Summary
• Managing access to Skills
(App, Store)
• Managing the Onboarding
Experience
• Discoverability
• Invoking Skills
• Hints
• Confirmation
• Asking Yes/No Questions vs.
Multi-Item Questions
• SSML
• Silences
• Reprompting
• Coverage (prompt vs.
possible input)
• Managing technical
errors
• Timing and Timeouts
• Article matching the
noun
• Confirm and Correct
• Disambiguation
• Persona
• Playback of Dynamic
Data
• Maintaining State

What Makes a Good VUI Designer?
•Concern with the overall experience
• All of the channels that go into making up
how something happens
•Attention to “small” technical details
• Pauses
• SSML
•Writing skills!
• Dialog, not tech doc
• English majors, screenwriters

Session Description
• Amazon Skills for Alexa, Google Actions for Home
– Should your company build a conversational
voice interface for one of these systems, and if
so, how?
• What are the differences between a voice user
interface and other types of UIs? ✔
• What types of skills does a VUI designer need? ✔
• What are some best practices for these VUIs? ✔
• You’ll walk away with answers to the questions
“If, Why, and How” you might choose to explore
this interesting new area of design.
✔

If, Why, How
•What are you trying to build?
•Existing guidelines / research
•User testing is key
• Especially if you’re trying to do something
complicated

If, Why, How: Beyond Skills
Write an app (skill) for
an agent such as
Google Assistant /
Alexa
Use cloud APIs to add
ASR / NLU to your app /
device / page / gadget
Download software and
use full-featured
capabilities for more robust
recognition on a specific
device
Build your own

If, Why, How: What’s the Use Case?
•Enabling application
• User can’t do it any other way
• New tasks
•Enhancing application
• User can do it now
• But speech makes it better
• Faster
• Safer

API-Based
Device-
Based
Roll Your
Own /
Open-
Source
•Flexibility
•Power
•Customization
•Time
•Difficulty

Existing Guidelines / Research
• Caveat: Best practices evolved in one
modality (e.g. voice-only) may not apply the
same way in another (e.g. combined voice +
touch)
• But they could be adapted
• Association for Voice Interaction Design
(AVIxD.org)
• Wiki
• Peer-Reviewed Journal
• Virtual “Brown Bags”
• Academic Sources, Books

AVIxD.org
CUI Working Group is actively recruiting!

@crispinTX
Crispin Reedy
Thank You!

Voice Interface Design: Skills, Actions, Tips for Building Voice Apps

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Voice Interface Design: Skills, Actions, Tips for Building Voice Apps

Similar to Voice Interface Design: Skills, Actions, Tips for Building Voice Apps (20)

More from Crispin Reedy

More from Crispin Reedy (15)

Recently uploaded

Recently uploaded (20)

Voice Interface Design: Skills, Actions, Tips for Building Voice Apps

Editor's Notes