By @mrjoe http://mrjoe.uk The poet John Keats famously blamed scientists experimenting with light for 'unweaving the magic of the rainbow'. Joe will look at applying science to design to make our apps and websites better. We'll look at different types of data, from user research and analytics, to psychology. How to research, collect, source, asses and most importantly design using data without losing the magic.

1. @mrjoe
How to design with science
January 8, 2015
Joe Leech
Waving einstein
How to
design with
science and not
destroy the magic
@mrjoe

2. Hi, I’m @mrjoe
but you can
call me Joe

3. @mrjoe
⅓ ⅓⅓
Research DesignDigital
Strategy

4. @mrjoe
[Newton] has destroyed
all the poetry of the
rainbow, by reducing it
to the prismatic colours.
– John Keats

5. @mrjoe
The product goes
live and it's lost
it’s magic
The team
/ client
continually
questioning your
choices
Being
micromanaged by
your boss / client
Ever heard these
issues? Hands-up?
I had the first one so
badly for about a year, I
called it the curse of
Joe.

6. @mrjoe
It started with a need
6
http://
www.cxpartner
s.co.uk/cxblog/
the_myth_of_th
e_page_fold_ev
idence_from_us
er_testing/
It started with a need. I had heard the
same thing again and again.
I wrote this, have you read it? Almost
half a million of you have.

7. @mrjoe
Here’s the data, from an eyetracking study. It shows
2 designs, one to the left with stuff crammed above
the fold. The one on the right where things are
spaced out more. People explored more when there
was less stuff crammed above the fold.
It showed me there was a need for evidence,
reasoning and science to support the design
process.

8. @mrjoe
Design is not so
much a design issue
as a power struggle.
– Alan Cooper
Alan cooper, created visual basic amongst
other things, teaches design leadership.
Harsh words!
So what do we do?

9. @mrjoe
Design is the process of
choosing and organising code
words, images and messages
into a form that communicates
and influences its audience
– @mrjoe + The UK Design
council

10. Talk stories.

11. @mrjoe
Here’s some research I did many years
ago for the trainline. What does this
photo show?
This one photo saved countless time for
thousands of people.

12. @mrjoe
2.
1.
It tells us two things.
1. When people travel they are often
carrying loads of stuff
2. most people have a phone and are
looking for the conformation email.
They don’t print it out.
We took the confirmation number and
put it as the email subject line.

13. @mrjoe
I didn’t see them [the asterisks].
There’s nothing that explains
what they mean.
How did you hear about us?
*
How old are you? *
I’ve heard this a few times in user research. What does it mean? Not
enough people to make it statistically significant but enough to change my
approach to forms.

14. @mrjoe
How did you hear about us?
How old are you?
Optional
We don’t mark mandatory, we mark
optional.
It’s enough to get me to change my design
approach. Is there enough evidence to prove
it from just a handful of users? No, it requires
another set of supporting data.
Which is why we need to use each approach
in combination with another.

15. Talk business.

16. @mrjoe
h‫מּ‬p://www.slate.com/articles/technology/technology/2014/06/
online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html?
utm_content=bufferfa367&utm_medium=social&utm_source=twi‫מּ‬er.com&utm_campaign=buffer
http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg
In 2003 Mel Karmazin, a Viacom executive. Viacom are a huge media
company where the bulk of their income is from advertising, visited the
Google offices. He met Sergey and Larry who preceded to tell him all the
ways they could measure and track advertising online.
His one comment:

17. @mrjoe
You’re f---ing with
the magic!
– Mel Karmazin, Viacom executive
h‫מּ‬p://www.slate.com/articles/technology/technology/2014/06/
online_advertising_effectiveness_for_large_brands_online_ads_may_be_worthless.single.html?
utm_content=bufferfa367&utm_medium=social&utm_source=twi‫מּ‬er.com&utm_campaign=buffer
http://carolinavega.com/web%20documents%20and%20pics/oreo.jpg

18. @mrjoe
http://
justinmrao.com/
lewis_rao_nearim
possibility.pdf
On the Near Impossibility of Measuring the
Returns to Advertising⇤
Randall A. Lewis
Google, Inc.
ralewis@google.com
Justin M. Rao
Microsoft Research
justin.rao@microsoft.com
April 23, 2013
Abstract
Classical theories of the firm assume access to reliable signals to measure
the causal impact of choice variables on profit. For advertising expenditure
we show, using twenty-five online field experiments (representing $2.8 mil-
lion) with major U.S. retailers and brokerages, that this assumption typically
does not hold. Statistical evidence from the randomized trials is very weak
because individual-level sales are incredibly volatile relative to the per capita
cost of a campaign—a “small” impact on a noisy dependent variable can gen-
erate positive returns. A concise statistical argument shows that the required
sample size for an experiment to generate su ciently informative confidence
intervals is typically in excess of ten million person-weeks. This also implies
that heterogeneity bias (or model misspecification) unaccounted for by ob-
servational methods only needs to explain a tiny fraction of the variation in
sales to severely bias estimates. The weak informational feedback means most
firms cannot even approach profit maximization.
This paper was published. The study was undertaken written when Lewis
and Rao were at Yahoo.
What does this tell us? There is no proof this works. It’s built on faith.

19. @mrjoe
The 5 basic business models.
19
http://mrjoe.uk
Advertising
Affiliate
feesSelling
Subscription
/ Licensing
Market
place
What does that mean for you. You need to understand
some business basics. I wrote an article. more coming
soon.
These are the 5 basic income streams for organisations.
Understand how they generate money and you have
more science to base your designs on.
Read more at http://mrjoe.uk

20. @mrjoe
Strong ideas,
weakly held.
http://mrjoe.uk/strong-ideas-weakly-held/

21. Talk data.

22. @mrjoe 22
5 years ago. I was a single man. At a point in life where it was becoming
harder to meet new people. Someone suggested internet dating. I thought
wow, ok, I can do this. My UX background is perfect for meeting the perfect
person.

23. @mrjoe
http://blog.okcupid.com/
index.php/the-4-big-myths-of-
profile-pictures/
I went in search of data, OK Cupid has a blog. Full of great insights, like what
is the perfect profile picture.
I set about getting all UX on it, user researching my profile. At one point I
emailed OK Cupid and asked if I could drop a google analytics tag my page.
I learned a lot about how data can inform design. I also met my wife.

24. @mrjoe
Classic
-Conversion (sales / visits)
-Product page conversion
(sales / visits to product page)
-Funnel analysis
-Basket abandonment
-Bounce rate
-Sales
-Leads
-Subscribers
-Unique visitors
-Returning visitors
-Page views per visit
-Visit to order ratio
-Load time
-Registrations
-Visit / session length
-Page views per visit
-Time on page
-Time on site
-Form abandonment
-Failed internal search
-Referring pages / links
-Geographic locations
-Print page
Banners:
-Click through rate
-Impressions
Financial / sales / business
-Average order value
-Basket value
-Profit margin
-Average sales price
-Cross sell
-Gross margin
-Category margin
-Cost per lead
-Customer acquisition cost
-Lifetime customer value
-Average customer value
-Membership / subscription
churn
RSS
-Feedburner subscriptions
-Shares on Google Reader
Call centre calls
-Average call length
-Support vs sales calls
-Inbound vs outbound calls
-Web generated calls (unique
number on website)
-Web fulfilled information calls
SEO
-SEM keyword value
-SEO positioning
-Changes in SERP results/
rankings
-Top entry pages
-Number of keywords
triggering results for your site
-Number of clicks to your site
from keywords
-Google trends
-Inbound links (back link
discovery)
-Percentage share of each
engine
-Branded vs non-branded
searches
-Affiliate links
-Affiliate fees
Social media
-Facebook referrals
-Incoming Twitter links
-Facebook sends/shares/
mentions
-Facebook likes
-Facebook fans
-Facebook fan rates
-Tweets
-Retweets
-@s on Twitter
-Twitter followers
-Twitter follow rate
-Google +1s
-Bookmarks on Delicious
-StumbleUpon thumbs up
-StumpleUpon reviews
-Diggs
-Google BlogSearch links
-Blog comments
-Blog articles
-Video views
-Youtube favourites
-Youtube channel
subscriptions
-Youtube channel comments
-Youtube video reviews
-Slideshare views
-Forum mentions
-Thread size
-Online review mentions
-Stars in reviews
-Bit.ly / URL shorteners
usage / clicks
Third party / benchmarks
-Comscore
-Hitwise
-Alexa
-Compete (US)
Email & campaigns
-Email newsletter churn
-Email sign-up
-Emails sent
-Emails bounced (bad
address)
-Email forwards
-Email campaigns
-Open rate
-Delivery rate (sent - bounces)
-Click through rate (CTR)
-Email related to conversion /
other metric
-Unsubscribe rate
-SMS subscribers
-Via print publication / 3rd
party (unique URLs)
Internal search
-Search no results
-# Search 1 - 10 results
-# Search 10 - 25 results
-# Search over 50 results
IA:
-GOMS technique
-Time to content
-Clicks to content
-Back button clicks
-Task completion
-Errors
Testing metrics:
-Words Recognition Rate
-Reported expectations and
performance
-Facial reaction
-Number of back presses
-Gap satisfaction
-Path's taken as a measure of
scent
-Work-flow matches mental
model or not
Satisfaction measurements:
-Net Promoter Score
-ASQ PDF: After Scenario
Questionnaire (3 Questions)
-NASA-TLX : NASA's task load
index is a measure of mental
effort (5 Questions)
-SMEQPDF: Subjective Mental
Effort Questionnaire
-UMEPDF : Usability
Magnitude Estimation
-SEQ PDF: Single Ease
Question
A bewildering amount of metrics. Here’s some I collected over a couple of
years. This can be dangerous, organisations can be tied to data, refusing to
make a decision without it.

25. @mrjoe
I had a recent debate over whether a
border should be 3, 4 or 5 pixels
wide, and was asked to prove my
case. I can’t operate in an
environment like that…
… I won’t miss a design philosophy
that lives or dies strictly by the sword
of data.
– Doug Bowmen
http://stopdesign.com/archive/2009/03/20/goodbye-google.html
This is taken from Doug Bowmen’s blog, after he left
Google, he was their first visual designer. Pretty damning.

26. @mrjoe
Classic
-Conversion (sales / visits)
-Product page conversion
(sales / visits to product page)
-Funnel analysis
-Basket abandonment
-Bounce rate
-Sales
-Leads
-Subscribers
-Unique visitors
-Returning visitors
-Page views per visit
-Visit to order ratio
-Load time
-Registrations
-Visit / session length
-Page views per visit
-Time on page
-Time on site
-Form abandonment
-Failed internal search
-Referring pages / links
-Geographic locations
-Print page
Banners:
-Click through rate
-Impressions
Financial / sales / business
-Average order value
-Basket value
-Profit margin
-Average sales price
-Cross sell
-Gross margin
-Category margin
-Cost per lead
-Customer acquisition cost
-Lifetime customer value
-Average customer value
-Membership / subscription
churn
RSS
-Feedburner subscriptions
-Shares on Google Reader
Call centre calls
-Average call length
-Support vs sales calls
-Inbound vs outbound calls
-Web generated calls (unique
number on website)
-Web fulfilled information calls
SEO
-SEM keyword value
-SEO positioning
-Changes in SERP results/
rankings
-Top entry pages
-Number of keywords
triggering results for your site
-Number of clicks to your site
from keywords
-Google trends
-Inbound links (back link
discovery)
-Percentage share of each
engine
-Branded vs non-branded
searches
-Affiliate links
-Affiliate fees
Social media
-Facebook referrals
-Incoming Twitter links
-Facebook sends/shares/
mentions
-Facebook likes
-Facebook fans
-Facebook fan rates
-Tweets
-Retweets
-@s on Twitter
-Twitter followers
-Twitter follow rate
-Google +1s
-Bookmarks on Delicious
-StumbleUpon thumbs up
-StumpleUpon reviews
-Diggs
-Google BlogSearch links
-Blog comments
-Blog articles
-Video views
-Youtube favourites
-Youtube channel
subscriptions
-Youtube channel comments
-Youtube video reviews
-Slideshare views
-Forum mentions
-Thread size
-Online review mentions
-Stars in reviews
-Bit.ly / URL shorteners
usage / clicks
Third party / benchmarks
-Comscore
-Hitwise
-Alexa
-Compete (US)
Email & campaigns
-Email newsletter churn
-Email sign-up
-Emails sent
-Emails bounced (bad
address)
-Email forwards
-Email campaigns
-Open rate
-Delivery rate (sent - bounces)
-Click through rate (CTR)
-Email related to conversion /
other metric
-Unsubscribe rate
-SMS subscribers
-Via print publication / 3rd
party (unique URLs)
Internal search
-Search no results
-# Search 1 - 10 results
-# Search 10 - 25 results
-# Search over 50 results
IA:
-GOMS technique
-Time to content
-Clicks to content
-Back button clicks
-Task completion
-Errors
Testing metrics:
-Words Recognition Rate
-Reported expectations and
performance
-Facial reaction
-Number of back presses
-Gap satisfaction
-Path's taken as a measure of
scent
-Work-flow matches mental
model or not
Satisfaction measurements:
-Net Promoter Score
-ASQ PDF: After Scenario
Questionnaire (3 Questions)
-NASA-TLX : NASA's task load
index is a measure of mental
effort (5 Questions)
-SMEQPDF: Subjective Mental
Effort Questionnaire
-UMEPDF : Usability
Magnitude Estimation
-SEQ PDF: Single Ease
Question
Measure
2 to 3
things.
Each has:
1. Timescale
2. Benchmark
3. Reason to be
reported
4. Associated
action
http://mrjoe.uk/ux-and-roi-what-to-measure-and-what-to-expect/
Truth is it’s important to measure. It shows confidence in your design.
Here’s some tips to get it right. See a talk by me on this here http://mrjoe.uk/
ux-and-roi-what-to-measure-and-what-to-expect/

27. @mrjoe 27
Data shows
confidence in the
design

28. @mrjoe http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/
The squirrel evolved bit by bit to fly. Small incremental
improvements that allowed him to jump further, land better, and
eventually to glide. The squirrel has reached a point where he isn’t
going to get better at flying (called a local maximum).

29. @mrjoe
The soaring eagle. Elegantly drifting
through the air with little effort. The
eagle also evolves, little by little to its
own maxim. It’s a superb flyer, soaring
to 15,000 feet instead of making long
leaps between trees. The eagle started
from a better set of basics: front limbs
to wings, reptile scales to feathers.
Good design gives a good starting
point for evolution. MVT can get you to
a maxim, but it can’t help you choose
the best starting point to get to the
highest maxim. A strong design leads
to elegant flight. A weak design leads
to hopping between branches. We
need to combine MVT with craft. Read
more: http://www.cxpartners.co.uk/
cxblog/
multi_variant_testing_and_the_flying_

30. @mrjoe
Sore like an eagle
don’t leap like a
squirrel
http://www.cxpartners.co.uk/cxblog/multi_variant_testing_and_the_flying_squirrel/

31. Talk theory.

32. @mrjoe
James Vicary, in 1957, flashed up hungry eat
popcorn, thirsty drink coke as single frames
during a movie showing.

33. @mrjoe
58%A reported 58% increase in sales of popcorn.
But it was later found out to be a hoax. He made it up to
prop up his failing advertising business.

34. @mrjoe
Some examples, but there is no proof.
http://www.badabingrecords.com/2011/03/the-subliminal-
scares-a-short-history-of-an-american-obsession/
http://www.businessinsider.com/subliminal-ads-2011-5?op=1

35. @mrjoe
Strikingly, there
was no
significant effect.
http://www.bbc.co.uk/news/magazine-30878843
The media’s obsession. They were shown the same three-
minute clip from the BBC/Kudos drama Spooks twice, but
each time one group wore the blindfolds. The clip watched
by the group with red blindfolds contained a 10-millisecond
flash, every five seconds, of the word “Lipton", http://
www.bbc.co.uk/news/magazine-30878843
So what can we do?

36. @mrjoe
There are resources out there that help with theory. This is
from Ribot, alumni of my psychology for designers
workshop. http://coglode.com/
Workshop: http://mrjoe.uk/psychology-design-ux-
workshop/

37. @mrjoe
PsychologyForDesigners.com
Or you can buy my book, I show you how to find a theory
and apply it to design. How to use psychology to advocate
design.
PsychologyForDesigners.com

38. @mrjoe
Title Text
38
A designer
who doesn't
understand
psychology
is going to be no
more successful
than an architect
who doesn't
understand physics
Image: http://victorenrich.com/archives/155
@mrjoe

39. @mrjoe
The biggest myth
ever perpetuated
in the design field
is that good
design sells
itself. – Mike
Montiero

40. Talk objectively.

41. @mrjoe
'I feel the design,
minimalist in style,
reflects a modern
design aesthetic’
- The Designer

42. @mrjoe
‘Well I feel more that
it’s a website and not
a f#cking coffee table.’
- The Product
Manager

43. @mrjoe
'I feel the design,
minimalist in style,
reflects a modern
design aesthetic’
- The Designer
What was wrong with that statement?
Anytime you say I feel, it leaves the door open for the ‘Well
I feel’. It becomes a battle of wills, there is no subjectivity
to judge the design against.

44. @mrjoe http://www.gensleron.com/storage/post-images/cities/reinventingaclassicreuniontowerobservationdeckreopens/1.jpg
Which reminds me, I was once looking over a beautiful view, but I got annoyed at the poor
information design of the support panel. My wife simply said, chill the f$ck out and enjoy
the view, the view is the focus not the design of the information panel.

45. @mrjoe
Art: …works to be appreciated
primarily for their beauty or
emotional power.
– Merriam Webster Dictionary

46. @mrjoe
http://
www.ise.bgu.ac.il/
faculty/noam/papers/
00_nt_ask_di_iwc.pdf
Art does support design. Pretty things are easier to use.

47. @mrjoe
What is beautiful is
usable.

48. 1. Talk stories
2. Talk business
3. Talk data
4. Talk theory
5. Talk objectively

49. @mrjoe
mrjoe.uk