Discussion of why it is sometimes desirable to have more than the required (either in terms of validity or statistical significance) number of participants in card sorts and similar usability exercises. Specifically, how large-scale usability exercises allow usability professionals and site owners to reach out to a broad audience and include them in the usability lifecycle.

1. It starts with a guy introducing himself…
Hey, I’m Jeff Pass.
I’m the one that mentioned
male impregnation during my
presentation at last year’s UX
Camp DC… remember me?

2. Then he makes his pitch…
I’m working on a presentation for the IA Summit.
It’s called, “When Too Many Is Just Enough: Citizen
Engagement and Federal Government Websites.”
I’d like to workshop part of it with you.
Here’s the (boring) summary:
Statistical significance is an important
consideration in usability studies. You need a
certain minimum sampling to ensure valid
results, but at the same time, too large a sampling
creates more work without increasing significance
or validity. Nevertheless, there is a case to be made
for casting a wide net and engaging a larger
audience in order to actively engage users.

3. Then he gets to the point...
So today I want to talk about online card sorts.
Open, closed, reverse, it really doesn’t matter…
any kind of card sort you like as long as it is online.
More to the point, I want to talk about how many
participants you want/need.

4. So, you’ve got yourself a card sort…

5. How many participants do you need? (1)
According to Optimal Workshop, for an open card sort you need:

6. How many participants do you need? (2)
According to Optimal Workshop, for a closed card sort you need:

7. How many participants do you need? (3)
And there are lots of other opinions…

8. So how many did we do?
???

9. Over 1109 (1110 to be precise)

10. But why? What were you thinking?
The 2012 Digital Government
Strategy calls for large-scale
citizen engagement.
Increased participation doesn’t
increase validity (and can
complicate analysis and
reporting), but it does allow us to
directly engage the wider public
in usability studies and give them
a voice in organizing the (federal)
websites that are meant to serve
them.

11. And how did you engage them?
Directly…
Social media was our recruiter
A blog post was our screener
OptimalSort and TreeJack were our test vehicles*
*And yes, we did include a free text comment option and yes, analyzing the results was
onerous, but the additional effort was well worth it because while quantitative data (the sort
results) had pretty well stabilized by 25-30 participants, every comment provided valuable
qualitative data.

12. So what do you think?
Share your thoughts and experiences about large-
scale usability studies and direct user engagement
But enough about me; what do you think about me?
Err… I mean… what do you think about large-scale
usability studies, direct user engagement, and too
many being just enough?

13. Thanks for your time & participation!
Jeffrey Ryan Pass
Lead User Experience Consultant
Aquilent (www.aquilent.com)
jeff.pass@aquilent.com
@jeffpass
UX Camp DC 2013