A draft theory of how to design effective multimedia & interactive content, with brief case studies.

1. Toy Systems
and
Taco Trucks
Case studies of effective
multimedia & interactive content
Original Client: HHMI / Tangled Bank Studios
Prepared by: John Pavlus / Pavlus Office
Updated: 12 Dec 2015

2. Introduction: Attention vs. Engagement
The most attention-grabbing interactive and multimedia
journalism of the past few years — from The New York Times’
Snow Fall to The National Film Board of Canada’s One
Millionth Tower — dazzled web users (and the press) with
immersive interfaces and technically innovative formats. But
did these experiences drive sustained engagement with
the content?
The answer seems to be “no.” Few such features continue to
be regularly produced. The technical complexity of producing
them could be a limiting factor, but it’s more likely that readers
just don’t find them all that compelling in the long run.
Meanwhile, in 2014 Buzzfeed quietly introduced a new
interactive component called a “slider” to its already
multimedia-heavy content. The slider was nothing more than a
vertical line that, when “dragged” horizontally across an image
via the mouse, revealed another image “underneath”. No
breathless media coverage followed.
What did follow was massive engagement. Posts with the
slider garnered millions of views, and the format has been
sustainable: a year later, Buzzfeed regularly deploys the slider
not just to augment celebrity-driven clickbait, but to
enhance its explanatory and public-service journalism.
Attention does not necessarily translate to engagement.
The following case studies identify some values that do drive
sustainable engagement with interactives and multimedia.
They don’t demand expensive tech or flashy functionality; nor
do they encourage superficiality or “dumbing down” of content.
They do require, however, some new definitions.

3. Effective Interactives as Toy Systems
Which object best fits with your common understanding of the
word “interactive”: a flight manual or a flight simulator? Both
describe the behavior of a thing (the airplane), but only the
latter does so by dynamically modeling a system so that:
• you can freely explore its behavior
• by manipulating its features
• and responding to its feedback.
Which system would you rather use to learn more about how
airplanes work: a NASA flight simulator or an educational video
game? Both are interactive, but only the latter is a “toy”– i.e.,
a version of the system that is:
• casual (easy to start/escape/resume at any point; requires
few or no special prerequisites),
• bounded (the “rules” or “edges” of the system are legible
and evident),
• forgiving (may have robust “help” designed into it; errors
are easily reversible or have negligible cost).
Content that doesn’t dynamically model a system isn’t
meaningfully “interactive”; and systems that aren’t
appropriately toylike lack appeal as content. Interactives are
most effective when, like a Rubik’s Cube, they support
“meaningful fiddling around.”

4. Effective Multimedia as Taco Trucks
A wise man once wrote that the web is primarily a customer
service medium. Form and content bow to context and
convenience. People may want to consume rich media online,
but what they want more is to simply avoid hassle.
And yet hassle is what most multimedia “experiences” are
made of. Waiting through pre-rolls or prologues. Evaluating
ambiguous options and quality. Parsing unfamiliar layouts.
Consuming multimedia is like ordering from a fussy waiter at a
restaurant, when it should be like eating from a food truck:
• up-front: clear, WYSIWYG value proposition
• familiar: approachable & satisfying vs. “innovative”
• cheap: low-commitment in time/attention
• easy: minimal interactivity, simple to sample/share/move
Like any good taco truck, successful multimedia presents a
simple but compelling offer: low stakes, high quality.

5. Case Study: TypeDrummer
WHAT IT IS: interactive music generator
WHAT IT DOES:
• maps electronic percussion samples to ASCII keyboard input
• generates real-time audio loops of percussion
• creates shareable URLs for audio loops
WHO MADE IT: Kyle Stetz (independent designer/developer)
WHAT WORKS:
Typedrummer is a toy system par excellence. The simple
interface is inviting and instantly understandable. The name,
illustration, and text box provide clear cues for what to do, how
to do it, and what to expect. The system is simple enough to
idly “fiddle with” but rich enough to generate novel patterns
and reward exploration of its deeper structure & relationships if
the user wishes to do so.

6. Case Study: TypeDrummer
“TOY SYSTEM” VALUES
• Direct manipulation with dynamic feedback: supports
“meaningful fiddling around”
• Bounded: One input box dominates the user interface.
Secondary functionality is limited to two explicitly labeled
text-buttons
• Forgiving: Instructions make clear what the “first move” is.
Large type makes it easy to see and change the system’s
state. No error messages for “breaking the rules” — the
system ignores inappropriate input without interrupting any
musical feedback already in progress
• Casual: familiar, available controls (i.e., typing) and dynamic
response (i.e., no “execute” button or mode mediating the
interactivity) mean that interaction can unfold on the fly
“TACO TRUCK” VALUES
• Up-front: simple layout, clearly worded
• Familiar: interface is just text and links
• Fast: plain-text URL sharing is universal & idiot-proof
• Cheap: loads instantly; single keystrokes generate
meaningful feedback

7. Case Study: The Ladder of Abstraction
WHAT IT IS: an interactive lesson / demonstration of how to
design effective interactive lessons / demonstrations
WHAT IT DOES:
• Describes fundamental concepts in the left column, and
models them with interactive examples in the right column
WHO MADE IT: Bret Victor (independent designer/developer)
WHAT WORKS:
The interactive richness of The Ladder of Abstraction is difficult
to convey in a static document like this — which is precisely its
creator’s point. I’m including it as a counterpoint to
Typedrummer, to show how “toy systems” can be deployed to
communicate powerful, subtle, complex ideas that are not
necessarily “toylike” themselves: abstraction, emergence,
algorithms, simulation, data structures, and the like.
The interactive models are info-dense but always
approachable and lucid. Thoughtful details reduce the
cognitive “noise” of manipulating the interface (e.g., sliders
move automatically when moused over, without requiring an
awkward “click and drag” gesture), making it much easier to
engage with the increasingly challenging concepts while
simultaneously reinforcing them.

8. Case Study: The Ladder of Abstraction
“TOY SYSTEM” VALUES
• Explorable via direct manipulation and dynamic feedback
• Bounded: visualizations are confined to simple, credit-card
sized cartoons that can be assessed at a glance; most
cartoons model just one parameter of the system at a time
• Casual: click-and-drag mouse gestures are automated,
creating effortless interactions that free up attention for
observing how the system behaves
• Forgiving: Clear instructions, multiple visual cues, and
synchronized animations take pains to keep the user
oriented within the system at every move; no “getting lost”
“TACO TRUCK” VALUES
• Up-front: Simple (but well crafted) 2D graphics
• Familiar: skeuomorphic controls reassure the user that
they’re in “safe territory” and they won’t “break” anything
• Easy: Small visual “chunks” are easily scannable /
browsable; right column is not overloaded with them
• Cheap: Mousing over graphics “activates” them; otherwise
they maintain their last state

9. Case Study: “Species in Pieces”
WHAT IT IS: A visual carousel of 30 endangered species
WHAT IT DOES:
• Depicts each species as a cluster of animated 3D polygons
• Displays conservation status statistics as infographics
• Displays fact-sheets and short videos about each species
WHO MADE IT: Bryan James (independent designer/
developer)
WHAT DOESN’T WORK:
“Species in Pieces” caught the attention of design-savvy
science nerds for its gorgeous visuals and impressive-looking
animations and infographics. But it’s not “interactive” in any
meaningful way. Take away the sexy animated transitions, and
it’s basically equivalent to a Powerpoint deck. It’s a relatively
thin content experience as well.
That said, dynamically modeling the behavior of an interesting
system was probably not “Species in Pieces”’s goal (it’s billed
as a digital “exhibition”). And it does embody one of the values
of effective multimedia: it’s quick to load and very responsive.
As a gallery of animated data posters (and a demonstration of
the expressive power of CSS), it’s undeniably effective; as an
engaging and rich content experience, less so.

10. Case Study: “Species in Pieces”
“TOY SYSTEM” VALUES: not applicable
• “Interactivity” limited to navigating between static content
cards and selecting menu options
“TACO TRUCK” VALUES:
• Fast: can navigate quickly, and easily get “in” and “out” of
content once the interface is learned
WEAKNESSES:
• Inscrutable interface elements
• All-caps body typography discourages close reading
• Chartjunk & difficult-to-interpret infographics

11. Sidebar: Non-Interactive Multimedia
The previous case studies show that meaningfully interactive
content can be challenging to create, especially when mere
content navigation is often (and mistakenly) “promoted” to the
status of interactivity.
But effective multimedia doesn’t even need to be
interactive. In fact, non-interactivity might be multimedia
content’s ideal stance, as the following examples demonstrate.
• A science explainer that reimagines the web slideshow as a
grown-up storybook, by The New York Times
• A map of the Silk Road in China that matches photos and
video loops to locations as the user scrolls down the page,
by The New York Times
• A “filmstrip” of captioned GIFs that sets up a visual metaphor
to explain your brain on drugs, by PBS NewsHour

12. Conclusion: Meaning > Marvels
It’s easy to assume that in the overcrowded marketplace of
attention online, the best way to communicate a message is to
wrap it in the most attention-commanding package you can.
With interactive and multimedia content, this approach often
takes the form of “innovating” — coming up with some novel
way to format or present information.
But while people may like this kind of novelty-driven
“innovation”, what they want is something different, simpler.
They want what they do online to make more sense.
The litmus test for evaluating the effectiveness of multimedia or
interactive content is not: Will they think this is cool?
But rather: Will this make what they’re doing, right now,
make more sense?
The answers may surprise you.
Marvels
I’m pretty ambivalent about this new strain of multimedia
journalism. … These kinds of things, I think, are meant to
be marveled at more than they are meant to be read.
Khoi Vinh, former design director of NYTimes.com
Ignore the structure of current [software] designs, and ask
only, “Why is a person using this?” Abstracted, the
answer almost always is, “To learn.”
Bret Victor, author of “Up and Down the Ladder of Abstraction”
Use the SLIDEY THING to investigate these doppelgängers
for yourself!
John Gara, author of BuzzFeed’s “34 Celebrities Who Share
The Same Face”