Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects

IBM Total Information Experience
Defining and Evaluating Success:
Metrics and Metrics Frameworks
for Information Architects
Andrea L. Ames
Information Experience Strategist/Architect/Designer
Alyson Riley
TIE Content Strategist
STC Summit
07 May 2013
© IBM Corporation 2013. All Rights Reserved

About Andrea
Technical communicator since 1983
Areas of expertise
Information experience design: Content strategy,
information architecture, and interaction design
for content display and delivery, within products
and interactive information delivery systems
Architecture, design, and development of embedded assistance
(content within or near the product user interface)
Information and product usability, from analysis through validation
User-centered process for information development and
information experience design
IBM Senior Technical Staff Member on corporate Total Information
Experience team in IBM CIO’s office
University of CA Extension certificate coordinator and instructor
STC Fellow, past president (2004-05), former member of
Board of Directors (1998-2006), and Intercom columnist (with Alyson
Riley) of The Strategic IA
ACM Distinguished Engineer 2

3
About Alyson
Technical communicator since 1995
Areas of expertise
Content strategy
Content metrics—the business value of content
Information architecture (my first love!)
Interaction design for content delivery vehicles,
and interactive content
Information and product usability, from analysis
through validation
User-centered processes for content strategy
and scenario-driven information architecture
IBM Senior Content Strategist on corporate
Total Information Experience team in the IBM
CIO Office
Member of STC, and Intercom columnist (with
Andrea Ames) of The Strategic IA

Agenda
Telling the right story through metrics
Telling the right story to the right audience
The importance of a closed-loop metrics process
Evaluation frameworks
Building an evaluation framework
Questions

Jared Spool says it SO well…
Metric: Something you can measure
The time a user spends on a Web site
Whether a particular Web page has a high
“bounce rate” (indicates it’s often the last
page users visit)
How much revenue a Web site makes
How many times the letter E is used on a Web site home page
All of these are measurable…BUT…being measurable
doesn’t make those measures informative or useful

We care, because…
An information architect is (among other things)
a story-teller:
Define the right vision
Tell a compelling, true story that inspires people to buy in
What makes a story true? Facts—things you can prove.
What makes a story compelling? It speaks to what matters most.
What matters most? Depends on your audience. Duh, right?
Prove the value of information architecture and content with metrics
Value is in the eye of the beholder.
Who’s your “beholder?” Understand who your beholders actually are—
that is, the real decision-makers and influencers in your world. (Manage
your stakeholders!)
Use metrics that target actual decision-makers.
Your actual decision-makers are probably business people—executives,
managers, and others who hold the purse-strings.
Figure out what your audience values—their metrics for success.

Problem: Metrics have gotten a bad rap
Numbers can be hard for word people
The right numbers are hard for everyone
Getting metrics to work for you requires a significant shift in thinking
Solution: Rethink metrics
Metrics are another form of audience analysis (who cares about what?)
Metrics are another form of usability testing (what works for whom?)
Motivation for change: Metrics are a powerful tool for getting what you want
(and making sure you want the right things)
Metrics transform opinion into fact
Metrics remove emotion from analysis
Strategize with metrics: Use metrics at every project phase
Beginning: identify opportunity, prove the strategy is right
Middle: show incremental progress, course-correct
End: prove value and earn investment for the future

Be sure you’re talking to the right audience
Audience 1: Business people
Unless you can make a direct connection between your IA metrics and the
metrics that drive business, you are telling the wrong story for this
audience.
You need this audience! The business community funds us. We have to
sell our vision to them, with a metrics story that resonates with them.
We must learn to speak “business”—that is, prove the value of content and
the information experience using metrics that matter to business.
Audience 2: Content people
Typically many kinds of content people will help implement an information
architecture—your work may span departments and business units.
Content people tend to reflect the values of their leadership and business
unit in which they’re located.
This means that even kindred spirits—other content people—can have
widely different goals and metrics.
Your job is to define common ground by speaking to what matters most to
this audience, too.

Selling information architecture to a business audience
The metrics that we use to build effective content
strategies don’t resonate with most executives,
managers, and finance people.
 Sometimes we “talk to ourselves”—that is, use
metrics that resonate with content people, not the
actual people we need to support our strategy.
 “Page hits” resonate with us. “Sales leads”
resonates with business.
You cannot directly connect things like page hits
and bounce rates to core business metrics.
You need an informational professional’s intuition
to know how content supports business metrics—
most business people don’t have that intuition.
The business audience funds us. We have to sell
our vision and prove our value to them, with a
metrics story that speaks to what they care about
most.
Example
business metrics:
Revenue streams
Sales leads
Cost per lead
Customer satisfaction
Customer loyalty
Return on investment (ROI)
Time to value
Market share
Mindshare

Selling content strategy to a content audience
Analyze each team that contributes to your information architecture
In what business unit are they located?
Who are their executives, sponsors, and stakeholders?
Who “grades them” on their performance?
Who funds them?
What matters to them?
How do they measure their progress or results?
What are they doing well (both in your analysis and theirs)?
Where can they improve (both in your analysis and theirs)?
Identify areas of similarity and difference
Where do their goals align with yours? build bridges!
Where do their goals conflict with yours? build business cases!
Use metrics to craft a story that:
Shows problems and opportunities that the content team cares about
Maps in key areas to their goals for content
Diverges from their current goals in ways that would increase their value to sponsors
and stakeholders

Critical: A closed-loop process
Closed loop: end up at the beginning!
Start with metrics—use at project outset to:
Identify problems and opportunities
Define the vision
Prove that the vision is right
Continue with metrics—use during implementation to:
Measure the success of your progress in small increments
Stay on-target through implementation
Determine when it’s time to course-correct (before change gets expensive)
Keep your sponsors and stakeholders engaged throughout the long haul
Ensure that you remain connected to the broader goals and metrics of the
surrounding business
Ensure that you stay responsive and adapt to change
End with metrics—use at project conclusion to:
Prove the business value of information architecture
Prove the business value of your work—enhance your credibility and career
Encourage future investment in information architecture

How do you close the loop?
Measure what you’ve said you’ll measure, of course…
Build a framework, aka, scorecard, to evaluate how
you’re doing
Before you begin: Identify and begin managing stakeholders!
1. Identify what to measure and collect, and determine types *
2. Normalize “results” to “scores” *
3. Categorize and weight metrics *
4. Create summaries *
5. Validate the framework *
6. Launch! *
*Stakeholder-management point

Before you begin: Identify and begin managing stakeholders
Your best political asset—your stakeholders!
A rigorous stakeholder management process will help you take
rigorous advantage of this key asset
Think through the ways that your stakeholders can help you—
start by identifying and analyzing:
Their status relative to your project—advocate, supporter, neutral,
critic, blocker
Their top interests and hot issues
Their key performance indicators (KPIs) and metrics
The level of support you desire from them
The role on your project that you desire for them
The actions that you want them to take (and their priority)
The messages that you need to craft for them to enable the
outcome you want
The actions and communication that you need to make happen
with each stakeholder to achieve your desired outcome
Keep your stakeholder management plan current
“Stakeholder
management is
critical to the
success of every
project in every
organization … By
engaging the right
people in the right
way in your project,
you can make a big
difference to its
success...
and to your career.”
—Rachel Thompson
Source and free
stakeholder
management
worksheet here:
Thompson, Rachel.
Stakeholder
Management:
Planning
Stakeholder
Communication.
MindTools. Web.
12 April 2013.
http://bit.ly/8UnUdj

1. Determine what to measure by considering:
– Who are your stakeholders?
– What is important to them?
– What are your goals?
– What will best help you tell your story to get your
stakeholders aligned around your goals?
1. Determine type
1
Identify what to measure, and determine types

Metrics for an audience of business people (stakeholders)
Use the research you did
during the today-state analysis
phase
Target the key decisions-
makers—those who hold the
purse-strings
Identify what the key business
decision-makers care about
Use language that resonates
with that business audience
Remember: unless you can tie
a particular goal or result to a
measurement that the
stakeholder cares about, that
result ultimately doesn’t matter
Stakeholder Example metrics
Marketing
Executive
 ROI
 Cost per lead
 Campaign performance
 Conversion metrics
Sales
Executive
 Viable leads
 Sales growth
 Product performance
Support
Executive
 Call volume
 Call length
 Customer satisfaction
 Ticket deflection
Development
Executive
 Development costs
 Market share
 Lines of code
 Compliance
 Quality and test results

Metrics for an audience of content people
Map the
people who
deliver
content to the
metrics they
care about
Remember
that each
content team
has their own
decision-
makers who:
Approve their
goals
Determine
their funding
Determine
their futures
Stakeholder Example
metrics
Example associated
content teams
Example
content metrics
Marketing
Executive
 ROI
 Cost per lead
 Campaign
performance
 Conversion
metrics
 Web team
 Social team
 Event team
 Web traffic
 Click-throughs
 Likes and shares
 Conversions
 Collateral distributed
 Cost per unit produced
Sales
Executive
 Viable leads
 Sales growth
 Product
performance
 Sales enablement
 Education & training
 Beta programs
 Proofs of Concept
(PoCs) to sale
 Number of classes
 Beta program
participants
Support
Executive
 Call volume
 Call length
 Customer sat.
 Ticket
deflection
 Web support team
 Call center team
 Amount of web
information produced
 Number of calls reduced
 Time of calls reduced
Development
Executive
 Dev cost
 Market share
 Lines of code
 Compliance
 Quality and test
 Product
documentation team
 Developers who
publish whitepapers
and case studies
 Product community
forums and wikis
 Lines of text, number of
pages, etc.
 Web traffic
 Number of forum
participants

Metrics-based goals
So what are the goals for your information architecture? Express those goals in
the form of business metrics and content metrics. Some examples:
Business metrics Sample IA metrics Sample content goals
Purchase decisions
(revenue)
 Reach—visits, etc.
 Engagement—referrals,
etc.
Contribute to revenue stream
through referrals from
technical content that become
sales leads.
Product quality
(customer loyalty)
 Reach—visits, etc.
 Engagement—referrals,
etc.
Contribute to product quality
through by simplifying the
amount of content in the user
experience.
Customer satisfaction
(ROI)
 Web traffic
 Direct feedback
 Ratings
 Shares (social)
Create high value content that
speeds customer time to
success.
Perceptions of
company (mindshare)
 Sentiment—nature of
social dialogue, etc.
 Direct feedback
Create high quality, highly
usable content delivered in an
elegant information
experience.

What types of metrics are you measuring?
Quantitative
Describe the what, or how many of the what
Can be measured with numbers—absolutely, mathematically
Examples: See the last three charts 
Qualitative
Describe intangibles, like the why
Non-numerical
Examples: Sentiment, how important something is, how much the
respondents like something, how likely they are to recommend
Compliance items
Quantitative—you can count them
But you’re only counting “1”
More important is the perceived value of that “1”
Examples: Models applied, templates used, processes followed

A simple example: Running
Quantitative
Time
Distance
Heartrate
Compliance
Doping
Qualitative
How I feel at start of run
How I feel at end of run
How the weather conditions
were

1. Evaluate qualitative metrics – are scales the same
2. Evaluate quantitative metrics – how good is good
3. Evaluate compliance items – how important is it
4. Select a scale
• Numerical
• Good/bad
• High/med/low (e.g., priority)
• Crawl/walk/run 
Make room for an “innovation” or “exemplary” level—the
“top” of your scale should be attainable by everyone!
Consider:
Do teams have a way to be successful from where
they are? (e.g., can they reasonably get to crawl?)
Will you be able to measure incremental improvement
2
Normalize results to scores

Qualitative metrics: Are the result scales the
same?
Horrible OK Good Great!
What the weather conditions were*
Freezing Cold Cool Warm/humid Hot/humid
* More accurate to measure temperature and humidity as quantitative metrics

Qualitative metrics: Better!
What the weather conditions were*
(Hot/humid or freezing  cool/dry)
* More accurate to measure temperature and humidity as quantitative metrics

Quantitative metrics: How good is “good?”
Time
Beginner: 15 minute mile
Intermediate: 11 minute mile
Athlete: 8 minute mile
Distance: % of goal
25-50%
50-75%
75-100%
Heartrate
Below fat-burning zone
Fat burning zone
Aerobic zone

Quantitative metrics: Normalized
Time
Horrible (implied)  More than 15-minute mile
OK  Beginner: 15 minute mile
Good  Intermediate: 11 minute mile
Great  Athlete: 8 minute mile
Distance: % of goal
Horrible (implied)  Less than 25%
OK  25-50%
Good  50-75%
Great  75-100%
Heart rate
Horrible (implied)  No heart rate 
OK  Below fat burning zone
Good  Fat burning zone
Great  Aerobic zone

1. Determine what to measure by considering:
– Who are your stakeholders?
– What is important to them?
– What are your goals?
– What will best help you tell your story to get your
stakeholders aligned around your goals?
1. Use your IA brain; this is a classic IA problem. 
3
Categorize and weight scores

A possible categorization scheme
Think about how you
want to tell your story for
most impact
What is your clients’
product-use life cycle?
Other possible schemes
By function (to make
specific business cases to
them)
By “standard” (to
understand specific
compliance issues)
A generalized view of
IBM’s product-use lifecycle

1. Determine what summaries you need to tell the
right story at a glance
• For management
• For the larger organization
• For the team being measured
1. Be careful not to create an unwieldy mess 
4
Create a summaries

This could be an entire session by itself!

1. Find as many diverse “products” as possible to
measure.
2. Gut check with several “independent” IAs
• Individual metrics
• Categories
• The whole
1. Compare results across products for
consistency
2. Compare results to uncover gaps: Is something
note being measured that should be?
3. If possible, find other frameworks to compare to
your approach—not necessarily an IA
framework…maybe a user experience or
support framework
5
Validate framework

1. Determine how the framework will be used
• As a self-evaluation tool?
• In an objective, outside evaluation?
• Do users need to be “certified” to use it? Or
evaluators to evaluate with it?
1. Communicate!
2. Educate!
3. Consult?
6
Launch!

Questions?

References
Jared Spool: KPIs are metrics, but not all metrics are KPIs from UIE
Brainsparks blog, Oct 5, 2012: http://bit.ly/VFYvF2
Bhapkar, Neil. 8 KPIs Your Content Marketing Measurements
Should Include. Content Marketing Institute. Web. 12 April 2013.
http://bit.ly/Wnb7Cy
Klipfolio. The KPI Dashboard—Evolved. Web. 12 April 2013.
http://bit.ly/LhzeL9
Muldoon, Pamela. 4 metrics every content marketer needs to
measure: Interview with Jay Baer. Content Marketing Institute.
Web. 12 April 2013. http://bit.ly/X8IvMJ
Thompson, Rachel. Stakeholder management: Planning
stakeholder communication. MindTools. Web. 12 April 2013.
http://bit.ly/8UnUdj

Proving the business value of content—IBM example
Shameless
ad:
Watch for the
May issue of
STC’s Intercom
magazine for a
new article that
we wrote on
proving the
business value
of content.
At IBM, we’re learning to tell a better story for a business audience
We conducted a survey from 2010-2012 with clients and prospective
clients about the value of content—here’s the hot-off-the-press data:

A map of content metrics to stakeholder metrics
Tie your IA metrics to the metrics that matter most to your stakeholders
so you can tell a story that inspires the outcomes you want.
This means researching how content influences the metrics that are
most important to the specific people you need for success.
Start your research with these hints:
How does content speed user
success and time-to-value?
direct link to customer value
How does content drive
purchase decisions?
direct link to the revenue stream
How does content impact
product quality?
direct link to customer loyalty
How does content influence
customer satisfaction?
direct link to ROI
How does content shape clients’
perceptions of your
company?
direct link to mindshare

Evaluation frameworks: Heuristics and heuristic evaluations
Heuristics
Rules of thumb
No one single “rule” or set of “rules”
Heuristic evaluations
Inspection method
Performed by “experts”
Based on heuristics
Both: Based on principles derived from observing and
analyzing user actions and intentions
These principles enable development of your own
evaluation framework, based on your own user data

Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (10)

Similar to Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects

Similar to Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects (20)

More from Andrea L. Ames

More from Andrea L. Ames (20)

Recently uploaded

Recently uploaded (12)

Defining and Evaluating Success: Metrics and Metric Frameworks for Information Architects

Editor's Notes