How do you measure API success? What KPIs do APIs need? What mistakes should I avoid? Find out what you should, and shouldn't, be measuring as part of your API program in this Business of APIs Conference NYC talk. Dive into a breadth of API metrics, the 6 keys to better API metrics, and the traps to beware of (the important do's and don'ts). Also real-world API case studies show who measures what.

1. KPIs for APIs
or,
why API Calls are the new Web Hits
and you may be measuring all wrong
John
Musser
/
@johnmusser
/
API
Science
/
apiscience.com
Business
of
APIs
Conference,
NYC,
2014

2. Why do we need
a Web site?
1995
Of course we have
a Web site
Of course we
have an API
Why do we need
an API?
2000 2005 2010

3. Why do we need
a Web site?
1995
Of course we have
a Web site
You are here
Of course we
have an API
Why do we need
an API?
2000 2005 2010

4. Remember hit counters?

5. Evolution of web site metrics
Hits
Page
Views
Visitors
Conversions

6. Evolution of API metrics?
Calls
?
?
?

7. Great APIs get measured

8. But measure
what?

9. Well,
it depends…

10. Why
do you have an API?

11. Marketing channel BizDev / LeadGen
New line of business
Extend product
Drive innovation
User acquisition
Upsell opportunity
Device and mobile support
Content acquisition
API as Product
Distribution channel
Increase stickiness
Accelerate internal projects
Partner opportunities
Drive traffic
Increase footprint

12. Who
is your API for?

13. API consumer continuum
You
Your partners
& customers
Everyone
else
Internal Private Open

14. The answers to
who & why
should drive your KPIs

15. The answers to
who & why
should drive your KPIs

16. Many ways to think about API KPIs…
Traffic
Total calls
Top methods
Call chains
Quota faults
Developers
Total developers
Active developers
Top developers
Trending apps
Retention
Service
Performance
Availability
Error rates
Code defects
Marketing
Dev registrations
Dev portal funnel
Traffic sources
Event metrics
Support
Support tickets
Response times
Business
Direct revenue
Indirect revenue
Market share
Costs

17. More ways to think about API KPIs…
Customer
Satisfaction
NPS
Churn
Community
Forum activity
Social media
(both for API and
apps built on it)
Quality
Defects
Error prone APIs
Repair rate
API Maturity
Stability
Maturity metric
Change rate
Innovation
Number of products
Number of apps
Speed to market
Channel
Call volume: Mobile,
web, others
Revenue by channel

18. There are a lot of
API KPIs
you could measure

19. Don’t drown in API KPIs

20. KPI SECRET #1
Great APIs
prioritize
what to measure

21. Choose your “KPI lens”
CFO lens
CMO lens
CEO lens
Ops lens
QA lens
CRO lens
PM lens
PR lens

22. CFO lens: financial metrics
Total direct and indirect revenue
Margin
ROI
Marginal cost
Capex / Opex investment
CLTV per developer

23. DevOps lens: operational metrics
Performance
Availability
Error rates
Faults by API type
Defect backlog
API traffic trends

24. CMO lens: marketing metrics
Developer acquisition cost
Developer demographics
Portal visitor count & traffic sources
SEO / SEM metrics
Social media metrics
Evangelism and event metrics

25. KPI SECRET #2
Know your
Developer Funnel

26. KPI SECRET #2
Know your
Developer Funnel
(for public APIs)

27. Web sites have funnels
Acquisi9on
Ac9va9on
Reten9on
Revenue
Referral
How
do
users
find
you?
Does
the
user
have
a
good
first
experience?
Do
users
return?
How
do
you
make
money?
Do
users
tell
others?
Dave McClure’s “Startup Metrics for Pirates”

28. APIs have funnels too
Acquisi9on
Ac9va9on
Reten9on
Revenue
Referral
How
do
developers
find
you?
Does
the
developer
have
a
good
first
experience?
Do
developers
return?
How
do
you
and
developers
make
money?
Do
developers
tell
others?

29. Potential KPIs
Acquisi9on
Ac9va9on
Reten9on
Revenue
Referral
Dev
portal
site:
sources,
traffic
vol,
uniques
#
dev
registraCons,
API
keys
issued
#
acCve
devs,
API
call
volume
by
dev,
#
apps
$
per
dev,
$
per
app,
#
end
users
#
dev
referrals,
dev/app
growth,
NPS

30. KPI SECRET #3
Measure your
API ecosystem

31. App Users Apps Developers
Your APIs You
API
1
API
2
API
3

32. App Users Apps Developers
Your APIs You
API
1
API
2
API
3
Total app users
Revenue per user
User growth rate
User churn

33. App Users Apps Developers
Your APIs You
API
1
API
2
API
3
Total apps
Trending apps
Revenue per app
Channel: mobile/web

34. App Users Apps Developers
Your APIs You
API
1
API
2
API
3
Total developers
Active developers
Revenue per developer
Top developers
Stuck developers

35. App Users Apps Developers
Your APIs You
Total API calls
Top API calls
API
1
Top API packages
Internal usage
API
2
API
3

36. App Users Apps Developers
Your APIs You
API
1
API
2
API
3
Total revenue
Partnerships
Market share
Innovation

37. KPI SECRET #4
Most APIs
obey the
Power Law

38. The Power Law of APIs
Developers
API call volume

39. The Power Law of APIs
Most of your API
traffic, revenue,
success, etc….
Developers
API call volume

40. Think quality, not just quantity
Use your API KPIs, to
find who, how, where,
when.
Developers
API call volume

41. KPI SECRET #5
API KPIs
need buy-in

42. Don’t let API KPIs live in a vacuum
Who are your stakeholders?
Who’s your sponsor?
Get executive buy-in on your top 3-5 KPIs

43. KPI SECRET #6
Measuring
internal API usage often just
as important as external

44. Guardian: 70% internal calls
API usage from
internal apps
External
usage

45. Evernote: 99% internal calls
API usage from
internal apps
External
usage

46. Netflix: 99.9% internal calls
API usage from
devices
External
usage

47. Netflix: 99.9% internal calls
Why
Enable Netflix experience
Who
Netflix device and UI teams

48. KPI WARNING #1
Beware of
API vanity metrics

49. API Billionaire Club
13 billion API calls / day (May 2011)
5 billion API calls / day (April 2010)
5 billion API calls / day (October 2009)
1.1 billion API calls / day (April 2011)
1 billion API calls / day (May 2012)
1 billion API calls / day (Q1 2012)
1 billion API calls / day (January 2012)

50. 5 Billion
API calls/day

51. “What if, all things being equal,
instead of 30 billion API calls a month,
we served 5 billion?”
Daniel Jacobson, Netflix

52. KPI WARNING #2
Beware of
API KPI traps

53. 3 common API KPI traps
Not having a metrics process
(have a metrics plan & a feedback loop)
Using the wrong metrics
(don’t be afraid to change what you track)
Unrealistic expectations
(“Absolutely, we’ll have X developers on-board by X date”)

54. One important metric
you may have missed...

55. What’s your TTFHW?
Time To First “Hello World”
aka: how long from zero to 60?

56. What’s your TTFHW?
Developer experience (DX)
can make or break your API.

57. What’s your TTFHW?
Too subjective to measure?

58. What’s your TTFHW?
Don’t forget about
qualitative metrics

59. What’s your TTFHW?
User testing.
Surveys.
Forums.
Social media…

60. Real world
API KPIs

61. KPI: Conversions
“We are seeing a trend where a person
who uses another app with Evernote is
50% more likely to move on to our
premium (paid) service”

62. KPI: Engagement
“Apps that have an API integration with
Evernote are seeing much more usage:
Pocket readers normally read 15% of what
they save, but Evernote users read 80% of
what they save in Pocket.”

63. KPI: Revenue
API now accounts
for 20-25%
of the charity’s
annual revenue

65. KPI: Photo Edits

66. KPI: Apps
3,500 Apps
Aviary,
March,
2013

67. Goal: “more eyeballs on our content”
API calls (requests)
Response volume (ex: # stories)
Impressions (via image beacon)
Loyalty
More
KPIs
KPI: Impressions

68. KPI: Recipe delivery
“We sell Vodka.
But people buy drinks.”
Goal: “ensure top quality drinks
consistently delivered to people
regardless of where people
encounter Absolut”

69. KPI: Recipe delivery
APIs let you be where
your customers are.
Measure how.
“We sell Vodka.
But people buy drinks.”
Goal: “ensure top quality drinks
consistently delivered to people
regardless of where people
encounter where people encounter
Absolut”

70. Evolution of API metrics?
Calls
?
?
?

71. Who & why drives KPIs
Have a KPI lens & process
Prioritize & repeat

72. Thank You
john@apiscience.com
@johnmusser

73. Photo Credits
Race
car:
hCp://www.flickr.com/photos/lim_lik_wei/3270522646/
Stopwatch:
hCp://www.flickr.com/photos/purplemaMish/3020016417/
Stop
sign:
hCps://www.flickr.com/photos/9toperez/5791194358
Power
law:
hCp://en.wikipedia.org/wiki/Power_law