Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value

1. Measuring Success and Business Value: Metrics and Analysis Ted Hopton, United Business Media Donna Cuomo, The MITRE Corporation © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved.

2. #e2conf @Ted_Hopton @@Donnalc300 Who We Are Ted Hopton Wiki Community Manager United Business Media www.adventuresinsocialmedia.org @Ted_Hopton Donna Cuomo, PhD Chief Information Architect The MITRE Corporation dcuomo@mitre.org @Donnalc300 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 2

3. #e2conf @Ted_Hopton @@Donnalc300 United Business Media  Who is UBM? – ~ 6000 employees world-wide across hundreds of offices plus telecommuters – B2B media:  “We serve specialist business communities with tradeshows and other live ‘in person’ events; data, marketing and information products; print products; and targeting, distribution and monitoring services.” – Organizationally stove-piped  More than a dozen divisions serving distinct markets  Each has a CEO and operates largely independently – Commonalities across divisions include  Exhibitions/conferences (BTW, we produce this event, E2.0!)  Print publications  Digital/Online Services – Launched internal Jive SBS community Sept. 2008 © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 3

5. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Minimal Engagement Level: Logins and Contributors Contributors = Created content of some kind © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 5

6. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Consuming Content: Page Views Per Member Views: Comparable over time as community grows © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 6

7. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members, Consumers and Contributors © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 7

10. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Page Views Per Employee Comparable view across divisions of different size © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 10

13. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Where In the Community Is Activity Taking Place? © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 13

14. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Who Is Creating and Viewing Content? Learn what the active members are doing so you can share success stories and ideas © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 14

15. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members by Level of Activity Minimal = <6 activities Light = 6 - 20 activities Moderate = 21 - 50 activities Heavy = 51- 200 activities Super = >200 activities © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 15

16. #e2conf @Ted_Hopton @@Donnalc300 Quantitative Metrics Active Members by Level of Activity Minimal = <6 activities Light = 6 - 20 activities Moderate = 21 - 50 activities Heavy = 51- 200 activities Super = >200 activities © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 16

18. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  Annual User Survey – How often they used it in specific ways  Several times/week  Weekly  Monthly  Occasionally  Never © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 18

19. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  List of positive outcomes – Strongly Agree – Somewhat Agree – Somewhat Disagree – Strongly Disagree © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 19

20. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  Why Don’t You Use It More? – Strongly Agree – Somewhat Agree – Somewhat Disagree – Strongly Disagree  Blunt, negative statements – Ask for it! – Let them tell you how they feel  OK, this is going to hurt a little – It’s a benchmark – Listen, learn, then improve © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 20

21. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Survey  The Bottom Line Question – How likely are you to recommend using it to a colleague? – Scale of 0-10, with zero least likely and 10 most likely  Net Promoter Score© (NPS) – Scores of 9-10 = Promoters (7-8 = Passive Positives) – Scores of 0-6 = Detractors – NPS = % Promoters - % Detractors  Positive NPS means more people promoting than detracting  Negative NPS means the opposite   A Truly Comparable Metric – Even very different communities can compare NPS © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 21

22. #e2conf @Ted_Hopton @@Donnalc300 Qualitative Metrics: Wins  Success Stories: “Wins” – Invite people to write up and submit wins  Small, medium and large  All kinds: •cost-saving, revenue •“Soft” wins: collaboration, communication, efficiency, innovation… – Celebrate Wins – Share them as examples for others © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 22

23. #e2conf @Ted_Hopton @@Donnalc300 Lessons Learned  Metrics will (and should) evolve – Technology evolves new measurement capabilities – Your skill with the technology will evolve better at pulling out data you need – Your understanding of your community & objectives will evolve with experience – Your community will evolve it’s alive, it will change, it will surprise you!  Numbers + Surveys + Stories – Collect all three – Look at the whole picture © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 23

24. #e2conf @Ted_Hopton @@Donnalc300 Lessons Learned  There are no generally accepted benchmarks for Enterprise 2.0 communities – No one knows what results you *should* see for your community – Your Culture + Your Objectives = A Unique Community  Benchmark Against Yourself – Aim for progress and improvement over time – Compare different parts of your community with each other © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 24

25. #e2conf @Ted_Hopton @@Donnalc300 The MITRE Corporation  MITRE is a private, independent, not-for-profit organization, chartered to work in the public interest  Founded in 1958 to provide engineering and technical services to Bedford, Mass. the U.S. Air Force  Currently manages 4 Federally Funded Research and Development Centers McLean, Va. – Department of Defense 7,000 employees worldwide – Federal Aviation Administration – Internal Revenue Service/Department of Veterans Affairs – Department of Homeland Security  Supports a broad and diverse set of sponsors within the U.S. government, as well as internationally © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 25 25

26. #e2conf @Ted_Hopton @@Donnalc300 Use Case 1: Improve MITRE’s Research Program Selection Process  Deploy an innovation management tool as the host environment for the research competition  Enable codification of the research competition process  Collect all ideas in one place and record all participation (eliminate early “weeding out” of ideas)  Better support “teaming” of proposers from across the corporation with similar ideas (improve collaboration)  Encourage much broader participation in both proposing ideas and commenting on ideas  Improve user experience and satisfaction of participation (visibility, feedback, standard process, idea targeting)  Use an externally-hosted application (cloud service) © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 26

27. #e2conf @Ted_Hopton @@Donnalc300 Idea Market FY10 Research Strategic Plan, powered by Sharepoint Wiki Idea Market, powered by Spigit as a cloud service Face-to-face “elevator proposals” © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 27 27

28. #e2conf @Ted_Hopton @@Donnalc300 First Year Numbers (as of 20 May 2009)  2842 of 7278 (39%) MITRE employees clicked into Idea Market – 1445 Read-only Visitors – 597 Idea Owners (non-stakeholders, submitted ideas, commented, voted) – 575 Voters (only voted on ideas) – 192 Commenting Users (non-stakeholders, non-idea owners who provided written content) – 33 Members are Active Stakeholders (MIP Leadership who submitted an idea, review, comment, reply or vote)  840 Ideas submitted  750 Comments (threads) initiated  969 Replies to Comments  463 Reviews submitted  643 MIP Tech Support inquiries handle  5564 Votes cast Achieved transparency, breadth of participation © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 28

29. #e2conf @Ted_Hopton @@Donnalc300 MIP User Satisfaction Summary: 2008 vs. 2009  Respondents reported considerable engagement with the process  Clear improvement in perceptions that the process is understandable and predictable  Clear improvement in perceptions of fairness/consistency in proposal evaluation  Still only a (sizable) minority of respondents who feel that they understand or have received adequate rationale for funding decisions  More participants (41%) agreed that the process improved their competition experience than disagreed (27%) Used survey technique to assess user experience © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 29

30. #e2conf @Ted_Hopton @@Donnalc300 Use Case 2: Social Bookmarking (with Laurie Damianos)  Hypothesized social bookmarking would improve resource sharing, leveraging others research across teams and corporation – Some use of external tools like delicious already occurring  Could feed expertise finding  Could replace current “knowledge zones” – subject-based websites maintained by corporate stewards (consider corporate goals, not just end user goals)  Increase participation in knowledge sharing  Subscription to topic areas  More granular information management technique via tagging  Ease of ‘re-finding’ information or highlighting resources (recommended, lesson learned, etc) © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 30

31. #e2conf @Ted_Hopton @@Donnalc300 Social Bookmarking RSS Tips Popular topics Corporately stewarded Bookmarked collections resources urc © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 31

32. #e2conf @Ted_Hopton @@Donnalc300 onomi Pilot Usage (from 2007/2008) Average information provider has 56 bookmarks tagged with 5.4 terms. 154K tags 12000 page visits/mos 18K unique 27K bookmarks tags 17% internal bookmarks Supports need for internal social bookmarking service onomi bookmarks by tag my bookmarks by tag other bookmarks 83% external bookmarks add bookmark edit bookmark my bookmarks by tag We have no other way to share external add bookmark by pop-up resources browse users my tag collection delete bookmark bookmark page (URL) most popular activity: viewing search other bookmarks by tag help other people’s bookmarks poponomi onomi tag collection add bookmark via bookmarklet tag filters 14% of onomi’s visitors are delicious top 15 quick lookup (user, tag) edit my tag contributors other tag collections © Copyright 2010 2.0 Adoption 0% Council, MITRE Corporation, United Business 30% Ltd. All rights reserved. 5% 10% 15% 20% 25% Media, 32

34. #e2conf @Ted_Hopton @@Donnalc300 MITRE-Babson SNA Study (Bill Donaldson, Donna Cuomo, Sal Parise, Bala Iyer) Size: tenure Study Findings • Its not just frequency of use that is important with these tools, but rather, who you are connecting to (uniqueness) • Brokerage, in both social networks and technology- mediated networks, has a positive impact on personal innovativeness • Both of the technology networks (ListServ and social bookmarking/ tagging) provide unique, significant value Center # Color • The social network & the 1 Lavender technology networks 2 Blue 5 Red complement each other 19 LightMagenta 20 Sepia 48 Melon © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 34

35. #e2conf @Ted_Hopton @@Donnalc300 Measuring Business Value  Understand and explicitly document your goals for deploying a particular social tool. What benefits do you expect to achieve?  Think about what evaluation methodologies and objective metrics would help you understand if these desired behaviors are occurring  End users are not your only user group – Research analysts, commenters/reviewers, expertise finding, recommending, ….  Increasing the number of unique connections in a person’s social network has value  Cutter study notes it isn’t always possible to measure business value of a social tool from a prototype – Need critical mass, long tail effect of many one-to-one benefits, takes 2-3 years for behavior changes to occur, etc  Use realistic benchmarks of participation to assess the adoption success of your tool – Social tool use and patterns not always comparable to business tool use © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 35

36. #e2conf @Ted_Hopton @@Donnalc300 Engage. Evangelize. Empower. www.20adoptioncouncil.com © Copyright 2010 2.0 Adoption Council, MITRE Corporation, United Business Media, Ltd. All rights reserved. 36

Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value

The 2.0 Adoption Council

Enterprise 2.0 Black Belt Workshop: Measuring Success and Business Value