SlideShare a Scribd company logo
1 of 115
(Lean) Startup Metrics & Analytics

      Nicola Junior Vitto - @njvitto @blomming
  September 26, 2012 - Better Software - Firenze, Italy
Entrepreneurship



...Entrepreneurs recognize they are going to have
 to follow their own internal compass and embrace
      the uncertainty as part of the journey.

                              - Steve Blank -
 more on: http://steveblank.com/2009/06/29/agile-opportunism-entrepreneurial-dna/
Get out of the building (by Steve Blank)




    Your business assumptions can be wrong
     Go out and speak with your customers
Vanity Metrics
Vanity Metrics - Timelines




There will always be something up and to the right
Vanity Metrics - Google Analytics




                  Forget it!
Vanity Metrics - Google Analytics




 ...and this could become like a drug...killing you!
Vanity Metrics - TechCrunch & Co


           “mDialog raises 5 Million
          from Blackberry partners”

          “GroupMe is now sending
          one Million texts every day”

         “Google Plus hits 10 Million
             users in 2 weeks”
Lean Metrics
What’s a startup?


A startup is first of all an experiment, it is a human
institution designed to deliver a new product or
service under conditions of extreme uncertainty

At its heart, a startup is a catalyst that transforms
ideas into product.

       (From Lessons Learned blog By Eric Ries)
What’s a Lean Startup?



     Lean startup is a rigorous process
           for iterating from Plan A
            to a plan that works.

               (by Ash Maurya)
What Metrics Matter?


    Example: Facebook on early days

           150k users, little revenue

          Many superior competitors


   What would indicate their future success?
What Metrics Matter?


    Example: Facebook on early days

 75% of users visit one or more times per day

   Within one month of launching on a new
  campus, can acquire 90% of the students
What Metrics Matter?

        Good Metrics (Three A’s)

              Are Actionable

              Can be Audited

              Are Accessible
Actionable Metric



     An actionable metric is one that ties
      specific and repeatable actions
             to observed results

                (by Ash Maurya)
A/B Testing
A/B Testing - New Blomming home




                  VS




            Define your Goals
A/B Testing - Define your goals


              A/B Testing Goals

          One single important Goal:
% Users that sign-up for the service (as a seller)

                 Secondary Goal:
         % Buyers that buy something
         after landing on the homepage
Multivariate Testing
Multivariate Testing at Google


   “Yes, it’s true that a team at Google couldn’t
 decide between two blues, so they’re testing 41
    shades between each blue to see whic one
performs better. I has a recent debate over wheter
a border should be 3, 4 or 5 pixels wide, and was
               asked to prove my case.
 I can’t operate in an environment like that.”

       (by Doug Bowman on Design at Google)
Multivariate Testing at Google
Multivariate Testing at Google
Multivariate Testing at Google




      Google                     Bing
Multivariate Testing at Google




      “That blue was worth at least $80M”

       (by Paul Ray - Product Manager at Bing)
Funnels
David Cancel’s Funnel




                 The Problem:
         98% of people who visit your site,
              leave never to return
David Cancel’s Funnel



                        2%
  “The average conversion rate in the United States of
   America wheter you’re selling elephants or iPods”

   (by Avinash Kaushik - Analytics Evangelist, Google)
David Cancel’s Funnel
David Cancel’s Funnel




                    The Solution:
          To find, engage and convert more
       of your website visitors into customers
Online Marketing
   is shifting
Online Marketing Paradigm Shift




                                  by David Cancel
Online Marketing Paradigm Shift




                                  by David Cancel
Online Marketing Paradigm Shift




                                  by David Cancel
Online Marketing Paradigm Shift




  Internet Marketing 3.0 (different goal)
                                    by David Cancel
Lifecycle Marketing




                      by David Cancel
Optimize
Optimize for learning




                 Optimize for learning
     by creating a data-driven marketing framework
Optimize for learning




          Analytics & A/B Testing are also
             not exactly the answer
Optimize for learning




         Use quantitative & qualitative data
           to test *your* assumptions
Optimize for learning




      The point of testing is to prove or disprove
                assumptions quickly
Optimize for learning




 Pay attention to “overtesting” using Multivariate testing
           because it generally takes too long
The Local Maxima Problem
Frameworks
AARRR
AARRR

 Dave McClure’s Startup Metrics for Pirates is a good
     framework for thinking about macro metrics
AARRR - Model
AARRR - Conversions Example
AARRR - Different orders (example)


      1. Activation, Retention (happiness)

            2. Revenue (sustainability)

         3. Acquisition, Referral (virality)

      (this is the Ash Maurya or 37signals order:
                 “Charge from day one”)
Acquisition
Tracking Marketing Channels


                 Google Adwords
                    Banners
                  Social Media
                  Partnerships
                       PR
                    Blogging

        ...anything that links to your site!
Tracking Marketing Channels


        Use unique urls (tracking parameters)
       on every url you create/give-out/pay for

                     Use bit.ly

         Use HTTP referrers to group PR
Measuring Cost of Visits



               Cost per click (site visits)

        Cost of campaign / Traffic generated

    (Social media) Cost of effort / Traffic generated
Measuring Cost of Acquisition

         Cost of one visit / conversion rate =
              Cost of an Acquisition

      Acquisition could be paid for acquisition
                 of free sign-up


          ...unfortunately is not so simple
Marketing Channels

                Per channel track:

                    Scale
                    Cost
                  Conversion


           Optimise and scale channels
       with best Cost of Acquisition (CoA)
             and potential to Scale
Acquisition



    Cost of Acquisition < Customer Lifetime Profit

    Then scale acquisition at a similar or lower rate

       A/B test acquisition channels to optimise
Acquisition including Referral


 If it costs €3 to acquire a customer through advertising

                 But a customer brought
         in 2 other customers through referral

             The COA of that channel is €1
Activation
Activation



             Sure, they signed-up!!!
Activation



             Sure, they signed-up!!!


             BUT did the like it???
Activation at Facebook



 (i) which data points predict whether a user will stay?


(ii) if they stay, which data points predict how active they’ll
                    be after three months?
Activation at Facebook

 (i) which data points predict whether a user will stay?
        having more than one session as a new user
           and entering basic profile information.


(ii) if they stay, which data points predict how active they’ll
                    be after three months?
how often a user was reached out by others, frequency of
  third party application use, and how available a user was
                           on the site.
Activation at Facebook

        “At the time, I heard many people criticize
 Facebook’s early investors, claiming that Facebook had
   ‘no business model’ and only modest revenues
    relative to the valuation offered by its investors...”

“...Facebook was different, because it employed a different
     Engine of Growth. It paid nothing for customer
     acquisition, and its high engagement meant that
    it was accumulating massive amount of customer
                   attention every day.”

              (Eric Ries - The Lean Startup)
Activation Metrics




    Activation metrics often drive other metrics
            in Referral, Retention, Revenue
Retention
Returning Users



                Tracking unregistered users
 that return often (useful for content sites with advertising)

                Tracking registered users
         that return and take some sort of action.
Cohort Analysis
Twitter Cohorts (example)




                      by Fred Wilson’s blog
   (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
Twitter Cohorts (example)




                      by Fred Wilson’s blog
   (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
Referral
Referral Mechanism


             Social Media Sharing

         Invite a friend type mechanism

                 App Reviews

                Word of Mouth
Viral Coefficient




         The average number of customers
            that each customer refers
Viral Coefficient - Example


              Each company customer
         invites 5 other customers via email

            20% click on that email link

     50% of those users convert into customers
Viral Coefficient - Example




Viral Coefficient = Referrals x click through x conversion

                   5 x 0.2 x 0.5 = 0.5
Viral Coefficient


          A viral co-efficient greater than 1
            means that every customer
   gets more than one other customer on average
Viral Coefficient


          A viral co-efficient greater than 1
            means that every customer
   gets more than one other customer on average



       Therefore your product will grow virally
Viral Coefficient


           Measure automatically for all
           your built in referral mechanism



         Give customers unique invite URLs
                and track conversions
Viral Cycle Time



     Viral Cycle Time = the average time taken for
           a referral to turn into a customer


              With the viral coefficient
                and viral cycle time
         you can determine your viral growth
Net Promoter Score



  Net Promoter Score: Number of people who would
         recommend your product to a friend



              A single question survey
Net Promoter Score


 “How likely is it you would recommend my product to a
                     friend or colleague?”


 0    1     2   3     4    5    6    7    8     9   10




          NPS = % Promoters - % Detractors
Net Promoter Score


            Survey a small percentage
       of your customers by email weekly


           Record who you sent to so
       you don’t repeat the same people
                   every week
Net Promoter Score



          NPS is not a direct measurement
           that translates to key metrics



   Can act as a leading indicator of other metrics
Revenue
Making Money




  Key for any business is to find out how much profit
             they make for every customer
         and scaling the number of customers
Lifetime Customer Value


             How much money you make
          for every customer you acquire

          For some this a single transaction

 For subscription businesses it depends on retention
Cancellation Rate


    Cancellation rate = is the percentage number of
 customers who cancel in any given month compared to
                total (paying) customers


                     [Cancellation rate] =
   [product utility] + [service quality] + [acceptable price]
Lifetime Customer Value (LTV)



     LTV = monthly revenue x n° months in lifetime

          N° months = 1 / cancellation rate

       LTV = monthly revenue / cancellation rate
Lifetime Customer Value (LTV)



                     For example:

  If €20 a month and cancellation rate is 10% a month

                     LTV = €200
Lifetime Customer Value (LTV)


              Often not that simple.
         Short term cancellation rates are
           much higher than long term



        Use Cohort Analysis to determine
Lifetime Customer Value (LTV)


       r = short-term cancellation rate (e.g. 0,15)
       p = long-term cancellation rate (e.g. 0.03)
  s = number of months in the “short-term” age group
                        (e.g. 3)


         (1-r)^s x (s + 1/p) = expected months
Some Tools
Event based tracking
Event based tracking
Event based tracking
Customer lifecycle
  management
Customer lifecycle management
Customer lifecycle management
Tips
OODA Loop
Innovation Accounting


  “To improve entrepreneurial outcomes, and to
hold entrepreneurs accountable, we need to focus
 on the boring stuff: how to measure progress,
  how to setup milestones, how to prioritize
 work. This requires a new kind of accounting,
              specific to startups”

            (Eric Ries - The Lean Startup)
Innovation Accounting



            Concentrate on macro metrics

 Everything you do should attempt to change a metric

          Try to use as few tools as possible
One Page Dashboard
Customer Development: pivoting
(Lean) Product Development



Problem:
unknown




                              Solution:
                             unknown
Scalable (lean) startup
SCRUM




        Do you know SCRUM?
Kanban Board




     I know: this isn’t an example of an exciting and well done Sprint :)
(Lean) Product Management



             Well, trying to connect the dots:
    what we learnt so far is how the Product Owner
     can define and prioritize stories at every Sprint
 in a Startup that wants to be called, you know, Lean! :)
Learning Boards - Lean Stack




    We will talk about the Lean Stack of Ash Maurya
     or other possible “lean mapping” frameworks
                  in a future presentation
Focus focus focus
Credits

           Stuart Eccles - Lean Startup Metrics

          Joshua Porter - Metrics-Driven Design

          David Cancel - Data-Driven Marketing

               Lean Startup Machine -
 How to measure the metrics that determine real progress

(and the others specified directly in the slides when picking
              some parts of books or blogs)
The Lean Startup
Running Lean
Business Models
An alternative?




                  Lifestyle business
Thanks!

          Blomming.com
          Company.Blomming.com
          Magazine.Blomming.com
            Help.Blomming.com
           Twitter.com/Blomming
          Facebook.com/Blomming

           nicola@blomming.com
          On FB & Twitter as: njvitto

More Related Content

What's hot

ITFT Media- AIDA Theory of advertising ppt [read only]
ITFT Media-  AIDA Theory of advertising ppt [read only]ITFT Media-  AIDA Theory of advertising ppt [read only]
ITFT Media- AIDA Theory of advertising ppt [read only]Shweta Chugh
 
Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona Dave McClure
 
Advertising Effectiveness
Advertising EffectivenessAdvertising Effectiveness
Advertising EffectivenessHs Prince
 
Why Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategyWhy Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategyMickey Alon
 
From startup to scale-up
From startup to scale-upFrom startup to scale-up
From startup to scale-upFound.ation
 
Top 10 marcom interview questions with answers
Top 10 marcom interview questions with answersTop 10 marcom interview questions with answers
Top 10 marcom interview questions with answersandersonshaun106
 
Writing a PR Plan
Writing a PR PlanWriting a PR Plan
Writing a PR Planbneiswender
 
Integrated GTM Strategy Case Study
Integrated GTM Strategy Case StudyIntegrated GTM Strategy Case Study
Integrated GTM Strategy Case StudyBruce Pharr
 
Impact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online ShoppingImpact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online ShoppingMd Abusaleh
 
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...Swati Bhargava
 
Social media marketing strategy
Social media marketing strategySocial media marketing strategy
Social media marketing strategyDave Chaffey
 
Motivating And Compensating Channel Members
Motivating And Compensating Channel MembersMotivating And Compensating Channel Members
Motivating And Compensating Channel Membersdcsastudent
 
Social media advertising slideshare
Social media advertising   slideshareSocial media advertising   slideshare
Social media advertising slideshareMJ Social
 
PR plan template free
PR plan template freePR plan template free
PR plan template freeDebbie Leven
 
Chap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional ProgramChap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional ProgramPhoenix media & event
 

What's hot (20)

ITFT Media- AIDA Theory of advertising ppt [read only]
ITFT Media-  AIDA Theory of advertising ppt [read only]ITFT Media-  AIDA Theory of advertising ppt [read only]
ITFT Media- AIDA Theory of advertising ppt [read only]
 
Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona
 
Advertising Effectiveness
Advertising EffectivenessAdvertising Effectiveness
Advertising Effectiveness
 
Why Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategyWhy Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategy
 
From startup to scale-up
From startup to scale-upFrom startup to scale-up
From startup to scale-up
 
Advertising Testing Techniques
Advertising Testing TechniquesAdvertising Testing Techniques
Advertising Testing Techniques
 
Top 10 marcom interview questions with answers
Top 10 marcom interview questions with answersTop 10 marcom interview questions with answers
Top 10 marcom interview questions with answers
 
Writing a PR Plan
Writing a PR PlanWriting a PR Plan
Writing a PR Plan
 
Dagmar Approach
Dagmar ApproachDagmar Approach
Dagmar Approach
 
Outdoor advertising
Outdoor advertisingOutdoor advertising
Outdoor advertising
 
AIDA
AIDA AIDA
AIDA
 
Integrated GTM Strategy Case Study
Integrated GTM Strategy Case StudyIntegrated GTM Strategy Case Study
Integrated GTM Strategy Case Study
 
Impact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online ShoppingImpact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online Shopping
 
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
 
Print Ads
Print AdsPrint Ads
Print Ads
 
Social media marketing strategy
Social media marketing strategySocial media marketing strategy
Social media marketing strategy
 
Motivating And Compensating Channel Members
Motivating And Compensating Channel MembersMotivating And Compensating Channel Members
Motivating And Compensating Channel Members
 
Social media advertising slideshare
Social media advertising   slideshareSocial media advertising   slideshare
Social media advertising slideshare
 
PR plan template free
PR plan template freePR plan template free
PR plan template free
 
Chap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional ProgramChap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional Program
 

Similar to Lean Startup Metrics & Analytics

Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist Bryan Ferguson
 
Applying the science of measurement to the art of advertising - 20 january 2...
Applying the science of measurement to the art of advertising  - 20 january 2...Applying the science of measurement to the art of advertising  - 20 january 2...
Applying the science of measurement to the art of advertising - 20 january 2...Ron Jacobs
 
Hotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with MetricsHotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with MetricsALPHA Camp
 
Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)wedu, Inc
 
The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution WP Engine UK
 
The Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth HackingThe Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth HackingFrederik Hermann
 
The Freemium Model and A/B Testing
The Freemium Model and A/B TestingThe Freemium Model and A/B Testing
The Freemium Model and A/B TestingWebs
 
Funnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GAFunnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GASofia Quintero
 
Social Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptxSocial Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptxJeff Salzsauler
 
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Percussion Software
 
How to measure Social Media ROI
How to measure Social Media ROIHow to measure Social Media ROI
How to measure Social Media ROIScott K. Wilder
 
Conversion Whitepaper
Conversion WhitepaperConversion Whitepaper
Conversion WhitepaperWSI Ensenada
 
Lean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer RelationshipsLean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer RelationshipsAaron Eden
 
How To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget RandolphHow To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget RandolphFeverBee Limited
 
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...Bridget Randolph
 

Similar to Lean Startup Metrics & Analytics (20)

Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist
 
Multi-Channel Attribution - Where do Leads Come From?
Multi-Channel Attribution - Where do Leads Come From?Multi-Channel Attribution - Where do Leads Come From?
Multi-Channel Attribution - Where do Leads Come From?
 
Applying the science of measurement to the art of advertising - 20 january 2...
Applying the science of measurement to the art of advertising  - 20 january 2...Applying the science of measurement to the art of advertising  - 20 january 2...
Applying the science of measurement to the art of advertising - 20 january 2...
 
Hotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with MetricsHotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with Metrics
 
Startup analytics
Startup analyticsStartup analytics
Startup analytics
 
Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)
 
The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution
 
Growth Hacking
Growth HackingGrowth Hacking
Growth Hacking
 
Beginners guide to_growth_hacking
Beginners guide to_growth_hackingBeginners guide to_growth_hacking
Beginners guide to_growth_hacking
 
The Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth HackingThe Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth Hacking
 
The Freemium Model and A/B Testing
The Freemium Model and A/B TestingThe Freemium Model and A/B Testing
The Freemium Model and A/B Testing
 
Funnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GAFunnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GA
 
Social Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptxSocial Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptx
 
Digital measurement &amp; analytics
Digital measurement &amp; analyticsDigital measurement &amp; analytics
Digital measurement &amp; analytics
 
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
 
How to measure Social Media ROI
How to measure Social Media ROIHow to measure Social Media ROI
How to measure Social Media ROI
 
Conversion Whitepaper
Conversion WhitepaperConversion Whitepaper
Conversion Whitepaper
 
Lean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer RelationshipsLean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer Relationships
 
How To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget RandolphHow To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
 
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
 

More from Nicola Junior Vitto

The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...Nicola Junior Vitto
 
SCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in EnterprisesSCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in EnterprisesNicola Junior Vitto
 
Building Blomming: A startup founder's experience
Building Blomming: A startup founder's experienceBuilding Blomming: A startup founder's experience
Building Blomming: A startup founder's experienceNicola Junior Vitto
 
Social Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerceSocial Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerceNicola Junior Vitto
 
Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011Nicola Junior Vitto
 
The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.Nicola Junior Vitto
 

More from Nicola Junior Vitto (8)

The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...
 
SCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in EnterprisesSCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in Enterprises
 
Building Blomming: A startup founder's experience
Building Blomming: A startup founder's experienceBuilding Blomming: A startup founder's experience
Building Blomming: A startup founder's experience
 
Lean Startup - the Italian way
Lean Startup - the Italian wayLean Startup - the Italian way
Lean Startup - the Italian way
 
Social Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerceSocial Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerce
 
Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011
 
The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.
 
SQL vs Code
SQL vs CodeSQL vs Code
SQL vs Code
 

Recently uploaded

NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023Steve Rader
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...AustraliaChapterIIBA
 
Anyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agencyAnyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agencyHanna Klim
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003believeminhh
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessAPCO
 
Intellectual Property Licensing Examples
Intellectual Property Licensing ExamplesIntellectual Property Licensing Examples
Intellectual Property Licensing Examplesamberjiles31
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxWorkforce Group
 
Plano de marketing- inglês em formato ppt
Plano de marketing- inglês  em formato pptPlano de marketing- inglês  em formato ppt
Plano de marketing- inglês em formato pptElizangelaSoaresdaCo
 
To Create Your Own Wig Online To Create Your Own Wig Online
To Create Your Own Wig Online  To Create Your Own Wig OnlineTo Create Your Own Wig Online  To Create Your Own Wig Online
To Create Your Own Wig Online To Create Your Own Wig Onlinelng ths
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.ukaroemirsr
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfAnhNguyen97152
 
Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Lviv Startup Club
 
Slicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinSlicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinAnton Skornyakov
 
Upgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsUpgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsIntellect Design Arena Ltd
 
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfPDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfHajeJanKamps
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)tazeenaila12
 
A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.mcshagufta46
 

Recently uploaded (20)

WAM Corporate Presentation Mar 25 2024.pdf
WAM Corporate Presentation Mar 25 2024.pdfWAM Corporate Presentation Mar 25 2024.pdf
WAM Corporate Presentation Mar 25 2024.pdf
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
 
Anyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agencyAnyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agency
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
 
Intellectual Property Licensing Examples
Intellectual Property Licensing ExamplesIntellectual Property Licensing Examples
Intellectual Property Licensing Examples
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
 
Plano de marketing- inglês em formato ppt
Plano de marketing- inglês  em formato pptPlano de marketing- inglês  em formato ppt
Plano de marketing- inglês em formato ppt
 
To Create Your Own Wig Online To Create Your Own Wig Online
To Create Your Own Wig Online  To Create Your Own Wig OnlineTo Create Your Own Wig Online  To Create Your Own Wig Online
To Create Your Own Wig Online To Create Your Own Wig Online
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.uk
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
 
Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)
 
Slicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinSlicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup Berlin
 
Upgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsUpgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking Applications
 
Investment Opportunity for Thailand's Automotive & EV Industries
Investment Opportunity for Thailand's Automotive & EV IndustriesInvestment Opportunity for Thailand's Automotive & EV Industries
Investment Opportunity for Thailand's Automotive & EV Industries
 
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfPDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
 
A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.
 

Lean Startup Metrics & Analytics

  • 1. (Lean) Startup Metrics & Analytics Nicola Junior Vitto - @njvitto @blomming September 26, 2012 - Better Software - Firenze, Italy
  • 2. Entrepreneurship ...Entrepreneurs recognize they are going to have to follow their own internal compass and embrace the uncertainty as part of the journey. - Steve Blank - more on: http://steveblank.com/2009/06/29/agile-opportunism-entrepreneurial-dna/
  • 3. Get out of the building (by Steve Blank) Your business assumptions can be wrong Go out and speak with your customers
  • 5. Vanity Metrics - Timelines There will always be something up and to the right
  • 6. Vanity Metrics - Google Analytics Forget it!
  • 7. Vanity Metrics - Google Analytics ...and this could become like a drug...killing you!
  • 8. Vanity Metrics - TechCrunch & Co “mDialog raises 5 Million from Blackberry partners” “GroupMe is now sending one Million texts every day” “Google Plus hits 10 Million users in 2 weeks”
  • 10. What’s a startup? A startup is first of all an experiment, it is a human institution designed to deliver a new product or service under conditions of extreme uncertainty At its heart, a startup is a catalyst that transforms ideas into product. (From Lessons Learned blog By Eric Ries)
  • 11. What’s a Lean Startup? Lean startup is a rigorous process for iterating from Plan A to a plan that works. (by Ash Maurya)
  • 12. What Metrics Matter? Example: Facebook on early days 150k users, little revenue Many superior competitors What would indicate their future success?
  • 13. What Metrics Matter? Example: Facebook on early days 75% of users visit one or more times per day Within one month of launching on a new campus, can acquire 90% of the students
  • 14. What Metrics Matter? Good Metrics (Three A’s) Are Actionable Can be Audited Are Accessible
  • 15. Actionable Metric An actionable metric is one that ties specific and repeatable actions to observed results (by Ash Maurya)
  • 17. A/B Testing - New Blomming home VS Define your Goals
  • 18. A/B Testing - Define your goals A/B Testing Goals One single important Goal: % Users that sign-up for the service (as a seller) Secondary Goal: % Buyers that buy something after landing on the homepage
  • 20. Multivariate Testing at Google “Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see whic one performs better. I has a recent debate over wheter a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that.” (by Doug Bowman on Design at Google)
  • 23. Multivariate Testing at Google Google Bing
  • 24. Multivariate Testing at Google “That blue was worth at least $80M” (by Paul Ray - Product Manager at Bing)
  • 26. David Cancel’s Funnel The Problem: 98% of people who visit your site, leave never to return
  • 27. David Cancel’s Funnel 2% “The average conversion rate in the United States of America wheter you’re selling elephants or iPods” (by Avinash Kaushik - Analytics Evangelist, Google)
  • 29. David Cancel’s Funnel The Solution: To find, engage and convert more of your website visitors into customers
  • 30. Online Marketing is shifting
  • 31. Online Marketing Paradigm Shift by David Cancel
  • 32. Online Marketing Paradigm Shift by David Cancel
  • 33. Online Marketing Paradigm Shift by David Cancel
  • 34. Online Marketing Paradigm Shift Internet Marketing 3.0 (different goal) by David Cancel
  • 35. Lifecycle Marketing by David Cancel
  • 37. Optimize for learning Optimize for learning by creating a data-driven marketing framework
  • 38. Optimize for learning Analytics & A/B Testing are also not exactly the answer
  • 39. Optimize for learning Use quantitative & qualitative data to test *your* assumptions
  • 40. Optimize for learning The point of testing is to prove or disprove assumptions quickly
  • 41. Optimize for learning Pay attention to “overtesting” using Multivariate testing because it generally takes too long
  • 42. The Local Maxima Problem
  • 44. AARRR
  • 45. AARRR Dave McClure’s Startup Metrics for Pirates is a good framework for thinking about macro metrics
  • 48. AARRR - Different orders (example) 1. Activation, Retention (happiness) 2. Revenue (sustainability) 3. Acquisition, Referral (virality) (this is the Ash Maurya or 37signals order: “Charge from day one”)
  • 50. Tracking Marketing Channels Google Adwords Banners Social Media Partnerships PR Blogging ...anything that links to your site!
  • 51. Tracking Marketing Channels Use unique urls (tracking parameters) on every url you create/give-out/pay for Use bit.ly Use HTTP referrers to group PR
  • 52. Measuring Cost of Visits Cost per click (site visits) Cost of campaign / Traffic generated (Social media) Cost of effort / Traffic generated
  • 53. Measuring Cost of Acquisition Cost of one visit / conversion rate = Cost of an Acquisition Acquisition could be paid for acquisition of free sign-up ...unfortunately is not so simple
  • 54. Marketing Channels Per channel track: Scale Cost Conversion Optimise and scale channels with best Cost of Acquisition (CoA) and potential to Scale
  • 55. Acquisition Cost of Acquisition < Customer Lifetime Profit Then scale acquisition at a similar or lower rate A/B test acquisition channels to optimise
  • 56. Acquisition including Referral If it costs €3 to acquire a customer through advertising But a customer brought in 2 other customers through referral The COA of that channel is €1
  • 58. Activation Sure, they signed-up!!!
  • 59. Activation Sure, they signed-up!!! BUT did the like it???
  • 60. Activation at Facebook (i) which data points predict whether a user will stay? (ii) if they stay, which data points predict how active they’ll be after three months?
  • 61. Activation at Facebook (i) which data points predict whether a user will stay? having more than one session as a new user and entering basic profile information. (ii) if they stay, which data points predict how active they’ll be after three months? how often a user was reached out by others, frequency of third party application use, and how available a user was on the site.
  • 62. Activation at Facebook “At the time, I heard many people criticize Facebook’s early investors, claiming that Facebook had ‘no business model’ and only modest revenues relative to the valuation offered by its investors...” “...Facebook was different, because it employed a different Engine of Growth. It paid nothing for customer acquisition, and its high engagement meant that it was accumulating massive amount of customer attention every day.” (Eric Ries - The Lean Startup)
  • 63. Activation Metrics Activation metrics often drive other metrics in Referral, Retention, Revenue
  • 65. Returning Users Tracking unregistered users that return often (useful for content sites with advertising) Tracking registered users that return and take some sort of action.
  • 67. Twitter Cohorts (example) by Fred Wilson’s blog (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
  • 68. Twitter Cohorts (example) by Fred Wilson’s blog (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
  • 70. Referral Mechanism Social Media Sharing Invite a friend type mechanism App Reviews Word of Mouth
  • 71. Viral Coefficient The average number of customers that each customer refers
  • 72. Viral Coefficient - Example Each company customer invites 5 other customers via email 20% click on that email link 50% of those users convert into customers
  • 73. Viral Coefficient - Example Viral Coefficient = Referrals x click through x conversion 5 x 0.2 x 0.5 = 0.5
  • 74. Viral Coefficient A viral co-efficient greater than 1 means that every customer gets more than one other customer on average
  • 75. Viral Coefficient A viral co-efficient greater than 1 means that every customer gets more than one other customer on average Therefore your product will grow virally
  • 76. Viral Coefficient Measure automatically for all your built in referral mechanism Give customers unique invite URLs and track conversions
  • 77. Viral Cycle Time Viral Cycle Time = the average time taken for a referral to turn into a customer With the viral coefficient and viral cycle time you can determine your viral growth
  • 78. Net Promoter Score Net Promoter Score: Number of people who would recommend your product to a friend A single question survey
  • 79. Net Promoter Score “How likely is it you would recommend my product to a friend or colleague?” 0 1 2 3 4 5 6 7 8 9 10 NPS = % Promoters - % Detractors
  • 80. Net Promoter Score Survey a small percentage of your customers by email weekly Record who you sent to so you don’t repeat the same people every week
  • 81. Net Promoter Score NPS is not a direct measurement that translates to key metrics Can act as a leading indicator of other metrics
  • 83. Making Money Key for any business is to find out how much profit they make for every customer and scaling the number of customers
  • 84. Lifetime Customer Value How much money you make for every customer you acquire For some this a single transaction For subscription businesses it depends on retention
  • 85. Cancellation Rate Cancellation rate = is the percentage number of customers who cancel in any given month compared to total (paying) customers [Cancellation rate] = [product utility] + [service quality] + [acceptable price]
  • 86. Lifetime Customer Value (LTV) LTV = monthly revenue x n° months in lifetime N° months = 1 / cancellation rate LTV = monthly revenue / cancellation rate
  • 87. Lifetime Customer Value (LTV) For example: If €20 a month and cancellation rate is 10% a month LTV = €200
  • 88. Lifetime Customer Value (LTV) Often not that simple. Short term cancellation rates are much higher than long term Use Cohort Analysis to determine
  • 89. Lifetime Customer Value (LTV) r = short-term cancellation rate (e.g. 0,15) p = long-term cancellation rate (e.g. 0.03) s = number of months in the “short-term” age group (e.g. 3) (1-r)^s x (s + 1/p) = expected months
  • 94. Customer lifecycle management
  • 97. Tips
  • 99. Innovation Accounting “To improve entrepreneurial outcomes, and to hold entrepreneurs accountable, we need to focus on the boring stuff: how to measure progress, how to setup milestones, how to prioritize work. This requires a new kind of accounting, specific to startups” (Eric Ries - The Lean Startup)
  • 100. Innovation Accounting Concentrate on macro metrics Everything you do should attempt to change a metric Try to use as few tools as possible
  • 105. SCRUM Do you know SCRUM?
  • 106. Kanban Board I know: this isn’t an example of an exciting and well done Sprint :)
  • 107. (Lean) Product Management Well, trying to connect the dots: what we learnt so far is how the Product Owner can define and prioritize stories at every Sprint in a Startup that wants to be called, you know, Lean! :)
  • 108. Learning Boards - Lean Stack We will talk about the Lean Stack of Ash Maurya or other possible “lean mapping” frameworks in a future presentation
  • 110. Credits Stuart Eccles - Lean Startup Metrics Joshua Porter - Metrics-Driven Design David Cancel - Data-Driven Marketing Lean Startup Machine - How to measure the metrics that determine real progress (and the others specified directly in the slides when picking some parts of books or blogs)
  • 114. An alternative? Lifestyle business
  • 115. Thanks! Blomming.com Company.Blomming.com Magazine.Blomming.com Help.Blomming.com Twitter.com/Blomming Facebook.com/Blomming nicola@blomming.com On FB & Twitter as: njvitto

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. \n
  58. \n
  59. \n
  60. \n
  61. \n
  62. \n
  63. \n
  64. \n
  65. \n
  66. \n
  67. \n
  68. \n
  69. \n
  70. \n
  71. \n
  72. \n
  73. \n
  74. \n
  75. \n
  76. \n
  77. \n
  78. \n
  79. \n
  80. \n
  81. \n
  82. \n
  83. \n
  84. \n
  85. \n
  86. \n
  87. \n
  88. \n
  89. \n
  90. \n
  91. \n
  92. \n
  93. \n
  94. \n
  95. \n
  96. \n
  97. \n
  98. \n
  99. \n
  100. \n
  101. \n
  102. \n
  103. \n
  104. \n
  105. \n
  106. \n
  107. \n
  108. \n
  109. \n
  110. \n
  111. \n
  112. \n
  113. \n
  114. \n
  115. \n