SlideShare a Scribd company logo
1 of 115
(Lean) Startup Metrics & Analytics

      Nicola Junior Vitto - @njvitto @blomming
  September 26, 2012 - Better Software - Firenze, Italy
Entrepreneurship



...Entrepreneurs recognize they are going to have
 to follow their own internal compass and embrace
      the uncertainty as part of the journey.

                              - Steve Blank -
 more on: http://steveblank.com/2009/06/29/agile-opportunism-entrepreneurial-dna/
Get out of the building (by Steve Blank)




    Your business assumptions can be wrong
     Go out and speak with your customers
Vanity Metrics
Vanity Metrics - Timelines




There will always be something up and to the right
Vanity Metrics - Google Analytics




                  Forget it!
Vanity Metrics - Google Analytics




 ...and this could become like a drug...killing you!
Vanity Metrics - TechCrunch & Co


           “mDialog raises 5 Million
          from Blackberry partners”

          “GroupMe is now sending
          one Million texts every day”

         “Google Plus hits 10 Million
             users in 2 weeks”
Lean Metrics
What’s a startup?


A startup is first of all an experiment, it is a human
institution designed to deliver a new product or
service under conditions of extreme uncertainty

At its heart, a startup is a catalyst that transforms
ideas into product.

       (From Lessons Learned blog By Eric Ries)
What’s a Lean Startup?



     Lean startup is a rigorous process
           for iterating from Plan A
            to a plan that works.

               (by Ash Maurya)
What Metrics Matter?


    Example: Facebook on early days

           150k users, little revenue

          Many superior competitors


   What would indicate their future success?
What Metrics Matter?


    Example: Facebook on early days

 75% of users visit one or more times per day

   Within one month of launching on a new
  campus, can acquire 90% of the students
What Metrics Matter?

        Good Metrics (Three A’s)

              Are Actionable

              Can be Audited

              Are Accessible
Actionable Metric



     An actionable metric is one that ties
      specific and repeatable actions
             to observed results

                (by Ash Maurya)
A/B Testing
A/B Testing - New Blomming home




                  VS




            Define your Goals
A/B Testing - Define your goals


              A/B Testing Goals

          One single important Goal:
% Users that sign-up for the service (as a seller)

                 Secondary Goal:
         % Buyers that buy something
         after landing on the homepage
Multivariate Testing
Multivariate Testing at Google


   “Yes, it’s true that a team at Google couldn’t
 decide between two blues, so they’re testing 41
    shades between each blue to see whic one
performs better. I has a recent debate over wheter
a border should be 3, 4 or 5 pixels wide, and was
               asked to prove my case.
 I can’t operate in an environment like that.”

       (by Doug Bowman on Design at Google)
Multivariate Testing at Google
Multivariate Testing at Google
Multivariate Testing at Google




      Google                     Bing
Multivariate Testing at Google




      “That blue was worth at least $80M”

       (by Paul Ray - Product Manager at Bing)
Funnels
David Cancel’s Funnel




                 The Problem:
         98% of people who visit your site,
              leave never to return
David Cancel’s Funnel



                        2%
  “The average conversion rate in the United States of
   America wheter you’re selling elephants or iPods”

   (by Avinash Kaushik - Analytics Evangelist, Google)
David Cancel’s Funnel
David Cancel’s Funnel




                    The Solution:
          To find, engage and convert more
       of your website visitors into customers
Online Marketing
   is shifting
Online Marketing Paradigm Shift




                                  by David Cancel
Online Marketing Paradigm Shift




                                  by David Cancel
Online Marketing Paradigm Shift




                                  by David Cancel
Online Marketing Paradigm Shift




  Internet Marketing 3.0 (different goal)
                                    by David Cancel
Lifecycle Marketing




                      by David Cancel
Optimize
Optimize for learning




                 Optimize for learning
     by creating a data-driven marketing framework
Optimize for learning




          Analytics & A/B Testing are also
             not exactly the answer
Optimize for learning




         Use quantitative & qualitative data
           to test *your* assumptions
Optimize for learning




      The point of testing is to prove or disprove
                assumptions quickly
Optimize for learning




 Pay attention to “overtesting” using Multivariate testing
           because it generally takes too long
The Local Maxima Problem
Frameworks
AARRR
AARRR

 Dave McClure’s Startup Metrics for Pirates is a good
     framework for thinking about macro metrics
AARRR - Model
AARRR - Conversions Example
AARRR - Different orders (example)


      1. Activation, Retention (happiness)

            2. Revenue (sustainability)

         3. Acquisition, Referral (virality)

      (this is the Ash Maurya or 37signals order:
                 “Charge from day one”)
Acquisition
Tracking Marketing Channels


                 Google Adwords
                    Banners
                  Social Media
                  Partnerships
                       PR
                    Blogging

        ...anything that links to your site!
Tracking Marketing Channels


        Use unique urls (tracking parameters)
       on every url you create/give-out/pay for

                     Use bit.ly

         Use HTTP referrers to group PR
Measuring Cost of Visits



               Cost per click (site visits)

        Cost of campaign / Traffic generated

    (Social media) Cost of effort / Traffic generated
Measuring Cost of Acquisition

         Cost of one visit / conversion rate =
              Cost of an Acquisition

      Acquisition could be paid for acquisition
                 of free sign-up


          ...unfortunately is not so simple
Marketing Channels

                Per channel track:

                    Scale
                    Cost
                  Conversion


           Optimise and scale channels
       with best Cost of Acquisition (CoA)
             and potential to Scale
Acquisition



    Cost of Acquisition < Customer Lifetime Profit

    Then scale acquisition at a similar or lower rate

       A/B test acquisition channels to optimise
Acquisition including Referral


 If it costs €3 to acquire a customer through advertising

                 But a customer brought
         in 2 other customers through referral

             The COA of that channel is €1
Activation
Activation



             Sure, they signed-up!!!
Activation



             Sure, they signed-up!!!


             BUT did the like it???
Activation at Facebook



 (i) which data points predict whether a user will stay?


(ii) if they stay, which data points predict how active they’ll
                    be after three months?
Activation at Facebook

 (i) which data points predict whether a user will stay?
        having more than one session as a new user
           and entering basic profile information.


(ii) if they stay, which data points predict how active they’ll
                    be after three months?
how often a user was reached out by others, frequency of
  third party application use, and how available a user was
                           on the site.
Activation at Facebook

        “At the time, I heard many people criticize
 Facebook’s early investors, claiming that Facebook had
   ‘no business model’ and only modest revenues
    relative to the valuation offered by its investors...”

“...Facebook was different, because it employed a different
     Engine of Growth. It paid nothing for customer
     acquisition, and its high engagement meant that
    it was accumulating massive amount of customer
                   attention every day.”

              (Eric Ries - The Lean Startup)
Activation Metrics




    Activation metrics often drive other metrics
            in Referral, Retention, Revenue
Retention
Returning Users



                Tracking unregistered users
 that return often (useful for content sites with advertising)

                Tracking registered users
         that return and take some sort of action.
Cohort Analysis
Twitter Cohorts (example)




                      by Fred Wilson’s blog
   (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
Twitter Cohorts (example)




                      by Fred Wilson’s blog
   (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
Referral
Referral Mechanism


             Social Media Sharing

         Invite a friend type mechanism

                 App Reviews

                Word of Mouth
Viral Coefficient




         The average number of customers
            that each customer refers
Viral Coefficient - Example


              Each company customer
         invites 5 other customers via email

            20% click on that email link

     50% of those users convert into customers
Viral Coefficient - Example




Viral Coefficient = Referrals x click through x conversion

                   5 x 0.2 x 0.5 = 0.5
Viral Coefficient


          A viral co-efficient greater than 1
            means that every customer
   gets more than one other customer on average
Viral Coefficient


          A viral co-efficient greater than 1
            means that every customer
   gets more than one other customer on average



       Therefore your product will grow virally
Viral Coefficient


           Measure automatically for all
           your built in referral mechanism



         Give customers unique invite URLs
                and track conversions
Viral Cycle Time



     Viral Cycle Time = the average time taken for
           a referral to turn into a customer


              With the viral coefficient
                and viral cycle time
         you can determine your viral growth
Net Promoter Score



  Net Promoter Score: Number of people who would
         recommend your product to a friend



              A single question survey
Net Promoter Score


 “How likely is it you would recommend my product to a
                     friend or colleague?”


 0    1     2   3     4    5    6    7    8     9   10




          NPS = % Promoters - % Detractors
Net Promoter Score


            Survey a small percentage
       of your customers by email weekly


           Record who you sent to so
       you don’t repeat the same people
                   every week
Net Promoter Score



          NPS is not a direct measurement
           that translates to key metrics



   Can act as a leading indicator of other metrics
Revenue
Making Money




  Key for any business is to find out how much profit
             they make for every customer
         and scaling the number of customers
Lifetime Customer Value


             How much money you make
          for every customer you acquire

          For some this a single transaction

 For subscription businesses it depends on retention
Cancellation Rate


    Cancellation rate = is the percentage number of
 customers who cancel in any given month compared to
                total (paying) customers


                     [Cancellation rate] =
   [product utility] + [service quality] + [acceptable price]
Lifetime Customer Value (LTV)



     LTV = monthly revenue x n° months in lifetime

          N° months = 1 / cancellation rate

       LTV = monthly revenue / cancellation rate
Lifetime Customer Value (LTV)



                     For example:

  If €20 a month and cancellation rate is 10% a month

                     LTV = €200
Lifetime Customer Value (LTV)


              Often not that simple.
         Short term cancellation rates are
           much higher than long term



        Use Cohort Analysis to determine
Lifetime Customer Value (LTV)


       r = short-term cancellation rate (e.g. 0,15)
       p = long-term cancellation rate (e.g. 0.03)
  s = number of months in the “short-term” age group
                        (e.g. 3)


         (1-r)^s x (s + 1/p) = expected months
Some Tools
Event based tracking
Event based tracking
Event based tracking
Customer lifecycle
  management
Customer lifecycle management
Customer lifecycle management
Tips
OODA Loop
Innovation Accounting


  “To improve entrepreneurial outcomes, and to
hold entrepreneurs accountable, we need to focus
 on the boring stuff: how to measure progress,
  how to setup milestones, how to prioritize
 work. This requires a new kind of accounting,
              specific to startups”

            (Eric Ries - The Lean Startup)
Innovation Accounting



            Concentrate on macro metrics

 Everything you do should attempt to change a metric

          Try to use as few tools as possible
One Page Dashboard
Customer Development: pivoting
(Lean) Product Development



Problem:
unknown




                              Solution:
                             unknown
Scalable (lean) startup
SCRUM




        Do you know SCRUM?
Kanban Board




     I know: this isn’t an example of an exciting and well done Sprint :)
(Lean) Product Management



             Well, trying to connect the dots:
    what we learnt so far is how the Product Owner
     can define and prioritize stories at every Sprint
 in a Startup that wants to be called, you know, Lean! :)
Learning Boards - Lean Stack




    We will talk about the Lean Stack of Ash Maurya
     or other possible “lean mapping” frameworks
                  in a future presentation
Focus focus focus
Credits

           Stuart Eccles - Lean Startup Metrics

          Joshua Porter - Metrics-Driven Design

          David Cancel - Data-Driven Marketing

               Lean Startup Machine -
 How to measure the metrics that determine real progress

(and the others specified directly in the slides when picking
              some parts of books or blogs)
The Lean Startup
Running Lean
Business Models
An alternative?




                  Lifestyle business
Thanks!

          Blomming.com
          Company.Blomming.com
          Magazine.Blomming.com
            Help.Blomming.com
           Twitter.com/Blomming
          Facebook.com/Blomming

           nicola@blomming.com
          On FB & Twitter as: njvitto

More Related Content

What's hot

ITFT Media- AIDA Theory of advertising ppt [read only]
ITFT Media-  AIDA Theory of advertising ppt [read only]ITFT Media-  AIDA Theory of advertising ppt [read only]
ITFT Media- AIDA Theory of advertising ppt [read only]Shweta Chugh
 
Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona Dave McClure
 
Advertising Effectiveness
Advertising EffectivenessAdvertising Effectiveness
Advertising EffectivenessHs Prince
 
Why Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategyWhy Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategyMickey Alon
 
From startup to scale-up
From startup to scale-upFrom startup to scale-up
From startup to scale-upFound.ation
 
Top 10 marcom interview questions with answers
Top 10 marcom interview questions with answersTop 10 marcom interview questions with answers
Top 10 marcom interview questions with answersandersonshaun106
 
Writing a PR Plan
Writing a PR PlanWriting a PR Plan
Writing a PR Planbneiswender
 
Integrated GTM Strategy Case Study
Integrated GTM Strategy Case StudyIntegrated GTM Strategy Case Study
Integrated GTM Strategy Case StudyBruce Pharr
 
Impact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online ShoppingImpact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online ShoppingMd Abusaleh
 
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...Swati Bhargava
 
Social media marketing strategy
Social media marketing strategySocial media marketing strategy
Social media marketing strategyDave Chaffey
 
Motivating And Compensating Channel Members
Motivating And Compensating Channel MembersMotivating And Compensating Channel Members
Motivating And Compensating Channel Membersdcsastudent
 
Social media advertising slideshare
Social media advertising   slideshareSocial media advertising   slideshare
Social media advertising slideshareMJ Social
 
PR plan template free
PR plan template freePR plan template free
PR plan template freeDebbie Leven
 
Chap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional ProgramChap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional ProgramPhoenix media & event
 

What's hot (20)

ITFT Media- AIDA Theory of advertising ppt [read only]
ITFT Media-  AIDA Theory of advertising ppt [read only]ITFT Media-  AIDA Theory of advertising ppt [read only]
ITFT Media- AIDA Theory of advertising ppt [read only]
 
Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona Tech and Venture Capital in the Time of Corona
Tech and Venture Capital in the Time of Corona
 
Advertising Effectiveness
Advertising EffectivenessAdvertising Effectiveness
Advertising Effectiveness
 
Why Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategyWhy Product-Led Growth is the most effective GTM strategy
Why Product-Led Growth is the most effective GTM strategy
 
From startup to scale-up
From startup to scale-upFrom startup to scale-up
From startup to scale-up
 
Advertising Testing Techniques
Advertising Testing TechniquesAdvertising Testing Techniques
Advertising Testing Techniques
 
Top 10 marcom interview questions with answers
Top 10 marcom interview questions with answersTop 10 marcom interview questions with answers
Top 10 marcom interview questions with answers
 
Writing a PR Plan
Writing a PR PlanWriting a PR Plan
Writing a PR Plan
 
Dagmar Approach
Dagmar ApproachDagmar Approach
Dagmar Approach
 
Outdoor advertising
Outdoor advertisingOutdoor advertising
Outdoor advertising
 
AIDA
AIDA AIDA
AIDA
 
Integrated GTM Strategy Case Study
Integrated GTM Strategy Case StudyIntegrated GTM Strategy Case Study
Integrated GTM Strategy Case Study
 
Impact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online ShoppingImpact of Digital Marketing on Online Shopping
Impact of Digital Marketing on Online Shopping
 
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
Affiliate Marketing in India - Presentation at Internet Retail Expo by Swati ...
 
Print Ads
Print AdsPrint Ads
Print Ads
 
Social media marketing strategy
Social media marketing strategySocial media marketing strategy
Social media marketing strategy
 
Motivating And Compensating Channel Members
Motivating And Compensating Channel MembersMotivating And Compensating Channel Members
Motivating And Compensating Channel Members
 
Social media advertising slideshare
Social media advertising   slideshareSocial media advertising   slideshare
Social media advertising slideshare
 
PR plan template free
PR plan template freePR plan template free
PR plan template free
 
Chap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional ProgramChap19 Measuring The Effectiveness Of The Promotional Program
Chap19 Measuring The Effectiveness Of The Promotional Program
 

Similar to Lean Startup Metrics & Analytics

Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist Bryan Ferguson
 
Applying the science of measurement to the art of advertising - 20 january 2...
Applying the science of measurement to the art of advertising  - 20 january 2...Applying the science of measurement to the art of advertising  - 20 january 2...
Applying the science of measurement to the art of advertising - 20 january 2...Ron Jacobs
 
Hotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with MetricsHotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with MetricsALPHA Camp
 
Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)wedu, Inc
 
The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution WP Engine UK
 
The Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth HackingThe Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth HackingFrederik Hermann
 
The Freemium Model and A/B Testing
The Freemium Model and A/B TestingThe Freemium Model and A/B Testing
The Freemium Model and A/B TestingWebs
 
Funnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GAFunnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GASofia Quintero
 
Social Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptxSocial Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptxJeff Salzsauler
 
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Percussion Software
 
How to measure Social Media ROI
How to measure Social Media ROIHow to measure Social Media ROI
How to measure Social Media ROIScott K. Wilder
 
Conversion Whitepaper
Conversion WhitepaperConversion Whitepaper
Conversion WhitepaperWSI Ensenada
 
Lean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer RelationshipsLean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer RelationshipsAaron Eden
 
How To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget RandolphHow To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget RandolphFeverBee Limited
 
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...Bridget Randolph
 

Similar to Lean Startup Metrics & Analytics (20)

Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist Growth Hacking - 10 Key Checklist
Growth Hacking - 10 Key Checklist
 
Multi-Channel Attribution - Where do Leads Come From?
Multi-Channel Attribution - Where do Leads Come From?Multi-Channel Attribution - Where do Leads Come From?
Multi-Channel Attribution - Where do Leads Come From?
 
Applying the science of measurement to the art of advertising - 20 january 2...
Applying the science of measurement to the art of advertising  - 20 january 2...Applying the science of measurement to the art of advertising  - 20 january 2...
Applying the science of measurement to the art of advertising - 20 january 2...
 
Hotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with MetricsHotspot ALPHA Camp_Setting Course with Metrics
Hotspot ALPHA Camp_Setting Course with Metrics
 
Startup analytics
Startup analyticsStartup analytics
Startup analytics
 
Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)Social Media Business Intelligence (SNHU Bootcamp Presentation)
Social Media Business Intelligence (SNHU Bootcamp Presentation)
 
The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution The Future of Analytics: Multichannel Attribution
The Future of Analytics: Multichannel Attribution
 
Growth Hacking
Growth HackingGrowth Hacking
Growth Hacking
 
Beginners guide to_growth_hacking
Beginners guide to_growth_hackingBeginners guide to_growth_hacking
Beginners guide to_growth_hacking
 
The Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth HackingThe Beginner’s Guide to Growth Hacking
The Beginner’s Guide to Growth Hacking
 
The Freemium Model and A/B Testing
The Freemium Model and A/B TestingThe Freemium Model and A/B Testing
The Freemium Model and A/B Testing
 
Funnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GAFunnels Workshop Web Summit 2014 @geckoboard @GA
Funnels Workshop Web Summit 2014 @geckoboard @GA
 
Social Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptxSocial Media Conversion, SFU Bus450, Fall 2015.pptx
Social Media Conversion, SFU Bus450, Fall 2015.pptx
 
Digital measurement &amp; analytics
Digital measurement &amp; analyticsDigital measurement &amp; analytics
Digital measurement &amp; analytics
 
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
Community Marketing: Using Customer & Peer Endorsement to Lift Conversions, G...
 
How to measure Social Media ROI
How to measure Social Media ROIHow to measure Social Media ROI
How to measure Social Media ROI
 
Conversion Whitepaper
Conversion WhitepaperConversion Whitepaper
Conversion Whitepaper
 
Lean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer RelationshipsLean Launchpad Tucson - Customer Relationships
Lean Launchpad Tucson - Customer Relationships
 
How To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget RandolphHow To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
How To Convert Digital Engagement Into Leads and Sales - Bridget Randolph
 
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
Feverbee SPRINT 2016 - Convert Engagement Into Leads and Sales - Bridget Rand...
 

More from Nicola Junior Vitto

The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...Nicola Junior Vitto
 
SCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in EnterprisesSCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in EnterprisesNicola Junior Vitto
 
Building Blomming: A startup founder's experience
Building Blomming: A startup founder's experienceBuilding Blomming: A startup founder's experience
Building Blomming: A startup founder's experienceNicola Junior Vitto
 
Social Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerceSocial Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerceNicola Junior Vitto
 
Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011Nicola Junior Vitto
 
The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.Nicola Junior Vitto
 

More from Nicola Junior Vitto (8)

The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...The new era of bots, multi-channel messaging solutions, dynamic routing and a...
The new era of bots, multi-channel messaging solutions, dynamic routing and a...
 
SCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in EnterprisesSCRUM & Lean Startup in Enterprises
SCRUM & Lean Startup in Enterprises
 
Building Blomming: A startup founder's experience
Building Blomming: A startup founder's experienceBuilding Blomming: A startup founder's experience
Building Blomming: A startup founder's experience
 
Lean Startup - the Italian way
Lean Startup - the Italian wayLean Startup - the Italian way
Lean Startup - the Italian way
 
Social Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerceSocial Commerce & the new ways for E-commerce
Social Commerce & the new ways for E-commerce
 
Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011Blomming Lean Startup @ Better Software 2011
Blomming Lean Startup @ Better Software 2011
 
The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.The Lean Startup and Customer Development model. In Italy.
The Lean Startup and Customer Development model. In Italy.
 
SQL vs Code
SQL vs CodeSQL vs Code
SQL vs Code
 

Recently uploaded

PDT 88 - 4 million seed - Seed - Protecto.pdf
PDT 88 - 4 million seed - Seed - Protecto.pdfPDT 88 - 4 million seed - Seed - Protecto.pdf
PDT 88 - 4 million seed - Seed - Protecto.pdfHajeJanKamps
 
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלימיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגליBen Shushan & Co. C.P.A (Isr.)
 
Entrepreneurship and individuals workshop
Entrepreneurship and individuals workshopEntrepreneurship and individuals workshop
Entrepreneurship and individuals workshopP&CO
 
Slicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinSlicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinAnton Skornyakov
 
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGlokeshwarmaha
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdfChicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdfSourav Sikder
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access
 
Human Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and AppraisalHuman Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and Appraisalgomezdominic3
 
Talent Management research intelligence_13 paradigm shifts_20 March 2024.pdf
Talent Management research intelligence_13 paradigm shifts_20 March 2024.pdfTalent Management research intelligence_13 paradigm shifts_20 March 2024.pdf
Talent Management research intelligence_13 paradigm shifts_20 March 2024.pdfCharles Cotter, PhD
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024Stephan Koning
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023Steve Rader
 
Developing Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, OursDeveloping Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, OursKaiNexus
 
Transform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in LauncestonTransform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in Launcestondjhbuildersau
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsWristbands Ireland
 
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptxHELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptxHelene Heckrotte
 
MC Heights construction company in Jhang
MC Heights construction company in JhangMC Heights construction company in Jhang
MC Heights construction company in Jhangmcgroupjeya
 

Recently uploaded (20)

PDT 88 - 4 million seed - Seed - Protecto.pdf
PDT 88 - 4 million seed - Seed - Protecto.pdfPDT 88 - 4 million seed - Seed - Protecto.pdf
PDT 88 - 4 million seed - Seed - Protecto.pdf
 
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלימיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
מיסוי רילוקיישן לפורטוגל- היבטי מיסוי ישראלי ופורטוגלי
 
Entrepreneurship and individuals workshop
Entrepreneurship and individuals workshopEntrepreneurship and individuals workshop
Entrepreneurship and individuals workshop
 
Slicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinSlicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup Berlin
 
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdfChicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
Chicago Medical Malpractice Lawyer Chicago Medical Malpractice Lawyer.pdf
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024
 
Human Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and AppraisalHuman Resource Management Chapter 9: Performance Management and Appraisal
Human Resource Management Chapter 9: Performance Management and Appraisal
 
Talent Management research intelligence_13 paradigm shifts_20 March 2024.pdf
Talent Management research intelligence_13 paradigm shifts_20 March 2024.pdfTalent Management research intelligence_13 paradigm shifts_20 March 2024.pdf
Talent Management research intelligence_13 paradigm shifts_20 March 2024.pdf
 
Deira Dubai Call Girls +971552825767 Call Girls In Downtown Dubai
Deira Dubai Call Girls +971552825767 Call Girls In Downtown DubaiDeira Dubai Call Girls +971552825767 Call Girls In Downtown Dubai
Deira Dubai Call Girls +971552825767 Call Girls In Downtown Dubai
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023
 
Developing Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, OursDeveloping Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, Ours
 
Investment Opportunity for Thailand's Automotive & EV Industries
Investment Opportunity for Thailand's Automotive & EV IndustriesInvestment Opportunity for Thailand's Automotive & EV Industries
Investment Opportunity for Thailand's Automotive & EV Industries
 
Transform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in LauncestonTransform Your Kitchen Essential Tips for Renovations in Launceston
Transform Your Kitchen Essential Tips for Renovations in Launceston
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and Festivals
 
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptxHELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
HELENE HECKROTTE'S PROFESSIONAL PORTFOLIO.pptx
 
MC Heights construction company in Jhang
MC Heights construction company in JhangMC Heights construction company in Jhang
MC Heights construction company in Jhang
 

Lean Startup Metrics & Analytics

  • 1. (Lean) Startup Metrics & Analytics Nicola Junior Vitto - @njvitto @blomming September 26, 2012 - Better Software - Firenze, Italy
  • 2. Entrepreneurship ...Entrepreneurs recognize they are going to have to follow their own internal compass and embrace the uncertainty as part of the journey. - Steve Blank - more on: http://steveblank.com/2009/06/29/agile-opportunism-entrepreneurial-dna/
  • 3. Get out of the building (by Steve Blank) Your business assumptions can be wrong Go out and speak with your customers
  • 5. Vanity Metrics - Timelines There will always be something up and to the right
  • 6. Vanity Metrics - Google Analytics Forget it!
  • 7. Vanity Metrics - Google Analytics ...and this could become like a drug...killing you!
  • 8. Vanity Metrics - TechCrunch & Co “mDialog raises 5 Million from Blackberry partners” “GroupMe is now sending one Million texts every day” “Google Plus hits 10 Million users in 2 weeks”
  • 10. What’s a startup? A startup is first of all an experiment, it is a human institution designed to deliver a new product or service under conditions of extreme uncertainty At its heart, a startup is a catalyst that transforms ideas into product. (From Lessons Learned blog By Eric Ries)
  • 11. What’s a Lean Startup? Lean startup is a rigorous process for iterating from Plan A to a plan that works. (by Ash Maurya)
  • 12. What Metrics Matter? Example: Facebook on early days 150k users, little revenue Many superior competitors What would indicate their future success?
  • 13. What Metrics Matter? Example: Facebook on early days 75% of users visit one or more times per day Within one month of launching on a new campus, can acquire 90% of the students
  • 14. What Metrics Matter? Good Metrics (Three A’s) Are Actionable Can be Audited Are Accessible
  • 15. Actionable Metric An actionable metric is one that ties specific and repeatable actions to observed results (by Ash Maurya)
  • 17. A/B Testing - New Blomming home VS Define your Goals
  • 18. A/B Testing - Define your goals A/B Testing Goals One single important Goal: % Users that sign-up for the service (as a seller) Secondary Goal: % Buyers that buy something after landing on the homepage
  • 20. Multivariate Testing at Google “Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see whic one performs better. I has a recent debate over wheter a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that.” (by Doug Bowman on Design at Google)
  • 23. Multivariate Testing at Google Google Bing
  • 24. Multivariate Testing at Google “That blue was worth at least $80M” (by Paul Ray - Product Manager at Bing)
  • 26. David Cancel’s Funnel The Problem: 98% of people who visit your site, leave never to return
  • 27. David Cancel’s Funnel 2% “The average conversion rate in the United States of America wheter you’re selling elephants or iPods” (by Avinash Kaushik - Analytics Evangelist, Google)
  • 29. David Cancel’s Funnel The Solution: To find, engage and convert more of your website visitors into customers
  • 30. Online Marketing is shifting
  • 31. Online Marketing Paradigm Shift by David Cancel
  • 32. Online Marketing Paradigm Shift by David Cancel
  • 33. Online Marketing Paradigm Shift by David Cancel
  • 34. Online Marketing Paradigm Shift Internet Marketing 3.0 (different goal) by David Cancel
  • 35. Lifecycle Marketing by David Cancel
  • 37. Optimize for learning Optimize for learning by creating a data-driven marketing framework
  • 38. Optimize for learning Analytics & A/B Testing are also not exactly the answer
  • 39. Optimize for learning Use quantitative & qualitative data to test *your* assumptions
  • 40. Optimize for learning The point of testing is to prove or disprove assumptions quickly
  • 41. Optimize for learning Pay attention to “overtesting” using Multivariate testing because it generally takes too long
  • 42. The Local Maxima Problem
  • 44. AARRR
  • 45. AARRR Dave McClure’s Startup Metrics for Pirates is a good framework for thinking about macro metrics
  • 48. AARRR - Different orders (example) 1. Activation, Retention (happiness) 2. Revenue (sustainability) 3. Acquisition, Referral (virality) (this is the Ash Maurya or 37signals order: “Charge from day one”)
  • 50. Tracking Marketing Channels Google Adwords Banners Social Media Partnerships PR Blogging ...anything that links to your site!
  • 51. Tracking Marketing Channels Use unique urls (tracking parameters) on every url you create/give-out/pay for Use bit.ly Use HTTP referrers to group PR
  • 52. Measuring Cost of Visits Cost per click (site visits) Cost of campaign / Traffic generated (Social media) Cost of effort / Traffic generated
  • 53. Measuring Cost of Acquisition Cost of one visit / conversion rate = Cost of an Acquisition Acquisition could be paid for acquisition of free sign-up ...unfortunately is not so simple
  • 54. Marketing Channels Per channel track: Scale Cost Conversion Optimise and scale channels with best Cost of Acquisition (CoA) and potential to Scale
  • 55. Acquisition Cost of Acquisition < Customer Lifetime Profit Then scale acquisition at a similar or lower rate A/B test acquisition channels to optimise
  • 56. Acquisition including Referral If it costs €3 to acquire a customer through advertising But a customer brought in 2 other customers through referral The COA of that channel is €1
  • 58. Activation Sure, they signed-up!!!
  • 59. Activation Sure, they signed-up!!! BUT did the like it???
  • 60. Activation at Facebook (i) which data points predict whether a user will stay? (ii) if they stay, which data points predict how active they’ll be after three months?
  • 61. Activation at Facebook (i) which data points predict whether a user will stay? having more than one session as a new user and entering basic profile information. (ii) if they stay, which data points predict how active they’ll be after three months? how often a user was reached out by others, frequency of third party application use, and how available a user was on the site.
  • 62. Activation at Facebook “At the time, I heard many people criticize Facebook’s early investors, claiming that Facebook had ‘no business model’ and only modest revenues relative to the valuation offered by its investors...” “...Facebook was different, because it employed a different Engine of Growth. It paid nothing for customer acquisition, and its high engagement meant that it was accumulating massive amount of customer attention every day.” (Eric Ries - The Lean Startup)
  • 63. Activation Metrics Activation metrics often drive other metrics in Referral, Retention, Revenue
  • 65. Returning Users Tracking unregistered users that return often (useful for content sites with advertising) Tracking registered users that return and take some sort of action.
  • 67. Twitter Cohorts (example) by Fred Wilson’s blog (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
  • 68. Twitter Cohorts (example) by Fred Wilson’s blog (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)
  • 70. Referral Mechanism Social Media Sharing Invite a friend type mechanism App Reviews Word of Mouth
  • 71. Viral Coefficient The average number of customers that each customer refers
  • 72. Viral Coefficient - Example Each company customer invites 5 other customers via email 20% click on that email link 50% of those users convert into customers
  • 73. Viral Coefficient - Example Viral Coefficient = Referrals x click through x conversion 5 x 0.2 x 0.5 = 0.5
  • 74. Viral Coefficient A viral co-efficient greater than 1 means that every customer gets more than one other customer on average
  • 75. Viral Coefficient A viral co-efficient greater than 1 means that every customer gets more than one other customer on average Therefore your product will grow virally
  • 76. Viral Coefficient Measure automatically for all your built in referral mechanism Give customers unique invite URLs and track conversions
  • 77. Viral Cycle Time Viral Cycle Time = the average time taken for a referral to turn into a customer With the viral coefficient and viral cycle time you can determine your viral growth
  • 78. Net Promoter Score Net Promoter Score: Number of people who would recommend your product to a friend A single question survey
  • 79. Net Promoter Score “How likely is it you would recommend my product to a friend or colleague?” 0 1 2 3 4 5 6 7 8 9 10 NPS = % Promoters - % Detractors
  • 80. Net Promoter Score Survey a small percentage of your customers by email weekly Record who you sent to so you don’t repeat the same people every week
  • 81. Net Promoter Score NPS is not a direct measurement that translates to key metrics Can act as a leading indicator of other metrics
  • 83. Making Money Key for any business is to find out how much profit they make for every customer and scaling the number of customers
  • 84. Lifetime Customer Value How much money you make for every customer you acquire For some this a single transaction For subscription businesses it depends on retention
  • 85. Cancellation Rate Cancellation rate = is the percentage number of customers who cancel in any given month compared to total (paying) customers [Cancellation rate] = [product utility] + [service quality] + [acceptable price]
  • 86. Lifetime Customer Value (LTV) LTV = monthly revenue x n° months in lifetime N° months = 1 / cancellation rate LTV = monthly revenue / cancellation rate
  • 87. Lifetime Customer Value (LTV) For example: If €20 a month and cancellation rate is 10% a month LTV = €200
  • 88. Lifetime Customer Value (LTV) Often not that simple. Short term cancellation rates are much higher than long term Use Cohort Analysis to determine
  • 89. Lifetime Customer Value (LTV) r = short-term cancellation rate (e.g. 0,15) p = long-term cancellation rate (e.g. 0.03) s = number of months in the “short-term” age group (e.g. 3) (1-r)^s x (s + 1/p) = expected months
  • 94. Customer lifecycle management
  • 97. Tips
  • 99. Innovation Accounting “To improve entrepreneurial outcomes, and to hold entrepreneurs accountable, we need to focus on the boring stuff: how to measure progress, how to setup milestones, how to prioritize work. This requires a new kind of accounting, specific to startups” (Eric Ries - The Lean Startup)
  • 100. Innovation Accounting Concentrate on macro metrics Everything you do should attempt to change a metric Try to use as few tools as possible
  • 105. SCRUM Do you know SCRUM?
  • 106. Kanban Board I know: this isn’t an example of an exciting and well done Sprint :)
  • 107. (Lean) Product Management Well, trying to connect the dots: what we learnt so far is how the Product Owner can define and prioritize stories at every Sprint in a Startup that wants to be called, you know, Lean! :)
  • 108. Learning Boards - Lean Stack We will talk about the Lean Stack of Ash Maurya or other possible “lean mapping” frameworks in a future presentation
  • 110. Credits Stuart Eccles - Lean Startup Metrics Joshua Porter - Metrics-Driven Design David Cancel - Data-Driven Marketing Lean Startup Machine - How to measure the metrics that determine real progress (and the others specified directly in the slides when picking some parts of books or blogs)
  • 114. An alternative? Lifestyle business
  • 115. Thanks! Blomming.com Company.Blomming.com Magazine.Blomming.com Help.Blomming.com Twitter.com/Blomming Facebook.com/Blomming nicola@blomming.com On FB & Twitter as: njvitto

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. \n
  58. \n
  59. \n
  60. \n
  61. \n
  62. \n
  63. \n
  64. \n
  65. \n
  66. \n
  67. \n
  68. \n
  69. \n
  70. \n
  71. \n
  72. \n
  73. \n
  74. \n
  75. \n
  76. \n
  77. \n
  78. \n
  79. \n
  80. \n
  81. \n
  82. \n
  83. \n
  84. \n
  85. \n
  86. \n
  87. \n
  88. \n
  89. \n
  90. \n
  91. \n
  92. \n
  93. \n
  94. \n
  95. \n
  96. \n
  97. \n
  98. \n
  99. \n
  100. \n
  101. \n
  102. \n
  103. \n
  104. \n
  105. \n
  106. \n
  107. \n
  108. \n
  109. \n
  110. \n
  111. \n
  112. \n
  113. \n
  114. \n
  115. \n