This is a talk given to my class on User Experience by Jen Ruffner, a Product Manager on the art of optimization.
It is critical for modern designers, product managers and start-up folks ot understand how to think about designing and executing tests.
13. Features and your KPIs/OKRs
Create Company and Team Wide OKR’s (Objective and
Key Results)
Specific, measureable objectives that every feature in your
roadmap should fall under.
Identify Key Performance Metrics that support the company
OKRs
Example OKRs
Increase Daily Active User; N DAU per day
Increase Revenue; $N Revenue per day
Maintain Net Promoter Score (NPS); >25
Increase social actions; N Social Interactions pp/day
14. Roadmap Planning
Given our 2-3 KPI’s; what features will support them?
Start with sub metrics that support those features
Build features that support them
Which features do you prioritize? How many dev days should it get? It’s about
diligence.
Brainstorm
What features do we want this quarter?
Hypothesize
What metrics will those features move?
Expected Outcomes vs Dev Days
What are the goals of the feature?
What metrics do we all agree no that we will use to measure the success of this feature?
How many dev days is the feature? How much revenue, engagement, and/or social actions will we
get out of it?
Every 12-15 dev days = 1 PM day estimating
Build
Analyze and Discuss
Weekly team Company wide reviews discussing learnings
Every 4 dev days = 1 PM day analyzing
18. Estimate and Expected Day
Measure in Funnels
When figuring out how
to track a feature, start
by building out the Energy pinch was an unnecessary gate on virals
funnel.
Use comparable data to
fill in the blanks for
your forecast
20. User Acquisition: Where does traffic come
from?
Email
Mainly invitations from friends, or users sharing content
Organic
The best kind! User’s coming directly to your site
SEO
25% of LinkedIn’s new users came from search
Ads/Partnerships
Social Feeds
Facebook, LinkedIn, Twitter, Pinterest
Wherever possible, use user generated content as messaging to increase the response to
your outreach. What feature within your site prompt generating user generated content? It’s
likely those have your highest conversions and are a place you could start to double down.
http://blog.adamnash.com/2012/03/28/product-leaders-user-acquisition-series/
21. Measuring Virality
You are viral when every 1 user brings in more than 1 new
user
K-Factor (wikipedia)
i = number of invites sent by each customer c = percent conversion
of each invite K-Factor = i * c
Z-Factor (Mixpanel/Dave McClure)
x = % of users who invite
y = avg number of people they invited
z= % of users who accepted an invitation
Z-Factor =x*y*z
22. The often overlooked piece of virality
Cycle Time
“Given that I get a new customer today, how many new customers
will they bring in over the next N days?”
Let’s assume a viral factor of 0.5, and an N of 7. User
Acquisition will look like the below which can be
used for projections:
1 + 0.5 + 0.25 + 0.125 ….
24. Get to & complete registration
Registered users is a key metric for success
Once a user is registered:
You have a way to reach out to them
You have some key demographics to target advertising
25. Sample Invitation Funnel
Data Notes
Receipts
Receipts/Recipient Are you sending too many per day? Causing spam to
increase?
Recipients
% getting inboxed IP Sender Score, Domain Reputation
Inboxed emails
% Opened Sender Name, Subject, Time Sending etc.
Opened Emails
% Clicked On Messaging for CTAs, color of CTA, # of CTAs,
Placement of CTA (above the fold)
Unique Clickers
26. Email Test Checklist
Delivery Theme Tests
Sender Friend name (Jen, Jen Ruffner, Jen R.), Unique ID,
Company
Subject Under 60 characters, test different messaging tactics
below
Body Messaging tactics, CTA’s above the fold, # of CTA’s,
Time Highest open rate given the hour in users time zone,
Other performance issues associated with email blasts?
Frequency Digest information, Aggregate messages, Test opt-out
positions vs spam rates
Messaging Theme Tests
Statement Type Declarative, Imperative, Interrogative, Conditional
Sentence Tense Past, Present, Future
Persuasiveness Light vs. Heavy
Wildcards !! … ;) >>
35. Great that it’s an
HTML button, but I
would change the text
to “See 15 more >>”
Messaging is all about
me and my profile,
playing to my vanity.
36. Subject line is
compelling that there
is a specific number of
things waiting for me
to see.
Personally I wouldn’t
put the main message
on a grey background,
but the faces help to
offset that.
Unsub info is grey text
on a grey background
which helps you
ignore it.
46. Increasing Conversion
Users Users
Entering Completing
Flow Flow
- Make it dead simple
- Increase Entry Points
- Hold their hand
- Overlay social on everything
- Reward them
- Test everything
Introduce social learning throughout these flows to teach the
user how to use the site
Editor's Notes
Hi My name is Jen Ruffner. As Christina mentioned I’m going to talk about Site Flow OptimizationBefore before I jump into a website --- I want to visit an experience that we are all familiar with – eating out. I’ll talk about some of the industry secrets that restaurants use to improve the check’s bottom line. What’s interesting is that some of them translate very nicely to what you can do on a website, and some just seem to go against what we would believe to be common sense.
High margin items in the upper right hand corner of the menu “1978 issue of The Cornell Hotel and Restaurant Administration Quarterly”High margin items in the upper right hand corner of the menu
And then counter clockwise to the right bottom ---- Which is why you see appetizers in the top left“People tend to remember the top two items on a list and the bottom item” Steve Miller This lesson here is that every site is different. Some things work everywhere and other sites have communities that go against the norm. You should always be iterating by learning from your users and testing new approaches.
Much like search results
Reduce any hurdle that you can see between the user and what you want them to do. In this case, not only does the user have to remember to ask the waiter what the market price is before ordering, but they also risk embarassment
Things I told CW – Game Mechanics (pinch, unlock, expiration, handholding, tell users what to do)FunnelsData every dayDesigners are part of the triangle
Best practice from Zynga and also something that mixpanel heavily preaches is to think about everything in funnels. When you’re planning
If your feature isn’t simple enough to fit in a Google drawing, adoption will be difficult. Every major step that you add has 10-25% drop off. Increasing handholding (like arrows, familiar faces, additional marketing and reminders) reduces drop off between steps. You can literally tell your users want to do.
Most important part of feature development is expected outcomes. - start by agreeing on goals how you would measure success. Often times what you think is a simple If you don’t have great comps, guess
Email - Drives growth, engagement, and retentionOrganic – the best kind which usually yields the highest retention. Cold joins, word of mouth, Seo Search – 25% of LI’s traffic comes from hereAds - $250m/year spend by ebay. Under $2 conversion rate on FB for an app install is pretty good. Social Feeds – Works really well when people take actions frequently (ex. Listened to a song)
Avg Viral coefficients Tagged went viral – but mostly avg’d above 0.4LinkedIn hovered around 0.3
The first flow that I’m going to walk through is registration. Registration is obviously an important flow for just about any web site.As I walk through the flow, I’m going to pointing out very small details that can make a big difference to the bottom line. While we’re walking through these slides, I’d encourage you to think of a few flows on your site where some optimization would be beneficial. Looking at the same site day in and day out, it’s sometimes easy to overlook some obvious changes. Hopefully this presentation will help you get in the mindset for optimization and look at things with a fresh eye.
Subject lines are easy to test for a lot of reasonsLook at who’s hot right now. Try to guess their hypothesis behind the test and see if you should test it out yourself.
One of the most popular ways to hear about a social site is through a friend Objective is to get people to open the email. How do you do that?Who is it sent from?Should use the inviter’s name – not the siteTwitter doesn’t Users are more likely to open an email when it’s from a friend instead of a website. Twitter doesn’t even mention my name in the subject line. People won’t recognize JenG24Is the subject line short, but compellingDoes it make you feel wanted and special?Twitter doesn’t even use my name in the subjectFacebook – first off when I sent this email it wasn’t to share photos. It was just to invite my friend to join the site. Second, it doesn’t make me feel special.Tagged makes me feel good because something is waiting there and it’s specifically for meSuggestions1. Make the reader feel like they need to respond, while still keeping it light.2. Wildcards – try jazzing up the subject line to see if it does anything [SMILEY FACE]
GoodPicture is great for conversation AND it clicks to registerThe stats about me tell the story of what you can do on Facebook (social learning)
BadToo many links (make one clear call to action and lock down the flow)Note that one link is in case you already have an account. Normally I’d say to remove it, but I’d imagine that FB gets a lot of CS requests to merge duplicate accounts. If that is costing a lot of money it may be worth keeping that link here).Too much text – don’t make the user read a story here – get them to join the site and let them learn from their friends.Call to action is to sign up for facebook – not look at Jen’s photosSuggestions – Images can be scary to put in the email because the user may not download it. In that event add text about where I live or where I went to school etc.
Good Buttons instead of a link is a very clear call to action.Both buttons click to reg – if you click no you register without becoming friends with the inviter.Posed a question which envokes the user to respond.Bad- The messaging is a little harsh
GoodNo navProgress barGood header - it’s inviting and rewardingHeld the users hand with info on the inviter and allows you to get more information inlineHandholding text as to why you should do certain things (rewarding the user)Country is prepopulatedThe button is a good color and has a good message. Buttons are important not only because they are the main call to action, but it’s the last thing that a user looks at before they decide whether or not to continue.BadGet rid of the marketing textAllow the user to skip things that aren’t optional
You’ve got to get it to one page. Why? Because once you have the user’s email, you can reach out to them and try to get them to come back. If you bog them down with excess requested information, you’re going to lose them. Email addressGoodHolding the users hand by keeping the inviter at the top of the page w/ good messaging like “Join your friends on MS”Mentions the word “Free”Absolute minimal requirements. - Prepopulated the email addressOrder!Submit button is above the fold and it’s a good clear call to askingIT’s a good idea that any page in this flow, the submit button needs to be above the fold. That doesn’t mean that you should take what was 3 really long pages and turn them into 6 short pages, because there is a drop off rate of about 20% per page. Make sure you get as much as you can and need before you’ve lost the opportunity to land them on the home page and let them do what they came there to do.Got rid of the checkbox that they agree to the TOSKept the captcha until later
BadGet rid of the navCirsor not defaultedShould have default selected one of the gendersFull looks like first
For a social website -- your goals are probably include growth, engagement, and revenue. So key flows and conversions you should be focussed on include: - getting a user registered- getting them to submit content or share something- or getting them to buy something….