Product lessons from the launch of Unified Search, a massive redesign of LinkedIn's search experience, presented at the Stanford Graduate School of Business High Tech Club on May 16, 2014
4. 5B queries a year
One of the top visited pages on LinkedIn
5. Why fix it?
• Selling us short – Discoverability
LinkedIn has more to offer than just people search, but
the other verticals don’t get discovered enough
• Inflexible - Not easy to iterate
Each search vertical is built on a different stack with low
leverage across verticals
• Design cadence with the rest of the site
Search verticals look completely different from each
other, and LinkedIn is doing a site-wide redesign
13. Change 1: The Vertical Selector
Before Unified Search
Searcher gets control of what
they are searching for
(People, Jobs, Companies,
Groups)
Unified Search
We remove the vertical selector
14. Change 2: Intent detector
Before Unified Search
Searcher specifies intent
explicitly
Unified Search
We algorithmically predict the
searcher’s intent
Query: “marketing”
?
19. And measured the isolated impact of major changes
Metrics
A/B tests
20. We optimized for speed of learning
• Quick experimental iterations designed to
answer the most burning questions
• Design -> Spec -> Dev -> QA -> Prod in
~1 week.
We identified and ramped the winning
changes
And either iterated on or killed the losing
ones
21. 3 months and ~30 experiments later
June 25, 2013: 100% en-US Launch
And we eventually rolled out Unified Search
100% worldwide over the following 3 months…
22. Results
Engagement from Search
(Page Views & Actions)
Searchers per vertical
Dead-end searches
Revenue from search
24. #1: Have opportunity analysis drive goal setting
Was an X% increase in searchers per vertical a realistic goal?
eg: How much of search traffic can we realistically expect to
distribute from people to other verticals with unified search?
25. Opportunity unclear? Test the waters - Quickly
Example of a test we ran within one dev quarter:
Structured suggestions to clarify user’s vertical intent
26. #2: Importance of controlled experimentation
• To understand the isolated impact of each
major change
• Especially so when you are changing
something working well
• Even when the combination of changes is
a huge net win (so we know what led to
the win)
• Often mistaken with going after
“incremental” wins - Disruptive changes
can be executed incrementally and can be
tested in a controlled fashion
29. “You have to kiss a lot of frogs to find one
prince. So how can you find your prince
faster? By finding more frogs and kissing
them faster and faster.”
Mike Moran
Do It Wrong Quickly: How the Web Changes the Old
Marketing Rules, 2007
30. #3: Agility in a crisis
Product launches after 1 year in development.
Metrics drop.
Panic sets.
All hands on deck.
Huge number of (emotional) people involved.
Huge number of options.
Behind schedule on ramp.
The clock is ticking.
Often the time for drastic measures…
And yet, it is important to be agile. We made controlled changes,
executing quickly and taking rational decisions based on data
31. Organizational alignment critical to pull this off…
• Product/Design: Micro-prioritization, mini-specs for
experiments with clear hypotheses
• Web-dev/Apps: Time-box efforts, limited scope (eg:
launch test in a subset of locales or browsers)
• Relevance: Practical hand-tuned approaches
• Analytics: A/B dashboards and custom analysis
• QA: Minimal QA automation & more manual checks
until test succeeds
• SRE/Ops: Frequent deployments
32. 3 Key Takeaways
• Analyze opportunity & test the waters
early, quickly, cheaply
• Control your biggest changes to
understand isolated impact
• Stay agile when things go wrong
History… personal take… Day 1…
Not showing numbers
-14% actions
-6% page views
-1% searchers
+10% dead-ends
-$X revenue
Talk through this…
3 major changes…
Expand CTA
Guess the biggest driver that turned things around, from this list of three
You can run multiple, not just conflict with each other… as a reason for why we need infrastructure for it… conceptually simple – but infra needs to support it… experimentation platform… take advantage.. Simplification of architecture.. Not only did people differ on this but they had strong reasons, backed by strong intuition…\
Explain the process…
Blurred – Readable… Show 3
+10% actions
+2% page views
Flat on searchers and dead-ends
+$X weekly ad revenue
Climax – story telling – Positioning unified search…
Platform a requirement for product changes. Talk about unified search as a platform…
How do you get a team that is used to building a product with a 1 year time-frame in mind to suddenly be iterative and experimental?
What hypotheses are worth testing? How much effort do you put into a particular test… What makes a test a good test…
Take slide from Kohavi’s talk
Frogs and faster… from management…