Relevance denotes how well a search result satisfies the user information need. In addition to the search engine components (i.e., indexer and query parser), there are many other components that impact relevance. e.g., user understanding , data optimization, domain knowledge, etc. Improving relevance remains the main and most challenging goal of each search engine. Indeed, relevance can be subjective, therefore hard to measure and to improve. In this talk, Saïd will demystify the concept of relevance by defining its main components. For each component, he will present the technology enablers, the data, and processes that are required in order to measure and improve relevance. In this talk, attendees will learn how to provide a relevant user experience and track it over time.

1. Anatomy Of Relevance: From Data To Action
Saïd Radhouani, Ph.D.
radhouani
Yellow Pages, Canada

2. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion

3. Who we are
• Leading media and marketing solutions company in Canada
• Champion the neighbourhood economy by connecting businesses
and consumers
300 000
Local and national advertisers
7.3 million
Unduplicated unique visitors to our network of websites
6.5 million
Downloads of our mobile apps
26%
Internet users reached in Canada each month

4. Who I am
• Entrepreneur
• Ph.D. in Search and Knowledge Management
• IT Director – Content, Search and Relevance
• Noble cause: leverage data to help build smarter organizations

5. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion

6. Local Search

7. The Importance Of Relevance
Businesses Consumers
Source:
unknown

8. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion

9. User
Objective
Understand the user intent to maximize their satisfaction
Data
• Who the user is (individual or group)
• Where they come from (direct, SEs, …)
• Their context (Location, TOD/DOW/Season, Device, …)
• Their interactions history
KPIs
• Coverage: %tracked user, %tracked user interactions
• Freshness: Daily updates of User profiles & interactions
User

10. Content
Objective
Have accurate and “fresh” content to satisfy your audience needs
Data
• Enriched Directory Content (Merchants, Products, Deals, etc.)
• Rich Editorial Content (Neighborhoods, Smart Tips, etc.)
• Third-party content (social networks, etc. )
• User-Generated Content
KPIs
• Coverage: %Canadian businesses, %Neighborhoods
• Completeness: e.g., for each merchant, have all required info
• Freshness: Frequency of validation & updates Content
User

11. Knowledge
Objective
Help to bridge the gap between user queries and content data
Data
• Linguistic (synonyms, acronyms, etc.)
• Rich and “fresh” geo data (location names, polygons, etc.)
• Multilingual
KPIs
• Coverage: %searched entities (keywords, location names, etc.)
• Completeness: e.g., for each location: centroid, polygon,
synonyms, relationships, etc.
• Freshness: Frequency of validation & updates
Content
Knowledge
User

12. Search
Objective
Action-driven, personalized and contextualized
relevant search experience
KPIs
Offline
• Recall & Precision
Online
• #returned results
• Mean Reciprocal Rank (MRR) – the multiplicative inverse of
the rank of the first correct answer. Perfect SE should have a
MRR of 1
Content
Knowledge
Search
User

13. Presentation
Objective
Allow users to access quickly to actionable content through
optimal & dynamic search results presentation
e.g., “Movie Theater” vs. “Restaurant”
KPIs [that really matter]
• Page Views
• Unique Visitors
• Visit Duration
• Bounce Rate
• Return Visits Frequency
• (Useful) Interactions per Visits
Search
Presenta4on
Content
Knowledge
User

14. Optimization
UserSatisfaction improveRelevance()
{
While (alive)
{
Act;
Measure;
Learn;
Grow;
}
}
Search
Presenta4on
Content
Op4miza4on
Knowledge
User

15. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion

16. A
problem
well-­‐stated
is
half-­‐solved.
Where To Start?
Charles
Ke2ering

17. Data-Driven Actions
Measure Act
• Implicit User Feedback (analytics & logs)
• Stable Test Collection
• Side-By-Side Comparison
• A/B Testing
• Explicit User Feedback (Survey)
• Find gaps in data and improve them
• Improve processes
• Improve presentation
• Improve algorithms
• Confirm hypothesis
KPIs

18. Know Your Data
• Web & Mobile Analytics
• Search logs Analysis
• Build KPIs

19. From Data To Action
Examples:
• Missing geo-location
• Missing businesses
• Seasonal trends (adapt search algo)
Manual
or
automa8c
ac8ons

20. Test Collection
KPIs: Recall & Precision
• Predefined test collection built manually
• Variety of use cases that cover most users needs
• Evaluate new updates before going to production

21. Side-By-Side Comparison
• Comparison of two (versions of the)
Search Engines
• Pre-defined set of real searches selected
from logs
• Pre-classified (keywords, business
names, language, geo location, etc.)
• “Blind” comparison
• Significant number of testers per search

22. A/B Testing
Principle
• Experiment one or many ingredients at a
time with real users
• Some non-trivial amount of traffic is
always being tested
KPIs
• Page Views
• Unique Visitors
• Visit Duration
• Bounce Rate
• Return Visits Frequency
• Useful Interactions per Visits
Simulation
• Run new versions on a significant
search logs sample
• Estimate the impact prior
deployment to prod and avoid bad
surprises

23. Relevance In Action
Valuable signals, NOT only for marketing!
e.g., Bounce Rate – Why the user left quickly?
• Who is the user?
• Where they come from (direct vs. SE)
• Their devices
• Which content was provided?
• Which page?
• etc.

24. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion

25. Conclusion
Stop coding and make your data talk!

26. Anatomy Of Relevance: From Data To Action
Saïd Radhouani, Ph.D.
radhouani
Yellow Pages, Canada