Presentation slides at RecSys 2016, Boston. At Quora, our mission is to share and grow the world’s knowledge. Recommender systems are at the core of this mission: we need to recommend the most important questions to people most likely to write great answers, and recommend the best answers to people interested in reading them. Driven by the above mission statement, we have a variety of interesting and challenging recommendation problems and a large, rich data set that we can work with to build novel solutions for them. In this talk, we will describe several of these recommendation problems and present our approaches solving them.

1. Recommending the World’s Knowledge
Application of Recommender Systems at Quora
Lei Yang (leiyang@quora.com)

2. Quora: Share and grow the world’s knowledge
Millions of questions & answers, millions of users, thousands of topics

3. Data and Scale

4. Topic Question
User
Answer
Actions

5. Lots of data relations

6. Complex network propagation effects

7. Complex topics network & semantics

8. Recommendation
Problems

9. Recommending the digest email
Recommend 10 most interesting answers in an
email to get users to re-engage on Quora
● Requirements
○ Relevance
○ Freshness
○ Quality
● Challenges
○ Many candidates
○ Personalized frequency and send times
● Personalized LTR model
○ Predict probability of click on answers
shown

10. Recommending the digest email
● Candidates
○ User/topic follows
○ Globally popular/viral content
○ Collaborative filtering suggestions
■ Relevant writers
■ Relevant topics
■ Relevant answers
● Features
○ Text features
○ User follow graph
○ Historic data
○ ...

11. Recommending writers for a question
Recommend experts who are best qualified
to answer a question
● Ask to Answer (A2A)
● Given a questions and a viewer, predict
how “well-suited” a candidate user is for an
A2A suggestion
● Well-suited
○ viewer send a request &
○ candidate write a good answer

12. Recommending writers for a question
● Candidates
○ Users who have written at least an
answer on one of the topics of the
question
○ A number of filters
● Features
○ Candidate history features
○ Viewer history features
○ Candidate-viewer relevance
○ ...

13. Recommending the home feed
Present most interesting stories for a user at
a given time
● Stories: questions + answers
● Relevance-ordered vs time-ordered = big
gains in engagement
● Challenges
○ Many candidate stories
○ Real-time ranking
○ Objective function

14. Recommending the home feed
● Candidates
○ Different temporal windows
○ Multi-stage solution with different
“streams”
● Features
○ Quality of question/answer
○ Topics the user is interested in
or knows about
○ Users the user is following
○ What is trending/popular
○ ...

15. Recommending topics and users
Recommend new topics and users for
the user to follow, based on
● Topics you already follow
● Users you already follow
● Interactions with
questions/answers/users
● Topic-related features
● Social graph
● …..

16. Many more problems…
Related Questions Sessions
Trending
Topic feed

17. What We’ve Learned

18. Learning to recommend
● ML can often solve recommendation
problems
○ Select candidates
○ ML model to score each candidate
○ Top scored candidates are returned
● Workflow
○ Build data pipeline
○ Define training label
○ Choose your model and metric
○ Create features

19. ML models used at Quora
● Logistic Regression
● Elastic Nets
● Gradient Boosted Decision Trees
● Random Forests
● (Deep) Neural Networks
● LambdaMART
● Matrix Factorization
● LDA
● ...

20. Open source project -- QMF
Quora Matrix Factorization
https://github.com/quora/qmf
● Currently BPR and WALS
● Multithreaded implementation in
C++14

21. Challenges
● Personalization
● Explicit v.s. implicit user feedback
● Explore v.s. exploit and the filter bubble
● Business logic in your system
● Trade-offs between different optimization goals
● Explain the recommendations
● ...

22. Conclusions

23. Conclusions
● At Quora we have not only big, but also “rich” data
● Many interesting recommendation problems and unsolved challenges
● We use ML to solve many of our recommendation problems
● Our algorithms need to understand and optimize complex aspects such
as quality, relevance, or user expertise

24. We are hiring!
www.quora.com/careers