Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics

16 views

Published on

Talk at the 19th Dutch-Belgian Information Retrieval Workshop (DIR). Antwerp, Belgium (online).

Published in: Science
  • Be the first to comment

  • Be the first to like this

Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics

  1. 1. 1 WIS Web Information Systems Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics Tim Draws1, Nava Tintarev1, Ujwal Gadiraju1, Alessandro Bozzon1, and Benjamin Timmermans2 1TU Delft, The Netherlands 2IBM, The Netherlands t.a.draws@tudelft.nl https://timdraws.net
  2. 2. 2 WIS Web Information Systems Biases in web search “Search Engine Manipulation Effect”1,2 Yes! Yes! Yes! Yes! Yes! No! No! How can we measure viewpoint diversity in search results?
  3. 3. 3 WIS Web Information Systems Ranking fairness metrics Rank Candidate gender 1 m 2 f 3 m 4 m 5 m … …
  4. 4. 4 WIS Web Information Systems Our paper RQ: Can ranking fairness metrics be used to assess viewpoint diversity in search results? What we did: • Defined two notions of viewpoint diversity • Conducted two simulation studies to 1. evaluate existing metrics 2. evaluate novel metric that we propose
  5. 5. 5 WIS Web Information Systems Representing viewpoints Should we all be vegan? Extremely opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Extremely supporting -3 -2 -1 0 +1 +2 +3
  6. 6. 6 WIS Web Information Systems Representing viewpoints Should we all be vegan? Strongly opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Strongly supporting protected non-protected Binomial viewpoint fairness -3 -2 -1 0 +1 +2 +3
  7. 7. 7 WIS Web Information Systems Representing viewpoints Should we all be vegan? Strongly opposing Opposing Somewhat opposing Neutral Somewhat supporting Supporting Strongly supporting protected Multinomial viewpoint fairness -3 -2 -1 0 +1 +2 +3
  8. 8. 8 WIS Web Information Systems Simulation studies • Three synthetic data sets S1, S2, S3 • Per set created rankings to simulate different levels of viewpoint diversity (ranking bias) • Computed metrics on each simulated ranking
  9. 9. 9 WIS Web Information Systems Results Considerations: – What is the underlying aim? – How balanced is the data overall? – How strong is the ranking bias? – What is the direction of ranking bias?0.0 0.2 0.4 0.6 0.8 1.0 −1.0 −0.8 −0.6 −0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0 Ranking bias MeannDDvalue
  10. 10. 10 WIS Web Information Systems Take home • Ranking fairness metrics can be used for assessing viewpoint diversity in search results – (when interpreted correctly) • Future work – Appropriate viewpoint labels? – Appropriate level of viewpoint diversity? – Assess viewpoint diversity in real search results – Align different metric and behavioral outcomes t.a.draws@tudelft.nl https://timdraws.net
  11. 11. 11 WIS Web Information Systems References [1] R. Epstein and R. E. Robertson. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences of the United States of America, 112(33):E4512–E4521, 2015. [2] F. A. Pogacar, A. Ghenai, M. D. Smucker, and C. L. Clarke. The positive and negative influence of search results on people’s decisions about the efficacy of medical treatments. ICTIR 2017 - Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval, pages 209– 216, 2017.

×