Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Helping Users Discover Perspectives: Enhancing Opinion Mining with Joint Topic Models

7 views

Published on

Paper presentation at the SENTIRE workshop at ICDM 2020. Sorrento, Italy (online).

Published in: Science
  • Be the first to comment

  • Be the first to like this

Helping Users Discover Perspectives: Enhancing Opinion Mining with Joint Topic Models

  1. 1. 1 WIS Web Information Systems Helping users discover perspectives Enhancing opinion mining with joint topic models Tim Draws, Jody Liu, Nava Tintarev TU Delft, The Netherlands t.a.draws@tudelft.nl https://timdraws.net
  2. 2. 2 WIS Web Information Systems Discovering perspectives
  3. 3. 3 WIS Web Information Systems Discovering perspectives Unstructured set of textual opinions ? Perspective 1 Supporting Perspective 2 Perspective 3 Perspective 4 Perspective 5 Perspective 6 Perspective 7 Perspective 8 Perspective 9 Perspective 10 Opposing Structured set of perspectives
  4. 4. 4 WIS Web Information Systems Topic models • Topic model = unsupervised model to discover hidden structures (i.e., topics) in corpora of text – Example: Latent Dirichlet Allocation (LDA) [1] – Topics are probability distributions over words – If applied to a corpus of documents related to a debate, topics could be interpreted as perspectives • Joint topic model = adding additional components (e.g., sentiment analysis) to a classical topic model (e.g., LDA)
  5. 5. 5 WIS Web Information Systems Our paper RQ1. Can joint topic models support users in discovering perspectives in a corpus of opinionated documents? RQ2. Do users interpret the output of joint topic models in line with their personal pre-existing stance? Contributions: 1. Perspective-annotated data set 2. User study
  6. 6. 6 WIS Web Information Systems Data Document Stance Perspective You cannot be a Christian and support abortion… Against Abortion is the killing of a human being, which defies the word of God. No one in the world has any right to judge over what someone else does with their body, … For Reproductive choice empowers women by giving them control over their own bodies. Why put a child through the pain of an unloving mother… For A baby should not come into the world unwanted. … … … Final data set: 600 documents; 6 perspectives
  7. 7. 7 WIS Web Information Systems Experimental setup 1 2 3 4 5 • Ran each model on the final data set (i.e., for 6 topics) • Between-subjects study: each participant sees output of one of the models • Participants need to identify the correct 6 perspectives from the model output
  8. 8. 8 WIS Web Information Systems Procedure Step 1 Step 2 Step 3 Participants state: • Age • Gender • Personal stance towards abortion • Familiarity with the abortion debate Participants state: • Perceived usefulness • Perceived awareness increase • Confidence in task performance
  9. 9. 9 WIS Web Information Systems Results: descriptive • 158 participants (recruited from Prolific) – After excluding 12 participants due to failing both honeypot topics – 150 required according to power analysis • 50.6% female, 49.4% male • 33.3 years old on average (range 18 to 64) • Most (57.8%) at least somewhat familiar with the topic • Sample skewed towards the supporting viewpoint
  10. 10. 10 WIS Web Information Systems Results: hypothesis tests H1: Users find more correct perspectives when being exposed to the output of a joint topic model compared to the output of a regular topic model or baseline. – We find a difference between models (p < 0.001, η2 = 0.126) – TAM is the only one that performs significantly better than the baseline 3 4 5 TF−IDF LDA JST VODUM TAM LAM Model MeannCor
  11. 11. 11 WIS Web Information Systems Results: hypothesis tests H2: Users are more likely to identify sets of keywords as perspectives that are in line with their personal stance compared to perspectives that they do not agree with. – No evidence for for such a relationship (ρ = 0.122, p = 0.163)
  12. 12. 13 WIS Web Information Systems Discussion and future work • Why did TAM perform better? – It extracted more keywords that appeared explicitly in the perspective expression Abortion is the killing of a human being, which defies the word of God. Reproductive choice empowers women by giving them control over their own bodies. A baby should not come into the world unwanted. • Future work: different domains, novel topic models
  13. 13. 14 WIS Web Information Systems Take home • Joint topic models such as TAM can perform perspective discovery • No evidence for tendency of users to interpret output in line with their personal stance • Implications for several areas: journalism, policy-making, generating explanations (All supplementary materials are openly available at https://osf.io/uns63/.)
  14. 14. 15 WIS Web Information Systems References [1] D. Blei, A. Ng, and M. Jordan, “Latent dirichlet allocation,” Journal of Machine Learning Research, vol. 3, pp. 993– 1022, 05 2003. [2] M. Paul and R. Girju, “A two-dimensional topic-aspect model for discovering multi-faceted topics.” in AAAI, vol. 1, 01 2010. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1. 226.3550&rep=rep1&type=pdf [3] C. Lin and Y. He, “Joint sentiment/topic model for sentiment analysis,” in Proceedings of the 18th ACM Conference on Information and Knowledge Management, ser. CIKM ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 375–384. [Online]. Available: https://doi.org/10.1145/1645953.1646003 [4] T. Thonet, G. Cabanac, M. Boughanem, and K. Pinel-Sauvagnat, “Vodum: A topic model unifying viewpoint, topic and opinion discovery,” in ECIR, vol. 9626. Toulouse, France: Springer, 03 2016, pp. 533– 545. [5] D. Vilares and Y. He, “Detecting perspectives in political debates,” in EMNLP. Association for Computational Linguistics, 01 2017, pp. 1573–1582.

×