Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Opinion Driven Decision Support System

Summary of Kavita Ganesan's PhD Thesis

  • Be the first to comment

Opinion Driven Decision Support System

  1. 1. Opinion-Driven Decision SupportSystem
  2. 2. Visiting a new city….Online Opinions  Which hotel to stay at?2
  3. 3. Visiting a new city….Online Opinions  What attractions to visit?Without opinions, decision making becomesdifficult!3
  4. 4. ODSS Components1. DataComprehensive set of opinions to support search and analysiscapabilities4. PresentationPutting it all altogether- easy way for users to explore results of searchand analysis components (ex. organizing and summarizing results)3. Search CapabilitiesAbility to find entities usingexisting opinionsFocus of ExistingWorkopinion summarizationstructured summaries1. Sentiment Summary(ex. +ve/-ve on a piece of text)2. Fine-grained Sentiment Summ.(ex. Battery life: 2 stars; Audio: 1 star)2. Analysis ToolsTools to help digest opinions(ex. Summaries, Opinion trendvisualization)Not a complete solution to supportdecision making based on opinions !4
  5. 5. ODSS Components1. DataComprehensive set of opinions to support search and analysiscapabilities4. PresentationPutting it all altogether- easy way for users to explore results of searchand analysis components (ex. organizing and summarizing results)3. Search CapabilitiesAbility to find entities usingexisting opinionsFocus of ExistingWorkopinion summarizationstructured summaries1. Sentiment Summary(ex. +ve/-ve on a piece of text)2. Fine-grained Sentiment Summ.(ex. Battery life: 2 stars; Audio: 1 star)2. Analysis ToolsTools to help digest opinions(ex. Summaries, Opinion trendvisualization)Need to address broader set of problemsto enable opinion driven decision support5
  6. 6.  We need data: large number of onlineopinions Allow users to get complete and unbiased picture▪ Opinions are very subjective and can vary a lot Currently: No study on how to systematicallycollect opinions from the web
  7. 7.  We need different analysis tools To help users analyze & digest opinions▪ Sentiment trend visualization▪ fluctuation over time▪ Aspect level sentiment summaries▪ Textual summaries, etc… Currently: focus on structured summarization
  8. 8.  We need to incorporate search Allow users find different items or entities basedon existing opinions This can improve user productivity  cuts downon the time spent on reading large numberopinions
  9. 9.  We also need to know how to organize &present opinions at hand effectively Aspect level summaries:▪ How to organize these summaries?▪ Scores or Visuals (stars)?▪ Do you show supporting phrases? Full opinions:▪ How to allow effective browsing of reviews/opinions? don’t overwhelm users
  10. 10. ODSS Components1. DataComprehensive set of opinions to support opinion basedsearch & analysis tasks3. Search CapabilitiesFind items/entities based onexisting opinions(ex. show “clean” hotels only)4. PresentationOrganizing opinions to support effective decision making2. Analysis ToolsTools to help analyze & digestopinions (ex. Summaries, Opiniontrend visualization)10
  11. 11. 1. Should be general Works across different domains & possibly contenttype2. Should be practical & lightweight Can be integrated into existing applications Can potentially scale up to large amounts of data11
  12. 12. Ganesan & Zhai 2012 (Information Retrieval)12
  13. 13.  Currently: No direct way of finding entitiesbased on online opinions Need to read opinions about different entitiesto find entities that fulfill personal criteria13Time consuming & impairs userproductivity!
  14. 14.  Use existing opinions to rank entities based ona set of unstructured user preferences Finding a hotel: “clean rooms, good service” Finding a restaurant: “authentic food, good ambience”14
  15. 15.  Use results of existing opinion mining methods Find sentiment ratings on different aspects Rank entities based on discovered aspect ratings Problem: Not practical! Costly - mine large amounts of textual content Need prior knowledge on set of queriable aspects Most existing methods rely on supervision▪ E.g. Overall user rating15
  16. 16.  Use existing text retrieval models for rankingentities based on preferences: Can scale up to large amounts of textual content Can be tweaked Do not require costly IE or text mining16
  17. 17.  Investigate use of text retrieval models for Opinion-Based Entity Ranking Compare 3 state-of-the-art retrieval models:BM25, PL2, DirichletLM – shown to work best for TR tasks Which one works best for this ranking task? Explore some extensions over existing IR models Can ranking improve with these extensions? Compile the first test set & propose evaluationmethod for this new ranking task17
  18. 18. 18
  19. 19.  Standard retrieval  cannot distinguish multiplepreferences in queryE.g. Query: “clean rooms, cheap, good service” Treated as long keyword query but actually 3 preferences Problem: An entity may score highly because of matchingone aspect extremely well To address this problem: Score each preference separately – multiple queries Combine the results of each query – different strategies▪ Score combination  works best▪ Average rank▪ Min rank▪ Max rank19
  20. 20.  In standard retrieval: Matching an opinionword & standard topic word is notdistinguished Opinion-Based Entity Ranking: Important to match opinion words in the query▪ opinion words have more variation than topic words▪ E.g. Great: excellent, good, fantastic, terrific… Intuition:▪ Expand a query with similar opinion words▪ Help emphasize matching of opinions20
  21. 21. 0.0%2.0%4.0%6.0%8.0%PL2 LM BM25QAM QAM + OpinExp0.0%0.5%1.0%1.5%2.0%2.5%PL2 LM BM25QAM QAM + OpinExpHotels CarsImprovement using QAMImprovement using QAM + OpinExp21
  22. 22. 0.0%2.0%4.0%6.0%8.0%PL2 LM BM25QAM QAM + OpinExp0.0%0.5%1.0%1.5%2.0%2.5%PL2 LM BM25QAM QAM + OpinExpHotels CarsImprovement using QAMImprovement using QAM + OpinExpQAM: Any modelcan be usedQAM: Any modelcan be used22
  23. 23. 0.0%2.0%4.0%6.0%8.0%PL2 LM BM25QAM QAM + OpinExp0.0%0.5%1.0%1.5%2.0%2.5%PL2 LM BM25QAM QAM + OpinExpHotels CarsImprovement using QAMImprovement using QAM + OpinExpQAM+OpinExp: BM25most effectiveQAM+OpinExp: BM25most effective23
  24. 24. 24
  25. 25. Current methods: Focus ongenerating structuredsummaries of opinions[Lu et al., 2009; Lerman et al., 2009;..]Opinion Summary for iPod
  26. 26. We need supporting textualsummaries!To know more: read manyredundant sentencesOpinion Summary for iPod
  27. 27.  Summarize the major opinions What are the major complaints/praise in the text? Concise◦ Easily digestible◦ Viewable on smaller screen Readable◦ Easily understood27
  28. 28.  Widely studied for years[Radev et al.2000; Erkan & Radev, 2004; Mihalcea & Tarau, 2004…] Not suitable for generating concise summaries Bias: with limit on summary size▪ Selected sentences may have missed critical info. Verbose: Not shortening sentencesWe need more of an abstractive approach
  29. 29. 2 Abstractive SummarizationMethodsOpinosis-Graph based summarization framework-Relies on structural redundancies in sentencesWebNgram-Optimization framework based on readability& representativeness scoring-Phrases generated by combining words inoriginal text29
  30. 30. InputSet of sentences:Topic specificPOS annotated30
  31. 31. mythe iphone is aphone calls frequentlytoowith.dropStep 1: Generategraph representation oftext (Opinosis-Graph)greatdeviceInputSet of sentences:Topic specificPOS annotated31
  32. 32. Step 2: Find promising paths(candidate summaries) &score the candidatesmythe iphone is aphone calls frequentlytoowith.dropStep 1: Generategraph representation oftext (Opinosis-Graph)greatdeviceInputSet of sentences:Topic specificPOS annotatedcalls frequentlydropgreat devicecandidate sum1candidate sum23.22.532
  33. 33. The iPhone is a greatdevice, but calls dropfrequently.Step 3: Select top scoringcandidates as final summarycalls frequentlydropgreat deviceStep 2: Find promising paths(candidate summaries) &score the candidatescandidate sum1candidate sum23.22.5mythe iphone is aphone calls frequentlytoowith.dropStep 1: Generategraph representation oftext (Opinosis-Graph)greatdeviceInputSet of sentences:Topic specificPOS annotated33
  34. 34. Assume: 2 sentences about “call quality of iphone”1. My phone calls drop frequently with the iPhone.2. Great device, but the calls drop too frequently.34
  35. 35. • One node for each unique word + POS combination• Sid and Pid maintained at each node• Edges indicate relationship between words in sentence 35great2:1device2:2,2:3but2:4.1:9, 2:10my1:1phone1:2drop1:4, 2:7frequently1:5, 2:9 with1:6the1:7, 2:5iphone1:8calls1:3, 2:6too2:8
  36. 36. great2:1device2:2,2:3but2:4.1:9, 2:10my1:1phone1:2drop1:4, 2:7frequently1:5, 2:9 with1:6the1:7, 2:5iphone1:8calls1:3, 2:6too2:8
  37. 37. great2:1device2:2,2:3but2:4.1:9, 2:10my1:1phone1:2drop1:4, 2:7frequently1:5, 2:9 with1:6the1:7, 2:5iphone1:8calls1:3, 2:6too2:8drop1:4, 2:7frequently1:5, 2:9calls1:3, 2:6Path shared by 2 sentences naturallycaptured by nodes37
  38. 38. great2:1device2:2,2:3but2:4.1:9, 2:10my1:1phone1:2drop1:4, 2:7frequently1:5, 2:9 with1:6the1:7, 2:5iphone1:8calls1:3, 2:6too2:8drop1:4, 2:7frequently1:5, 2:9calls1:3, 2:6Easily discover redundancies for highconfidence summaries38
  39. 39. great2:1device2:2,2:3but2:4.1:9, 2:10my1:1phone1:2drop1:4, 2:7frequently1:5, 2:9 with1:6the1:7, 2:5iphone1:8calls1:3, 2:6too2:8drop1:4, 2:7frequently1:5, 2:9calls1:3, 2:6Gap between words = 239
  40. 40. great2:1device2:2,2:3but2:4.1:9, 2:10my1:1phone1:2drop1:4, 2:7frequently1:5, 2:9 with1:6the1:7, 2:5iphone1:8calls1:3, 2:6too2:8drop1:4, 2:7frequently1:5, 2:9calls1:3, 2:6Gapped subsequences allow:• redundancy enforcements• discovery of new sentences40
  41. 41.  Calls drop frequently with the iPhone Calls drop frequently with the Black Berrydrop frequently with the iphonecallsblack berryOne common highredundancy pathHigh fan-out“calls drop frequently with the iphone andblack berry”41
  42. 42.  Input: Topic specific sentences from user reviews Evaluation Measure: Automatic ROUGE evaluation42
  43. 43. 0.31840.28310.49320.12930.08510.2316HUMAN(17 words)OPINOSISbest(15 words)MEAD(75 words)ROUGE-1 ROUGE-SU4ROUGE Recall0.34340.44820.09160.30880.32710.1515HUMAN(17 words)OPINOSISbest(15 words)MEAD(75 words)ROUGE PrecisionLowest precisionMuch longersentencesHighest recallMEAD does not do well in generatingconcise summaries.43
  44. 44. 0.31840.28310.49320.12930.08510.2316HUMAN(17 words)OPINOSISbest(15 words)MEAD(75 words)ROUGE-1 ROUGE-SU4ROUGE Recall0.34340.44820.09160.30880.32710.1515HUMAN(17 words)OPINOSISbest(15 words)MEAD(75 words)ROUGE Precisionsimilar similarPerformance of Opinosis is reasonable similar to human performance44
  45. 45.  Use existing words in original text to generatemicropinion summaries- set of short phrases Emphasis on 3 aspects: Compactness - use as few words as possible Representativeness – reflect major opinions in text Readability – fairly well formed45
  46. 46. kmmsim)(mS)(mSm)(mS)(mS...mmMjisimjireadireadrepirepsskiikiireadirepki,1(subject tomaxarg,),1146
  47. 47. kmmsim)(mS)(mSm)(mS)(mS...mmMjisimjireadireadrepirepsskiikiireadirepki,1(subject tomaxarg,),11Objective function: Optimizerepresentativeness & readabilityscores• Ensure: summaries reflect key opinions &reasonably well formed47
  48. 48. kmmsim)(mS)(mSm)(mS)(mS...mmMjisimjireadireadrepirepsskiikiireadirepki,1(subject tomaxarg,),11Readability score of miRepresentativeness score of mi48
  49. 49. kmmsim)(mS)(mSm)(mS)(mS...mmMjisimjireadireadrepirepsskiikiireadirepki,1(subject tomaxarg,),11Constraint 1: Maximumlength of summary.•User adjustable•Captures compactness.49
  50. 50. kmmsim)(mS)(mSm)(mS)(mS...mmMjisimjireadireadrepirepsskiikiireadirepki,1(subject tomaxarg,),11Constraint 2 &3: Minrepresentativeness & readability.•Helps improve efficiency•Does not affect performance50
  51. 51. kmmsim)(mS)(mSm)(mS)(mS...mmMjisimjireadireadrepirepsskiikiireadirepki,1(subject tomaxarg,),11Constraint 4: Maxsimilarity of phrases• User adjustable• Captures compactness byminimizing redundancies51
  52. 52.  Measure used: Standard Jaccard Similarity Measure Why important? Allows user to control amount of redundancy E.g. User desires good coverage of information onsmall device  request less redundancies !52
  53. 53.  Purpose: Measure how well a phrase representsopinions from the original text? 2 properties of a highly representative phrase:1. Words should be strongly associated in text2. Words should be sufficiently frequent in text Captured by a modified pointwise mutualinformation (PMI) function53)()(),(),(log)( 2,jijijijiwpwpwwcwwpwwpmiAdd frequency ofoccurrence withina window
  54. 54.  Purpose: Measure well-formedness of a phrase Readability scoring: Use Microsofts Web N-gram model (publicly available) Obtain conditional probabilities of phrases Intuition: A readable phrase would occur morefrequently according to the web than a non-readablephrase)|(log1)( 1...12... kqknqkknkread wwwpKwwS54chain rule to computejoint probability in terms ofconditional probabilities(averaged)
  55. 55.  Input: User reviews for 330 products (CNET) Evaluation Measure: Automatic ROUGE evaluation55
  56. 56. 0.000.010.020.030.040.050.060.070.080.095 10 15 20 25 30ROUGE-2RECALLSummary Size (max words)KEATfidfOpinosisWebNGramWebNgram: Performsthe best for this taskKEA: slightlybetter than tfidfTfidf: Worstperformance56
  57. 57. 0.000.010.020.030.040.050.060.070.080.095 10 15 20 25 30ROUGE-2RECALLSummary Size (max words)KEATfidfOpinosisWebNGramWebNgram: Performsthe best for this taskKEA: slightlybetter than tfidfTfidf: Worstperformance57PROSCONSFULL REVIEW
  58. 58. To Submit.58
  59. 59.  No easy way to obtain a comprehensive set ofopinions about an entity Where to get opinions now? Rely on content providers or crawl a few sourcesProblem :▪ Can result in source specific bias▪ Data sparseness for some entities59
  60. 60. 60 Automatically crawl online reviews forarbitrary entitiesE.g. Cars, Restaurants, Doctors Target online reviews  represent a bigportion of online opinions
  61. 61.  Meant to collect pages relevant to a topicE.g. “Databases Systems”, “Boston Terror Attack” Page type is not as important content news article, review pages, forum page, etc. Most focused crawlers are supervised require large amounts of training data for each topic Not suitable for review collection on arbitraryentities Need training data for each entity  will not scale up tolarge # of entities61
  62. 62.  Focused crawler for collecting reviews pageson arbitrary entities Unsupervised approach Does not require large amounts of training data Solves crawling problem efficiently Uses a special data structure for relevance scoring
  63. 63. Set of entities in a domain(e.g. All hotels in a city)Step 1: For each entity, obtaininitial set of Candidate ReviewPages (CRP).Find Initial CandidateReview Pages (CRP)Input1. Hampton Inn Champaign…2. I Hotel Conference Center…3. La Quinta Inn Champaign…4. Drury Inn5. ….Hampton Inn…Reviews63
  64. 64. Step 3: Score CRPs:• Entity relevance (Sent)• Review pg. relevance (Srev)Select: Srev > σrev ; Sent > σentExpand CRP Listtripadvisor.com/Hotel_Review-g36806-d903...tripadvisor.com/Hotels-g36806-Urbana_Cha...hamptoninn3.hilton.com/en/hotels/…tripadvisor.com/ShowUserReviews-g36806-.........…tripadvisor.com/Hotel_Review-g35790-d102…tripadvisor.com/Hotels-g36806-Urbana_Cha...hamptoninn3.hilton.com/en/hotels/…tripadvisor.com/ShowUserReviews-g36806-.........…Step 2: Expand list of CRPs byexploring links in neighborhoodof initial CRPs.Collect Relevant Review Pages64
  65. 65.  Use any general web search (e.g. Bing/Google) Per entity basis Search engines do partial matching of entities topages More likely pages in vicinity of search resultsrelated to entity QueryEntity QueryFormat: “entity name + brand / address” + “reviews”E.g. “Hampton Inn Champaign 1200 W University Ave Reviews”65
  66. 66.  Follow top-N URLs around vicinity of searchresults Use URL prioritization strategy: Bias crawl path towards entity related pages Score each URL: based on similarity between(a) URL + Entity Query, Sim(URL,EQ)(b) Anchor + Entity Query, Sim(Anchor,EQ)66
  67. 67.  To determine if page is indeed a review page Use review vocabulary: Lexicon with most commonly occurring wordswithin review pages – details in thesis Idea: score a page based on # of review page words67]10[)(,)()()(),(log)( 2pirevSnormalizerpiSpirevSVt twtiptcpirevSrawrevraw
  68. 68. ]10[)(,)()()(),(log)( 2pirevSnormalizerpiSpirevSVt twtiptcpirevSrawrevraw To determine if page is indeed a review page Use review vocabulary: Lexicon with most commonly occurring wordswithin review pages – details in thesis Idea: score a page based on # of review page wordsRaw review pagerelevance scoreNormalize to obtainfinal review pagerelevance score68t is a term in thereview vocabulary, Vc(t, pi) – freq. of t in page pi (tf).wt(t) - importanceweighting of t in RVNormalizer needed toset proper thresholds
  69. 69.  Explored 3 normalization options: SiteMax (SM) : Max Srevraw(pi) amongst all pagesrelated to a particular site - Normalize based on sitedensity EntityMax (EM) : Max Srevraw(pi) score amongst allpages related to an entity - Normalize based onentity popularity EntityMax + GlobalMax (GM) orSiteMax + GlobalMax (GM) :▪ To help with cases where SM/EM are unreliable69
  70. 70.  To determine if page is about target entity Based on similarity between a page URL & EntityQuery Why it works? Most review pages have highly descriptive URLs Entity Query is a detailed description of entity The more URL resembles query, more likely it isrelevant to target entity Similarity measure: Jaccard Similarity70
  71. 71.  Steps proposed so far, can be implemented in avariety of different ways Our goal: make the crawling framework usable inpractice71
  72. 72. 1. Efficiency: Allow review collection for large number of entities Task should terminate in reasonable time & accuracy Problem happens when cannot access requiredinformation quickly▪ E.g. Repeated access to term frequencies of different pages2. Rich Information Access (RIA): Allow client to access info. beyond crawled pagesE.g. Get all review pages from top 10 popular sites for entity X DB not suitable because you cannot naturally modelcomplex relationships and would yield in large joins72
  73. 73.  Heterogeneous graph data structure Models complex relationships betweendifferent components in a data collectionproblem73
  74. 74. Review VocabularyCurrent QueryQVt1t2t3t4t5tz....Term NodeswtwtwtwtwtwtwtwtwtwtwtE1Entity NodesE2EkHampton Inn ChampaignI-Hotel Conference CenterDrury inn ChampaigntttuuucccPage NodesP2P1P3P4P5P6PnSite NodesS2Sthotels.comlocal.yahoo.comS1tripadvisor.comt = title, u = url, c = contentLogical NodesOtherLogical Nodes74
  75. 75. Review VocabularyCurrent QueryQVOtherLogical Nodest1t2t3t4t5tz....Term NodeswtwtwtwtwtwtwtwtwtwtwtE1Entity NodesE2EkHampton Inn ChampaignI-Hotel Conference CenterDrury inn ChampaignuuucccPage NodesP2P1P3P4P5P6PnSite NodesS2Sthotels.comlocal.yahoo.comS1tripadvisor.comtttt = title, u = url, c = contentLogical Nodes75List of entities on whichreviews are requiredBased on set of CRPsfound for each entityAt thecore, made up oftermsOne nodeper uniqueterm
  76. 76.  Maintain one simple data structure: Access to various statistics▪ E.g TF of word in a page  EdgeWT(content node  term node) Access to complex relationships and global information Compact: can be an in memory data structure Network can be persisted and accessed later Client applications can use network to answerinteresting app. related questionsE.g. Get all review pages for entity X from top 10 popular sites76
  77. 77. t1t2t3t4t5tz....Term NodesPage NodesP2P1P3P4P5P6PnwtwtwtwtVCContent Node(logical node)tftftfReview Vocabulary Node(logical node)To compute Srevraw(pi) :-Terms present in both the Content node and RV node.-TF and weights can be obtained from edges-Lookup of review vocabulary words within a page is fast-No need to parse page contents each time encountered77Outgoing edges = term ownershipEdge weight = importance wtEdge weight = TF
  78. 78. Opinion VocabularyCurrent QueryQOOtherLogical Nodest1t2t3t4t5tz....Term NodeswtwtwtwtwtwtwtwtwtwtwtE1Entity NodeE2EkHampton Inn ChampaignI-Hotel Conference CenterDrury inn ChampaignuuucccPage NodesP2P1P3P4P5P6PnSite NodesS2Sthotels.comlocal.yahoo.comS1tripadvisor.comtttLogical NodesAccess all pages connectedto the site noderequires complete graph78
  79. 79. Opinion VocabularyCurrent QueryQOOtherLogical Nodest1t2t3t4t5tz....Term NodeswtwtwtwtwtwtwtwtwtwtwtE1Entity NodeE2EkHampton Inn ChampaignI-Hotel Conference CenterDrury inn ChampaignuuucccPage NodesP2P1P3P4P5P6PnSite NodesS2Sthotels.comlocal.yahoo.comtripadvisor.comS1tttLogical NodesAccess all pages connectedto entity noderequires complete graph79
  80. 80. t1t2t3t4t5tz....Term NodestfPage NodesP2P1P3P4P5P6Pntftftftfq1Entity Query Node(logical node)Hampton Inn Champaign 1200W Univ…Reviewstftftripadvisor.com/ShowUser…UURL Node(logical node)80
  81. 81.  Goal: Evaluate accuracy & give insights into efficiencyusing FetchGraph Evaluated in 3 domains: (5) – Electronics, (5) – Hotels, (4) - Attractions Only 14 entities  expensive to obtain judgments Gold standard: For each entity, explore top 50 Google results & linksaround vicinity of the results (up to depth 3) 3 Human judges used to determine relevance ofcollected links to entity query (crowd sourcing) Final judgment: majority voting81
  82. 82.  Baseline: Google search results Deemed relevant to entity query Evaluation measure: Precision Recall – estimate of coverage of review pages82)Pages(eGoldStdRel#)RelPages(e#)Recall(ekkk)ages(eRetrievedP#)RelPages(e#)Prec(ekkk
  83. 83. 83
  84. 84. 0.000.050.100.150.200.2510 20 30 40 50RecallNumber of search resultsGoogle OpinoFetch OpinoFetchUnnormalizedOpinoFetchOpinoFetchUnnormalizedGoogleGoogle: recallconsistently lowGoogle: recallconsistently lowGoogle: recallconsistently lowGoogle: recallconsistently low84Search results  not always relevant to EQ or notdirect pointers to actual review pages.
  85. 85. 0.000.050.100.150.200.2510 20 30 40 50RecallNumber of search resultsGoogle OpinoFetch OpinoFetchUnnormalizedOpinoFetchOpinoFetchUnnormalizedGoogleOpinoFetch: recallkeeps improvingOpinoFetch: recallkeeps improvingOpinoFetch: recallkeeps improvingOpinoFetch: recallkeeps improving85-A lot of relevant content in vicinity of search results-OpinoFetch is able to discover such relevant content
  86. 86. 0.000.050.100.150.200.2510 20 30 40 50RecallNumber of search resultsGoogle OpinoFetch OpinoFetchUnnormalizedOpinoFetchOpinoFetchUnnormalizedGoogleOpinoFetch: betterrecall with normalization-Scores are normalized using special normalizers(e.g. EntityMax / SiteMax)-Easier to distinguish relevant review pages86
  87. 87. 97.23%85.72%36.23%19.62%0%20%40%60%80%100%EntityMax +GlobalMaxEntityMax SiteMax +GlobalMaxSiteMax%Changeinprecision EM + GM: gives thebest precisionSM: gives lowestprecision87SM is worst performing: certain sites coverdifferent classes of entities. Max score from thesite may be unreliable for sparse entities
  88. 88. 0500001000001500002000002500003000003500004000004500000 200 400 600 800 1000GraphSize# pages crawledLinear growth without anyoptimization/compressionPossible to use FetchGraph as inmemory data structure88
  89. 89. Avg. Execution Time with/without FetchGraphWithFetchGraphWithoutFetchGraphSrevraw(pi) 0.09ms 8.60msEnityMaxNormalizer0.06ms 4.40 sWithout FetchGraph:-Parse page contents each timeWith FetchGraph:-Page loaded into memory once-Use FetchGraph to compute Srevraw(pi)89
  90. 90. Avg. Execution Time with/without FetchGraphWithFetchGraphWithoutFetchGraphSrevraw(pi) ~0.09ms ~8.60msEnityMaxNormalizer~0.06ms ~4.40sWithout FetchGraph:load sets of pages into memoryto find entity max normalizerWith FetchGraph:-Global info tracked till the end-Only need to do a lookup on related setsof pages to obtain entity max normalizer90
  91. 91.  Proposed: An unsupervised, practical methodfor collecting reviews on arbitrary entities Works with reasonable accuracy withoutrequiring large amounts of training data Proposed FetchGraph: Helps with efficient lookup of various statistics Useful for answering application related queries91
  92. 92. Ganesan & Zhai, WWW 201292
  93. 93.  Finds & ranks entities based on user preferences Unstructured opinion preferences - novel Structured preferences - e.g. price, brand, etc. Beyond search: Support for analysis of entities Ability to generate textual summaries of reviews Ability to display tag clouds of reviews Current version: Works in the hotels domain93
  94. 94. Search: Find entities basedon unstructured opinionpreferencesSearch: + Combine withstructured preferencesRanking: How well allpreferences arematched?94
  95. 95. Tag cloudsweighted by frequencyRelated snippets(“convenient location”)95
  96. 96. Opinion summariesreadable, well-formedRelated snippets96
  97. 97. Summary with Initial Reviews:-26 reviews in total-1-2 sourcesSummary with OpinoFetch Reviews:-135 reviews (8 sources)-Extracted with a baseline extractor-Not all reviews were included – filter• Based on length of review• Subjectivity score of review 97
  98. 98.  Opinion Based Entity Ranking Use click through & query logs to further improveranking of entities▪ Now possible  everything is logged by demo system Look into the use of phrasal search for ranking▪ Limit deviation from actual query (e.g. “close to university”)▪ Explore: “back-off” style scoring – score based on phrasethen remove the phrase restriction98
  99. 99.  Opinosis How to scale up to very large amounts of text?▪ Explore use of map reduce framework Would this approach work with other types of texts?▪ E.g. Tweets, Facebook comments – shorter texts Opinion Acquisition Compare OpinoFetch with a supervised crawler▪ Can achieve comparable results? How to improve recall of OpinoFetch?▪ To evaluate at a reasonable scale: approximate judgmentswithout relying on humans?99
  100. 100. [Barzilay and Lee2003] Barzilay, Regina and Lillian Lee. 2003. Learning to paraphrase: an unsupervisedapproach using multiple-sequence alignment. In NAACL ’03: Proceedings of the 2003 Conference of theNorth American Chapter of the Association for Computational Linguistics on Human LanguageTechnology, pages 16–23, Morristown, NJ, USA.[DeJong1982] DeJong, Gerald F. 1982. An overview of the FRUMP system. In Lehnert, Wendy G. and Martin H.Ringle, editors, Strategies for Natural Language Processing, pages 149–176. LawrenceErlbaum, Hillsdale, NJ.[Erkan and Radev2004] Erkan, G¨unes and Dragomir R. Radev. 2004. Lexrank: graph-based lexical centrality assalience in text summarization. J. Artif. Int. Res.,22(1):457–479.[Finley and Harabagiu2002] Finley, Sanda Harabagiu and Sanda M. Harabagiu. 2002. Generating single andmulti-document summaries with gistexter. In Proceedings of the workshop on automaticsummarization, pages 30–38.[Hu and Liu2004] Hu, Minqing and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD, pages168–177.[Jing and McKeown2000] Jing, Hongyan and Kathleen R. McKeown. 2000. Cut and paste based textsummarization. In Proceedings of the 1st North American chapter of the Association for ComputationalLinguistics conference, pages 178–185, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.[Lerman et al.2009] Lerman, Kevin, Sasha Blair-Goldensohn, and Ryan Mcdonald. 2009. Sentimentsummarization: Evaluating and learning user preferences. In 12th Conference of the European Chapter ofthe Association for Computational Linguistics (EACL-09).[Mihalcea and Tarau2004] Mihalcea, R. and P. Tarau. 2004. TextRank: Bringing order into texts. In Proceedingsof EMNLP-04and the 2004 Conference on Empirical Methods in Natural Language Processing, July.[Pang and Lee2004] Pang, Bo and Lillian Lee. 2004. A sentimental education: Sentiment analysis usingsubjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271–278.[Pang et al.2002] Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentimentclassification using machine learning techniques. In Proceedings of the 2002 Conference on EmpiricalMethods in Natural Language Processing (EMNLP), pages 79–86.[Radev and McKeown1998] Radev, DR and K. McKeown. 1998. Generating natural language summaries frommultiple on-line sources. Computational Linguistics, 24(3):469–500.[More in Thesis Report] 100

×