Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems

Web Science & Technologies
University of Koblenz ▪ Landau, Germany

Measuring the Influence of Tag Recommenders
on the Indexing Quality in Tagging Systems

Klaas Dellschaft
klaasd@uni-koblenz.de

Steffen Staab
staab@uni-koblenz.de

Collaborative Tagging Systems

 Objectives of tag recommenders:
 Improve indexing quality ⇒ retrieval results
 Reduce tagging effort
Measuring the Influence of Tag Recommenders Slide 2 of 21
Klaas Dellschaft (klaasd@uni-koblenz.de) http://west.uni-koblenz.de

Outline

 Measures of indexing quality
 What to understand under “indexing quality”?
 Inter-resource consistency ⇔ inter-indexer consistency

 Evaluation of the measures
 Are the measures correlated with each other?
 User study: Apply measures for two recommenders

 Evaluation results

 Conclusions


Measures of
Indexing Quality


What does “indexing quality” mean?
user perceived similarity

Resources
r1 r2 r3

describe
 science  4 0 0

Tag Vectors
       
 news  10  8 0
 humor  v1 =   v2 =   v3 =  
0 6 9
       
 patents  0 0 5
       

sim(v1, v2) sim(v2, v3)


Measures of indexing quality

 Inter-resource consistency
 Compare resource similarity to the tag vector distance
 Requires external knowledge about similarity of resources
 Direct but sophisticated measure of indexing quality

 Inter-indexer consistency
 Do users agree on common description for a resource?
 Assumption: Users select tags independent of each other
 Indirect but easy measure of indexing quality

 Which measure to use for evaluating tag recommenders?


Research Hypotheses

 Hypothesis: Inter-indexer consistency does not measure the
influence of tag recommenders on the indexing quality!

 Popular Tags: Suggest most popular tags of a resource
 H1a: Popular Tags increase the inter-indexer consistency
 H1b: Popular Tags decrease the inter-resource consistency

 User Tags: Suggest all tags previously applied by the user
 H2a: User Tags lead to a decreased or unchanged inter-indexer

consistency
 H2b: User Tags increase the inter-resource consistency

 The measures do not correlate when evaluating tag recommenders

Measuring Inter-Resource Consistency
 Idea: Compare resource similarity and tag vector distance
 ai: Average distance to resources in the same cluster
 bi: Average distance to resources in the closest other cluster
bi − ai
si =
max(ai , bi )
resource

cluster of similar resources
inconsistent consistent even more
-1 0 consistent +1

Measuring Inter-Indexer Consistency

Idea: Do users agree on common description for a resource?
Tag Reuse Rate
 Average number of users who apply a tag
 Used in the related work

 news  8 8 8
       
 humor   4 6  6
 fun   2  2  0
       
 patents  0 0  0
       

Tag Reuse Rate: 4.7 5.3 7


Evaluation


Experimental Setup

 Objective:
 Are inter-resource and inter-indexer correlated if tag
recommendations are given?

 Task given to users:
 Assign keywords to 10 web pages.
 After tagging, cluster web pages according to their
similarity (⇒ inter-resource consistency).

 Three different experimental conditions:
 No Suggestions
 User Tags
 Popular Tags
 Further divided into an English and German user group

Suggestion of Popular Tags – Screenshot


Clustering of Similar Web Pages – Screenshot


Results


Sizes of the Tagging Data Set

German User Group:
#Users #Tags #TAS #TAS / #User
No Suggestions 74 706 2134 28.84
Popular Tags 78 531 2228 28.56
User Tags 79 466 1507 19.08

English User Group:
#Users #Tags #TAS #TAS / #User
No Suggestions 115 973 3150 27.39
Popular Tags 118 550 3003 25.45
User Tags 118 819 2919 24.74


The Clustering Data Set

 In average, each user identified 4.59 clusters
 Overall, 146 distinct clusters have been identified
 11 most frequent clusters ⇒ 70% of the data

 The web pages cover ~7 topics
 3 web pages are on the border between two topics

Differences in the Topical Clusters

Cluster probabilities in English experiment

No Suggestions
Popular Tags
User Tags

The Onion + BBC The Onion + Patents
⇒ News ⇒ Humor

 English Popular Tags condition has to be excluded

Measuring the Inter-Resource Consistency

 H1a: Popular Tags decrease the inter-resource consistency
 H2a: User Tags increase the inter-resource consistency

 Expectation: E(spt,i) < E(sns,i) < E(sut,i)

E(spt,i) E(sns,i) E(sut,i)
German Users 0.1474 0.1847 0.2367
English Users N/A 0.1713 0.1915

(All differences are significant!)


Measuring the Inter-Indexer Consistency

 H1b: Popular Tags increase the inter-indexer consistency
 H2b: User Tags lead to a decreased or unchanged
inter-indexer consistency

 Expectation: E(trpt,i) > E(trns,i) ≥ E(trut,i)

E(trpt,i) E(trns,i) E(trut,i)
German Users 3.60 2.44 2.39*
English Users 4.67 2.76 2.68*

* Differences between E(trns,i) and E(trut,i) not significant


Conclusions
 Measures of indexing quality
 Inter-resource consistency
 Inter-indexer consistency
 Measures do not correlate if recommendations are given
 Only inter-resource consistency can be used

 Popular Tags
 Do not lead to consistent descriptions across resources
 Are rather counterproductive for indexing resources

 User Tags
 Lead to consistent descriptions across resource
 Consolidate the personomy of users


Paper:
K. Dellschaft & S. Staab. Measuring the Influence of Tag
Recommenders on the Indexing Quality in Tagging Systems.
Proceedings of the Hypertext Conference, 2012
http://dl.acm.org/citation.cfm?id=2310009

Experimental Interface:
http://userpages.uni-koblenz.de/~klaasd/experiment/

Data Set:
http://west.uni-koblenz.de/Research/DataSets/tagging-experiment/


Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

Similar to Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems

Similar to Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems (20)

Recently uploaded

Recently uploaded (20)

Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems