This document discusses various similarity measures that can be used to quantify the similarity between documents, queries, or a document and query in an information retrieval system. It describes classic measures like Dice coefficient, overlap coefficient, Jaccard coefficient, and cosine coefficient. It provides examples of calculating these measures and compares the relations between different measures. The document also discusses using term-document matrices and shows an example matrix.

1. Chapter 3 Similarity Measures Data Mining Technology

2. Chapter 3 Similarity Measures Written by Kevin E. Heinrich Presented by Zhao Xinyou [email_address] 2007.6.7 Some materials (Examples) are taken from Website.

4. Example

10. Simple Measure Technology Documents Set PP29 Retrieved A Relevant B Retrieved and Relevant A ∩B Precision = Returned Relevant Documents / Total Returned Documents Recall = Returned Relevant Documents / Total Relevant Documents P(A,B) = |A ∩B| / |A| R(A,B) = |A ∩B| / |B|

11. Example--Simple Measure Technology PP29 Documents Set A,C,E,G, H, I, J Relevant B,D,F Retrieved & Relevant W,Y Retrieved |B| = {relevant} ={A,B,C,D,E,F,G,H,I,J} = 10 |A| = {retrieved} = {B, D, F,W,Y} = 5 |A∩B| = {relevant} ∩ {retrieved} ={B,D,F} = 3 P = precision = 3/5 = 60% R = recall = 3/10 = 30%

16. Overlap Coefficient PP30-31 Documents Set A queries B Documents

17. Jaccard Coefficient-Cont’ PP31 Documents Set A queries B Documents

19. Cosine Coefficient-Cont’ PP31-32 (d 21 ,d 22 ,…d 2n ) (d 11 ,d 12 ,…d 1n ) (q 1 ,q 2 ,…q n )

21. Asymmetric PP31 dj di W ki -->w kj

22. Euclidean distance PP32

23. Manhattan block distance PP32

25. Comparison PP34

26. Comparison Simple matching Dice’s Coefficient Cosine Coefficient Overlap Coefficient Jaccard’s Coefficient |A|+|B|-|A ∩ B| ≥(|A|+|B|)/2 |A| ≥ |A ∩ B| |B| ≥ |A∩B| (|A|+|B|)/2 ≥ √ (|A|*|B|) √ (|A|*|B|) ≥ min (|A|, |B|) O≥C≥D≥J PP34

28. Example- Term-Document Matrix -Cont’ Matrix[q][A] PP34 1 1 1 1 1 2 Q 1 1 1 1 1 D7 1 1 1 D6 1 1 1 1 D5 1 1 1 D4 1 1 1 1 D3 1 1 1 D2 1 1 1 D1 T17 T16 T15 T14 13 T12 T11 T10 T9 T8 T7 T6 T5 T4 T3 T2 T1

29. Dice coefficient PP30, PP34

30. Final Results PP34 O≥C≥D≥J