Deep Semantic Similarity Model (DSSM) is a deep learning technique that represents text strings in a continuous semantic space to model semantic similarity. DSSM maps queries and documents to feature vectors using a neural network and computes cosine similarity between the vectors to rank documents. DSSM is trained using word hashing, convolutional and max-pooling layers, and negative sampling to maximize the likelihood of clicked documents. Evaluation on search engine queries showed DSSM improved ranking quality over baselines, and it has also been applied to recommendations by modeling similarities across domains.
3. Introduction
DSSM: stands for Deep Structured Semantic Model or
Deep Semantic Similarity Model
Is a deep neural network modelling technique for
representing text strings in a continuous semantic space
and modelling semantic similarity between two text
strings.
web search ranking question answering
knowledge inference image captioning
machine translation recommendation
5. Motivation
Fuzzy keyword matching
q: cold home remedy
Spelling correction
q: cold remeedies
Query alteration/expansion
q: flu treatment
Query/document semantic matching
q: how to deal with stuffy nose
best home
remedies for
cold and flu
6. Structure of DSSM
• Compute semantic similarity between two text strings X and Y
• Map X and Y to feature vectors in a latent semantic space via deep
neural net
• Compute the cosine similarity between the feature vectors
7. Structure of DSSM
1. Get the semantic representation of two vector
2. Normalize the two semantic vectors
3. Compute their similarity
4. Use semantic similarity to
rank documents
Semantic representation
9. Structure of DSSM
Semantic Relevance Score between a query Q and a document D
x: input
y: output
l: hidden layer
f: activation function
10. Structure of DSSM
Supervised Model: We assume that a query is relevant to the documents
that are clicked on for that query.
The posterior probability of a document given a query:
Where γ is the smoothing factor in the SoftMax function.
D denotes the set of candidate documents to be ranked.
11. Structure of DSSM
Maximize the likelihood of the clicked documents P(D|Q)
Equivalently, we need to minimize the following loss function
The model is trained using gradient-based numerical optimization
algorithms
12. Training Techniques
Word Hashing: use sub-word unit (e.g., letter n-gram) as raw input to
handle very large vocabulary,
Letter-trigram Representation
cat → #cat# → #-c-a, c-a-t, a-t-#
Only around 50K letter-trigrams in English
Advantages
• Capture sub-word semantics
• Control the dimensionality of the input space
• Words with small typos have similar raw representations
13. Training Techniques
Convolutional and Max-pooling layer: identify key words or concepts
Extract local features
using convolutional layer
Generate global features
using max-pooling
14. Training Techniques
Negative Sampling
Where γ is the smoothing factor in the SoftMax function.
D denotes the set of candidate documents to be ranked.
Ideally, D should contain all possible documents.
In practice, we usually approximate D by including clicked document set D+
and some randomly selected documents
posterior
probability
16. Evaluation Results
NDCG: Normalized Discounted Cumulative Gain
A measure of ranking quality
Measures the usefulness of a document based on its position in the result list
The evaluation data set contains
16510 English queries sampled
from one-year query log files of
a commercial search engine.
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation
Named Entity Mining from Click-Through Data Using Weakly Supervised Latent Dirichlet Allocation
Topic Regression Multi-Modal Latent Dirichlet Allocation for Image Annotation