SlideShare a Scribd company logo
1 of 26
Download to read offline
Metrics for Evaluating Quality of
Embeddings for Ontological Concepts
Faisal Alshargi
University of Leipzig
Saeedeh Shekarpour
University of Dayton
Tommaso Soru
University of Leipzig
Amit Sheth
Kno.e.sis Center
1
Motivation: the growth of Linked Open Data
2
March 2018
~3000 Datasets
>190 Triples
Motivation: Knowledge Transformation
It is necessary to dispose this valuable knowledge for extrinsic tasks such as natural language processing
or data mining.
➔ the knowledge (i.e. schema level and instance level) has to be transformed from discrete
representations to numerical representations (called embeddings).
3
Research Gap and our Contribution
Existing deficiency: is the lack of an evaluation framework for comprehensive and fair judgment on the
quality of the embeddings of concepts
Evaluation can be intrinsic or extrinsic
Our Contribution: concerned with evaluating the quality of embeddings for concepts. It extends the
state of the art by providing several intrinsic metrics for evaluating the quality of the embedding of
concepts on three aspects:
i. the categorization aspect,
ii. the hierarchical aspect, and
iii. the relational aspect.
4
Ontological Concept
Ontological concepts play a crucial role in
➔ capturing the semantics of a particular domain
➔ typing entities which bridge a schema level and
an instance level
➔ determining valid types of sources and
destinations for relations in a knowledge graph.
Thus, the embeddings of the concepts are expected to
truly reflect characteristics of ontological concepts in
the embedding space.
5
Contribution
contributing in providing several metrics for evaluating the quality of the embedding of concepts from
three perspectives:
(i) how the embedding of concepts behaves for categorizing their instantiated entities;
(ii) how the embedding of concepts behaves with respect to hierarchical semantics described in the
underlying ontology; and
(iii) how the embedding of concepts behaves with respect to relations
6
Embedding Models (1)
➔ Word2vec: Skip-Gram Model and Continuous Bag of Words (CBOW) Model. learns two separate
embeddings for each target word wi , (i) the word embedding and (ii) the context embedding.
➔ GloVe Model: The GloVe model (Pennington, Socher, and Manning 2014) is a global log-bilinear
regression model for the unsupervised learning of word embeddings. It captures global statistics of
words in a corpus and benefits the advantages of the other two models: (i) global matrix
factorization and (ii) local context window methods.
➔ RDF2Vec Model: RDF2Vec (Ristoski and Paulheim 2016) is an approach for learning embeddings
of entities in RDF graphs. It initially converts the RDF graphs into a set of sequences using two
strategies: (i) Weisfeiler-Lehman Subtree RDF Graph Kernels, and (ii) graph random walks. Then,
word2vec is employed for learning embeddings over these produced sequences.
7
Embedding Models (2)
➔ Translation-based Models: The TransE and TransH models assume that the embeddings of both
the entities and relations of a knowledge graph are represented in the same semantic space,
whereas the TransR considers two separate embedding spaces for entities and relations. All three
approaches share the same principle, summing the vectors of the subject and the predicate, one
can obtain an approximation of the vectors of the objects.
➔ Other Knowledge Graph Embedding (KGE) Models.
◆ HolE (Holographic Embeddings)
◆ DistMult
◆ ComplEx
◆ ...
All approaches above have shown to reach state-of-the-art performances on link prediction and triplet
classification.
8
Excluding of non-scalable KGE Approaches
➔ We selected the knowledge graph embedding approaches for the evaluation of our metrics among
RDF2Vec, TransE and three of the methods described in the previous subsection (i.e., HolE,
DistMult, and ComplEx)
➔ Differently from RDF2Vec, we could not find DBpedia embeddings pre-trained using any of the
other approaches online, thus we conducted a scalability test on them to verify their ability to
handle the size of DBpediawe decided to select only the more scalable RDF2Vec approach in our
evaluation
9
Data Preparation
➔ Corpus: DBpedia + Wikipedia
➔ From the DBpedia ontology, we selected 12 concepts, which are positioned in various levels of the
hierarchy. Furthermore, for each concept, we retrieved 10,000 entities typed by it (in case of
unavailability, all existing entities were retrieved).
➔ For each concept class, we retrieved 10,000 instances and their respective labels; in case of
unavailability, all existing instances were retrieved.
➔ Then, the embeddings of these concepts as well as their associated instances were computed from
the embedding models: (i) skip-gram, and (ii) CBOW and (iii) GloVe trained on Wikipedia and
DBpedia
10
Task 1: Evaluating the Categorization Aspect of
Concepts in Embeddings
➔ Ontological concepts C categorize entities by typing them, mainly using rdf:type. In other words, all
the entities with a common type share specific characteristics. For example, all the entities with
the type dbo:Country have common characteristics distinguishing them from the entities with the
type dbo:Person.
➔ Research Question: How far is the categorization aspect of concepts captured (i.e., encoded) by an
embedding model? In other words, we aim to measure the quality of the embeddings for concepts
via observing their behaviour in categorization tasks.
11
Categorization metric
➔ In the context of unstructured data, this metric aligns a clustering of words into different
categories
➔ how well the embedding of a concept performs as the background concept of the entities typed by
it.
➔ To quantify this metric, we compute the averaged vector of the embeddings of all the entities
having type c and then compute the cosine similarity of this averaged vector and the embedding of
the concept Vc
12
Experiment
13
Coherence metric
It measures whether or not a group of words adjacent in the embedding space are mutually related.
Commonly, this relatedness task has been evaluated in a subjective manner (i.e. using a human judge).
However, in the context of structured data we define the concept of relatedness as the related entities
which share a background concept, a background concept is the concept from which a given entity is
typed (i.e. inherited). For example, the entities dbr:Berlin and dbr:Sana’a are related because both are
typed by the concept dbo:City
14
Coherence metric
A. Quantitative Evaluation: For the given concept c and the given radius n, we find the n-top similar
entities from the pool (having the highest cosine similarity with Vc ). Then, the coherence metric
for the given concept c with the radius n is computed as the number of entities having the same
background concept as the given concept;
B. Qualitative Evaluation: visualization approach
15
Experiment (radius 20)
16
Task 2: Evaluating Hierarchical Aspect of
Concepts in Embeddings
There is a relatively long standing research for measuring the similarity of two given concepts s(ci , cj )
either across ontologies or inside a common ontology.
Typically, the similarity of concepts is calculated at the lexical level and at the conceptual level.
We present three metrics which can be employed for evaluating the embeddings of concepts with
respect to the hierarchical structure and the semantics.
17
Absolute Error
We introduce the metric absolute semantic error which quantitatively measures the quality of
embeddings for concepts against their semantic similarity.
18
Semantic Relatedness metric
We tune this metric from word embedding for knowledge graphs by exchanging words for concepts.
Typically, this metric represents the relatedness score of two given words. In the context of a knowledge
graph, we give a pair of concepts to human judges (usually domain experts) to rate the relatedness score
on a predefined scale, then, the correlation of the cosine similarity of the embeddings for concepts is
measured with human judge scores using Spearman or Pearson.
19
Visualization
The embeddings of all concepts of the knowledge graph can be represented in a two-dimensional
visualization. This approach is an appropriate means for qualitative evaluation of the hierarchical aspect
of concepts. The visualizations are given to a human who judges them to recognize patterns revealing the
hierarchical structure and the semantics.
20
Experiment
21
Task 3: Evaluating Relational Aspect of
Concepts in Embedding
A. Relation Validation: whether or not the inferred relation is compatible with the type of entities
engaged. For example, the relation capital is valid if it is recognized between entities with the types
country and city.
B. Domain and Range of Relations: This validation process in a knowledge graph is eased by
considering the axioms rdfs:domain and rdfs:range of the schema properties and rdf:type of entities.
The expectation from embeddings generated for relations is to truly reflect compatibility with the
embeddings of the concepts asserted in the domain and range
22
Metrics for evaluating the quality of the
embeddings for concepts and relations
A. Selectional preference: This metric presented in assesses the relevance of a given noun as a
subject or object of a given verb (e.g. people-eat or city-talk). We tune this metric for knowledge
graphs as pairs of concept-relation which are represented to a human judge for the approval or
disapproval of their compatibility
B. Semantic transition distance: The inspiration for this metric comes from where Mikolov
demonstrated that capital cities and their corresponding countries follow the same distance. We
introduce this metric relying on an objective assessment. This metric considers the relational
axioms (i.e. rdfs:domain and rdfs:range) in a knowledge graph.
23
Discussion
➔ there is no single embedding model which shows superior performance in every scenario. For
example, while the skip-gram model performs better in the categorization task, the GloVe and
CBOW model perform better for the hierarchical task. Thus, one conclusion is that each of these
models is suited for a specific scenario.
➔ The other conclusion is that it seems that each embedding model captures specific features of the
ontological concepts, so integrating or aligning these embeddings can be a solution for fully
capturing all of these features.
➔ Although our initial expectation was that the embeddings learned from the knowledge graph (i.e.
DBpedia) should have higher quality in comparison to the embeddings learned from unstructured
data (i.e. Wikipedia), in practice we did not observe that as a constant behaviour.
24
Conclusion
➔ Required to generate embeddings with respect to the these three aspects
➔ Improving and extending the evaluation scenario
25
26

More Related Content

What's hot

High availability and fault tolerance of openstack
High availability and fault tolerance of openstackHigh availability and fault tolerance of openstack
High availability and fault tolerance of openstackDeepak Mane
 
DataWarehouse
DataWarehouseDataWarehouse
DataWarehousenzuguem
 
Hadoop Hbase - Introduction
Hadoop Hbase - IntroductionHadoop Hbase - Introduction
Hadoop Hbase - IntroductionBlandine Larbret
 
EDB Failover Manager - Features and Demo
EDB Failover Manager - Features and DemoEDB Failover Manager - Features and Demo
EDB Failover Manager - Features and DemoEDB
 
How to add air-gapping and immutability to Veeam backup appliance
How to add air-gapping and immutability to Veeam backup applianceHow to add air-gapping and immutability to Veeam backup appliance
How to add air-gapping and immutability to Veeam backup applianceMaryJWilliams2
 
Cours Big Data Chap2
Cours Big Data Chap2Cours Big Data Chap2
Cours Big Data Chap2Amal Abid
 
Aioug connection poolsizingconcepts
Aioug connection poolsizingconceptsAioug connection poolsizingconcepts
Aioug connection poolsizingconceptsToon Koppelaars
 
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)Alphorm
 
WekaIO: Making Machine Learning Compute Bound Again
WekaIO: Making Machine Learning Compute Bound AgainWekaIO: Making Machine Learning Compute Bound Again
WekaIO: Making Machine Learning Compute Bound Againinside-BigData.com
 
BigData_Chp3: Data Processing
BigData_Chp3: Data ProcessingBigData_Chp3: Data Processing
BigData_Chp3: Data ProcessingLilia Sfaxi
 
A Rusty introduction to Apache Arrow and how it applies to a time series dat...
A Rusty introduction to Apache Arrow and how it applies to a  time series dat...A Rusty introduction to Apache Arrow and how it applies to a  time series dat...
A Rusty introduction to Apache Arrow and how it applies to a time series dat...Andrew Lamb
 
Physical Memory Models.pdf
Physical Memory Models.pdfPhysical Memory Models.pdf
Physical Memory Models.pdfAdrian Huang
 
TP1 Big Data - MapReduce
TP1 Big Data - MapReduceTP1 Big Data - MapReduce
TP1 Big Data - MapReduceAmal Abid
 
Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...
Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...
Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...Hatim CHAHDI
 

What's hot (20)

CephFS Update
CephFS UpdateCephFS Update
CephFS Update
 
High availability and fault tolerance of openstack
High availability and fault tolerance of openstackHigh availability and fault tolerance of openstack
High availability and fault tolerance of openstack
 
DataWarehouse
DataWarehouseDataWarehouse
DataWarehouse
 
Hadoop Hbase - Introduction
Hadoop Hbase - IntroductionHadoop Hbase - Introduction
Hadoop Hbase - Introduction
 
EDB Failover Manager - Features and Demo
EDB Failover Manager - Features and DemoEDB Failover Manager - Features and Demo
EDB Failover Manager - Features and Demo
 
How to add air-gapping and immutability to Veeam backup appliance
How to add air-gapping and immutability to Veeam backup applianceHow to add air-gapping and immutability to Veeam backup appliance
How to add air-gapping and immutability to Veeam backup appliance
 
Base des données réparties
Base des données répartiesBase des données réparties
Base des données réparties
 
spark_intro_1208
spark_intro_1208spark_intro_1208
spark_intro_1208
 
Cours Big Data Chap2
Cours Big Data Chap2Cours Big Data Chap2
Cours Big Data Chap2
 
Aioug connection poolsizingconcepts
Aioug connection poolsizingconceptsAioug connection poolsizingconcepts
Aioug connection poolsizingconcepts
 
Polymorphisme (cours, résumé)
Polymorphisme (cours, résumé)Polymorphisme (cours, résumé)
Polymorphisme (cours, résumé)
 
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
alphorm.com - Formation Oracle Database 11g DBA 1 (1Z0-052)
 
WekaIO: Making Machine Learning Compute Bound Again
WekaIO: Making Machine Learning Compute Bound AgainWekaIO: Making Machine Learning Compute Bound Again
WekaIO: Making Machine Learning Compute Bound Again
 
BigData_Chp3: Data Processing
BigData_Chp3: Data ProcessingBigData_Chp3: Data Processing
BigData_Chp3: Data Processing
 
A Rusty introduction to Apache Arrow and how it applies to a time series dat...
A Rusty introduction to Apache Arrow and how it applies to a  time series dat...A Rusty introduction to Apache Arrow and how it applies to a  time series dat...
A Rusty introduction to Apache Arrow and how it applies to a time series dat...
 
Physical Memory Models.pdf
Physical Memory Models.pdfPhysical Memory Models.pdf
Physical Memory Models.pdf
 
Data Guard Architecture & Setup
Data Guard Architecture & SetupData Guard Architecture & Setup
Data Guard Architecture & Setup
 
TP1 Big Data - MapReduce
TP1 Big Data - MapReduceTP1 Big Data - MapReduce
TP1 Big Data - MapReduce
 
Une introduction à HBase
Une introduction à HBaseUne introduction à HBase
Une introduction à HBase
 
Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...
Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...
Cours HBase et Base de Données Orientées Colonnes (HBase, Column Oriented Dat...
 

Similar to Metrics for Evaluating Quality of Embeddings for Ontological Concepts

An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...
An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...
An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...ijaia
 
G04124041046
G04124041046G04124041046
G04124041046IOSR-JEN
 
Barzilay & Lapata 2008 presentation
Barzilay & Lapata 2008 presentationBarzilay & Lapata 2008 presentation
Barzilay & Lapata 2008 presentationRichard Littauer
 
ONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATION
ONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATIONONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATION
ONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATIONIJDKP
 
SEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONS
SEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONSSEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONS
SEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONSIJDKP
 
Context Sensitive Relatedness Measure of Word Pairs
Context Sensitive Relatedness Measure of Word PairsContext Sensitive Relatedness Measure of Word Pairs
Context Sensitive Relatedness Measure of Word PairsIJCSIS Research Publications
 
Effect of word embedding vector dimensionality on sentiment analysis through ...
Effect of word embedding vector dimensionality on sentiment analysis through ...Effect of word embedding vector dimensionality on sentiment analysis through ...
Effect of word embedding vector dimensionality on sentiment analysis through ...IAESIJAI
 
Schema-agnositc queries over large-schema databases: a distributional semanti...
Schema-agnositc queries over large-schema databases: a distributional semanti...Schema-agnositc queries over large-schema databases: a distributional semanti...
Schema-agnositc queries over large-schema databases: a distributional semanti...Andre Freitas
 
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGINTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGdannyijwest
 
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED  ON SEMANTIC TAG RANKINGINTELLIGENT SOCIAL NETWORKS MODEL BASED  ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGdannyijwest
 
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGINTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGIJwest
 
Deep Neural Methods for Retrieval
Deep Neural Methods for RetrievalDeep Neural Methods for Retrieval
Deep Neural Methods for RetrievalBhaskar Mitra
 
factorization methods
factorization methodsfactorization methods
factorization methodsShaina Raza
 
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)ijceronline
 
Secured Ontology Mapping
Secured Ontology Mapping Secured Ontology Mapping
Secured Ontology Mapping dannyijwest
 
[Emnlp] what is glo ve part ii - towards data science
[Emnlp] what is glo ve  part ii - towards data science[Emnlp] what is glo ve  part ii - towards data science
[Emnlp] what is glo ve part ii - towards data scienceNikhil Jaiswal
 
Semantics at Scale: A Distributional Approach
Semantics at Scale: A Distributional ApproachSemantics at Scale: A Distributional Approach
Semantics at Scale: A Distributional ApproachAndre Freitas
 

Similar to Metrics for Evaluating Quality of Embeddings for Ontological Concepts (20)

Marvin_Capstone
Marvin_CapstoneMarvin_Capstone
Marvin_Capstone
 
An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...
An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...
An Entity-Driven Recursive Neural Network Model for Chinese Discourse Coheren...
 
G04124041046
G04124041046G04124041046
G04124041046
 
Barzilay & Lapata 2008 presentation
Barzilay & Lapata 2008 presentationBarzilay & Lapata 2008 presentation
Barzilay & Lapata 2008 presentation
 
ONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATION
ONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATIONONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATION
ONTOLOGY INTEGRATION APPROACHES AND ITS IMPACT ON TEXT CATEGORIZATION
 
SEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONS
SEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONSSEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONS
SEMANTICS GRAPH MINING FOR TOPIC DISCOVERY AND WORD ASSOCIATIONS
 
Context Sensitive Relatedness Measure of Word Pairs
Context Sensitive Relatedness Measure of Word PairsContext Sensitive Relatedness Measure of Word Pairs
Context Sensitive Relatedness Measure of Word Pairs
 
Effect of word embedding vector dimensionality on sentiment analysis through ...
Effect of word embedding vector dimensionality on sentiment analysis through ...Effect of word embedding vector dimensionality on sentiment analysis through ...
Effect of word embedding vector dimensionality on sentiment analysis through ...
 
Schema-agnositc queries over large-schema databases: a distributional semanti...
Schema-agnositc queries over large-schema databases: a distributional semanti...Schema-agnositc queries over large-schema databases: a distributional semanti...
Schema-agnositc queries over large-schema databases: a distributional semanti...
 
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGINTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
 
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED  ON SEMANTIC TAG RANKINGINTELLIGENT SOCIAL NETWORKS MODEL BASED  ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
 
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKINGINTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
INTELLIGENT SOCIAL NETWORKS MODEL BASED ON SEMANTIC TAG RANKING
 
Deep Neural Methods for Retrieval
Deep Neural Methods for RetrievalDeep Neural Methods for Retrieval
Deep Neural Methods for Retrieval
 
factorization methods
factorization methodsfactorization methods
factorization methods
 
call for papers, research paper publishing, where to publish research paper, ...
call for papers, research paper publishing, where to publish research paper, ...call for papers, research paper publishing, where to publish research paper, ...
call for papers, research paper publishing, where to publish research paper, ...
 
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
 
1808.10245v1 (1).pdf
1808.10245v1 (1).pdf1808.10245v1 (1).pdf
1808.10245v1 (1).pdf
 
Secured Ontology Mapping
Secured Ontology Mapping Secured Ontology Mapping
Secured Ontology Mapping
 
[Emnlp] what is glo ve part ii - towards data science
[Emnlp] what is glo ve  part ii - towards data science[Emnlp] what is glo ve  part ii - towards data science
[Emnlp] what is glo ve part ii - towards data science
 
Semantics at Scale: A Distributional Approach
Semantics at Scale: A Distributional ApproachSemantics at Scale: A Distributional Approach
Semantics at Scale: A Distributional Approach
 

More from Saeedeh Shekarpour

CEVO: Comprehensive EVent Ontology Enhancing Cognitive Annotation on Relations
CEVO: Comprehensive EVent Ontology  Enhancing Cognitive Annotation on RelationsCEVO: Comprehensive EVent Ontology  Enhancing Cognitive Annotation on Relations
CEVO: Comprehensive EVent Ontology Enhancing Cognitive Annotation on RelationsSaeedeh Shekarpour
 
A quality type aware annotated corpus and lexicon for harassment research
A quality type aware annotated corpus and lexicon for harassment researchA quality type aware annotated corpus and lexicon for harassment research
A quality type aware annotated corpus and lexicon for harassment researchSaeedeh Shekarpour
 
Tutorial on Question Answering Systems
Tutorial on Question Answering Systems Tutorial on Question Answering Systems
Tutorial on Question Answering Systems Saeedeh Shekarpour
 
Semantic Interpretation of User Query for Question Answering on Interlinked Data
Semantic Interpretation of User Query for Question Answering on Interlinked DataSemantic Interpretation of User Query for Question Answering on Interlinked Data
Semantic Interpretation of User Query for Question Answering on Interlinked DataSaeedeh Shekarpour
 

More from Saeedeh Shekarpour (7)

CEVO: Comprehensive EVent Ontology Enhancing Cognitive Annotation on Relations
CEVO: Comprehensive EVent Ontology  Enhancing Cognitive Annotation on RelationsCEVO: Comprehensive EVent Ontology  Enhancing Cognitive Annotation on Relations
CEVO: Comprehensive EVent Ontology Enhancing Cognitive Annotation on Relations
 
A quality type aware annotated corpus and lexicon for harassment research
A quality type aware annotated corpus and lexicon for harassment researchA quality type aware annotated corpus and lexicon for harassment research
A quality type aware annotated corpus and lexicon for harassment research
 
Windowing of attention
Windowing of attentionWindowing of attention
Windowing of attention
 
Tutorial on Question Answering Systems
Tutorial on Question Answering Systems Tutorial on Question Answering Systems
Tutorial on Question Answering Systems
 
Semantic Interpretation of User Query for Question Answering on Interlinked Data
Semantic Interpretation of User Query for Question Answering on Interlinked DataSemantic Interpretation of User Query for Question Answering on Interlinked Data
Semantic Interpretation of User Query for Question Answering on Interlinked Data
 
Sina presentation in IBM
Sina presentation in IBMSina presentation in IBM
Sina presentation in IBM
 
Wi presentation
Wi presentationWi presentation
Wi presentation
 

Recently uploaded

MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 

Recently uploaded (17)

MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 

Metrics for Evaluating Quality of Embeddings for Ontological Concepts

  • 1. Metrics for Evaluating Quality of Embeddings for Ontological Concepts Faisal Alshargi University of Leipzig Saeedeh Shekarpour University of Dayton Tommaso Soru University of Leipzig Amit Sheth Kno.e.sis Center 1
  • 2. Motivation: the growth of Linked Open Data 2 March 2018 ~3000 Datasets >190 Triples
  • 3. Motivation: Knowledge Transformation It is necessary to dispose this valuable knowledge for extrinsic tasks such as natural language processing or data mining. ➔ the knowledge (i.e. schema level and instance level) has to be transformed from discrete representations to numerical representations (called embeddings). 3
  • 4. Research Gap and our Contribution Existing deficiency: is the lack of an evaluation framework for comprehensive and fair judgment on the quality of the embeddings of concepts Evaluation can be intrinsic or extrinsic Our Contribution: concerned with evaluating the quality of embeddings for concepts. It extends the state of the art by providing several intrinsic metrics for evaluating the quality of the embedding of concepts on three aspects: i. the categorization aspect, ii. the hierarchical aspect, and iii. the relational aspect. 4
  • 5. Ontological Concept Ontological concepts play a crucial role in ➔ capturing the semantics of a particular domain ➔ typing entities which bridge a schema level and an instance level ➔ determining valid types of sources and destinations for relations in a knowledge graph. Thus, the embeddings of the concepts are expected to truly reflect characteristics of ontological concepts in the embedding space. 5
  • 6. Contribution contributing in providing several metrics for evaluating the quality of the embedding of concepts from three perspectives: (i) how the embedding of concepts behaves for categorizing their instantiated entities; (ii) how the embedding of concepts behaves with respect to hierarchical semantics described in the underlying ontology; and (iii) how the embedding of concepts behaves with respect to relations 6
  • 7. Embedding Models (1) ➔ Word2vec: Skip-Gram Model and Continuous Bag of Words (CBOW) Model. learns two separate embeddings for each target word wi , (i) the word embedding and (ii) the context embedding. ➔ GloVe Model: The GloVe model (Pennington, Socher, and Manning 2014) is a global log-bilinear regression model for the unsupervised learning of word embeddings. It captures global statistics of words in a corpus and benefits the advantages of the other two models: (i) global matrix factorization and (ii) local context window methods. ➔ RDF2Vec Model: RDF2Vec (Ristoski and Paulheim 2016) is an approach for learning embeddings of entities in RDF graphs. It initially converts the RDF graphs into a set of sequences using two strategies: (i) Weisfeiler-Lehman Subtree RDF Graph Kernels, and (ii) graph random walks. Then, word2vec is employed for learning embeddings over these produced sequences. 7
  • 8. Embedding Models (2) ➔ Translation-based Models: The TransE and TransH models assume that the embeddings of both the entities and relations of a knowledge graph are represented in the same semantic space, whereas the TransR considers two separate embedding spaces for entities and relations. All three approaches share the same principle, summing the vectors of the subject and the predicate, one can obtain an approximation of the vectors of the objects. ➔ Other Knowledge Graph Embedding (KGE) Models. ◆ HolE (Holographic Embeddings) ◆ DistMult ◆ ComplEx ◆ ... All approaches above have shown to reach state-of-the-art performances on link prediction and triplet classification. 8
  • 9. Excluding of non-scalable KGE Approaches ➔ We selected the knowledge graph embedding approaches for the evaluation of our metrics among RDF2Vec, TransE and three of the methods described in the previous subsection (i.e., HolE, DistMult, and ComplEx) ➔ Differently from RDF2Vec, we could not find DBpedia embeddings pre-trained using any of the other approaches online, thus we conducted a scalability test on them to verify their ability to handle the size of DBpediawe decided to select only the more scalable RDF2Vec approach in our evaluation 9
  • 10. Data Preparation ➔ Corpus: DBpedia + Wikipedia ➔ From the DBpedia ontology, we selected 12 concepts, which are positioned in various levels of the hierarchy. Furthermore, for each concept, we retrieved 10,000 entities typed by it (in case of unavailability, all existing entities were retrieved). ➔ For each concept class, we retrieved 10,000 instances and their respective labels; in case of unavailability, all existing instances were retrieved. ➔ Then, the embeddings of these concepts as well as their associated instances were computed from the embedding models: (i) skip-gram, and (ii) CBOW and (iii) GloVe trained on Wikipedia and DBpedia 10
  • 11. Task 1: Evaluating the Categorization Aspect of Concepts in Embeddings ➔ Ontological concepts C categorize entities by typing them, mainly using rdf:type. In other words, all the entities with a common type share specific characteristics. For example, all the entities with the type dbo:Country have common characteristics distinguishing them from the entities with the type dbo:Person. ➔ Research Question: How far is the categorization aspect of concepts captured (i.e., encoded) by an embedding model? In other words, we aim to measure the quality of the embeddings for concepts via observing their behaviour in categorization tasks. 11
  • 12. Categorization metric ➔ In the context of unstructured data, this metric aligns a clustering of words into different categories ➔ how well the embedding of a concept performs as the background concept of the entities typed by it. ➔ To quantify this metric, we compute the averaged vector of the embeddings of all the entities having type c and then compute the cosine similarity of this averaged vector and the embedding of the concept Vc 12
  • 14. Coherence metric It measures whether or not a group of words adjacent in the embedding space are mutually related. Commonly, this relatedness task has been evaluated in a subjective manner (i.e. using a human judge). However, in the context of structured data we define the concept of relatedness as the related entities which share a background concept, a background concept is the concept from which a given entity is typed (i.e. inherited). For example, the entities dbr:Berlin and dbr:Sana’a are related because both are typed by the concept dbo:City 14
  • 15. Coherence metric A. Quantitative Evaluation: For the given concept c and the given radius n, we find the n-top similar entities from the pool (having the highest cosine similarity with Vc ). Then, the coherence metric for the given concept c with the radius n is computed as the number of entities having the same background concept as the given concept; B. Qualitative Evaluation: visualization approach 15
  • 17. Task 2: Evaluating Hierarchical Aspect of Concepts in Embeddings There is a relatively long standing research for measuring the similarity of two given concepts s(ci , cj ) either across ontologies or inside a common ontology. Typically, the similarity of concepts is calculated at the lexical level and at the conceptual level. We present three metrics which can be employed for evaluating the embeddings of concepts with respect to the hierarchical structure and the semantics. 17
  • 18. Absolute Error We introduce the metric absolute semantic error which quantitatively measures the quality of embeddings for concepts against their semantic similarity. 18
  • 19. Semantic Relatedness metric We tune this metric from word embedding for knowledge graphs by exchanging words for concepts. Typically, this metric represents the relatedness score of two given words. In the context of a knowledge graph, we give a pair of concepts to human judges (usually domain experts) to rate the relatedness score on a predefined scale, then, the correlation of the cosine similarity of the embeddings for concepts is measured with human judge scores using Spearman or Pearson. 19
  • 20. Visualization The embeddings of all concepts of the knowledge graph can be represented in a two-dimensional visualization. This approach is an appropriate means for qualitative evaluation of the hierarchical aspect of concepts. The visualizations are given to a human who judges them to recognize patterns revealing the hierarchical structure and the semantics. 20
  • 22. Task 3: Evaluating Relational Aspect of Concepts in Embedding A. Relation Validation: whether or not the inferred relation is compatible with the type of entities engaged. For example, the relation capital is valid if it is recognized between entities with the types country and city. B. Domain and Range of Relations: This validation process in a knowledge graph is eased by considering the axioms rdfs:domain and rdfs:range of the schema properties and rdf:type of entities. The expectation from embeddings generated for relations is to truly reflect compatibility with the embeddings of the concepts asserted in the domain and range 22
  • 23. Metrics for evaluating the quality of the embeddings for concepts and relations A. Selectional preference: This metric presented in assesses the relevance of a given noun as a subject or object of a given verb (e.g. people-eat or city-talk). We tune this metric for knowledge graphs as pairs of concept-relation which are represented to a human judge for the approval or disapproval of their compatibility B. Semantic transition distance: The inspiration for this metric comes from where Mikolov demonstrated that capital cities and their corresponding countries follow the same distance. We introduce this metric relying on an objective assessment. This metric considers the relational axioms (i.e. rdfs:domain and rdfs:range) in a knowledge graph. 23
  • 24. Discussion ➔ there is no single embedding model which shows superior performance in every scenario. For example, while the skip-gram model performs better in the categorization task, the GloVe and CBOW model perform better for the hierarchical task. Thus, one conclusion is that each of these models is suited for a specific scenario. ➔ The other conclusion is that it seems that each embedding model captures specific features of the ontological concepts, so integrating or aligning these embeddings can be a solution for fully capturing all of these features. ➔ Although our initial expectation was that the embeddings learned from the knowledge graph (i.e. DBpedia) should have higher quality in comparison to the embeddings learned from unstructured data (i.e. Wikipedia), in practice we did not observe that as a constant behaviour. 24
  • 25. Conclusion ➔ Required to generate embeddings with respect to the these three aspects ➔ Improving and extending the evaluation scenario 25
  • 26. 26