SlideShare a Scribd company logo
1 of 46
Download to read offline
Label Propagation
                           Seminar:
Semi-supervised and unsupervised learning with Applications to NLP




                                               David Przybilla
                                  davida@coli.uni-saarland.de
Outline

●   What is Label Propagation

●   The Algorithm

●   The motivation behind the algorithm

●   Parameters of Label Propagation

●   Relation Extraction with Label Propagation
Label Propagation

●   Semi-supervised

●   Shows good results when the amount of
    annotated data is low with respect to the
    supervised options

●   Similar to kNN
K-Nearest Neighbors(KNN)

           ●   Shares similar ideas
               with Label Propagation

           ●   Label Propagation
               (LP) uses unlabeled
               instances during the
               process of finding out
               the labels
Idea of the Problem
                    Similar near Unlabeled
                    Instances should have
                    similar Labels




       L=set of Labeled Instances
       U =set of Unlabeled Instances
We want to find a function f such that:
The Model
●   A complete graph
     ● Each Node is an instance

     ●
       Each arc has a weight T xy




    ●   T xy is high if Nodes x and   y are similar.
The Model
●   Inside a Node:


               Soft Labels
Variables - Model
  ●   T is a matrix, holding all the weights of the graph

                                  N 1 ... N l = Labeled Data
             TllTlu               N l+1 .. N n=Unlabeled Data
             T u lT u u
Tll
Tlu
T ul
T uu
Variables - Model
●   Y is a matrix, holding the soft probabilities of
    each instance

                           YN   a                            n
                                    , R b is the probability of a
                                           being labeled as R b
            YL
            YU

                                          The problem to solve

R1 , R 2 ... R k each of the possible labels
N 1 , N 2 ... N n each of the instances to label
Algorithm




            Y will change in
              each iteration
How to Measure T?

                                                          Distance
                                                          Measure




                                         Euclidean Distance
Important Parameter
(ignore it at the moment) we will talk about this later
How to Initialize Y?
                                                 0
    ●   How to Correctly set the values of   Y       ?

    ●   Fill the known values (of the labeled data)

    ●   How to fill the values of the unlabeled data?
         → The initialization of this values can be
        arbitrary.


●   Transform T into T' (row normalization)
Propagation Step
●   During the process Y will change

                           0        1              k
                       Y    →   Y    → ... →   Y

    ●   Update   Y   during each iteration
Convergence
During the iteration
                  Clamped


     Yl                       ̄
                             T l l T̄l u               Yl
                   =
     Yu                      T̄u l T̄ u
                                     u
                                                       Yu

  Assumming we iterate infinite times then:
              1
            Y =T
              U
                ̄uu Y 0+ T ul Y L
                      u
                          ̄
              2
            Y =T
              U
                ̄uu ( T̄uu Y 0 + T ul Y L )+T ul Y L
                             u
                                  ̄          ̄
                     ...
Convergence
      ̄
Since T is normalized and                          ̄
                                 is a submatrix of T:



Doing it n times will lead to:




                                   Converges to Zero
After convergence
After convergence one can find   by solving:

               =
Optimization Problem


               w i j : Similarity between i j

   F should minimize the energy function



f (i ) and f ( j) should be similar for a high w i j
       in order to minimize
The graph laplacian
Let D be a diagonal matrix where

                            T̄i j            Rows are normalized so:
                                              D= I
The graph laplacian is defined as :

                                    ̄
                                    T

                    since   f :V → R

Then we can use the graph laplacian to act on it
So the energy function can be rewritten in terms of
Back to the optimization Problem
  Energy can be rewritten using laplacian



F should minimize the energy function.




                                                 ̄
                                  Δuu =( D uu −T uu)
                                              ̄
                                  Δuu =( I −T uu)
                                                ̄
                                  Δ ul =( Dul − T ul )
                                           ̄
                                  Δ ul =−T ul
Optimization Problem

                                                         ̄
                                          Δuu =( D uu −T uu)
 Delta can be rewritten in terms of   ̄
                                      T               ̄
                                          Δ uu=( I − T uu)
                                                        ̄
                                          Δ ul =( Dul − T ul )
                      ̄
            f u =( I −T uu)T ul f l                ̄
                                          Δ ul =−T ul




The algorithm converges to the
minimization of the Energy function
Sigma Parameter




Remember the Sigma parameter?

 ●   It strongly influences the behavior of LP.

 ●   There can be:
        ● just one
                   σ for the whole feature vector
        ● One σ per dimension
Sigma Parameter
            ●   What happens if   σ tends to be:
       –   0:
            ●   The label of an unknown instance is given by just the
                nearest labeled instance

       –   Infinite
             ● All the unlabaled instances receive the same influence

               from all labeled instances. The soft probabilities of each
               unlabeled instance is given by the class frecuency in the
               labeled data

●   There are heuristics for finding the appropiate value of sigma
Sigma Parameter - MST

        Label1

                                        Label2




This is the minimum arc connecting
two components with differents labels


                    (min weight (arc))
                 σ=
                            3
      Arc connects two components with different label
Sigma Parameter – Learning it
 How to learn sigma?
  ● Assumption :

       A good sigma will do classification with
       confidence and thus minimize entropy.

How to do it?
 ● Smoothing the transition Matrix T

 ● Finding the derivative of H (the entropy) w.r.t to

   sigma

  When to do it?
  ● when using a sigma for each dimension can

   be used to determine irrelevant dimensions
Labeling Approach
●   Once Yu is measured how do we assign labels
    to the instances?


                                 Yu




●   Take the most likely class
●   Class mass Normalization
●   Label Bidding
Labeling Approach
        ●   Take the most likely class




    ●   Simply, look at the rows of Yu, and choose for each instance
        the label with highest probability


●       Problem: no control on the proportion of classes
Labeling Approach
●   Class mass Normalization
●   Given some class proportions              P 1 , P 2 ... P k
●   Scalate each column C to             Pc




    ●   Then Simply, look at the rows of Yu, and choose for each
        instance the label with highest probability
Labeling Approach
●       Label bidding

    ●   Given some class proportions   P 1 , P 2 ... P k

1.estimate numbers of items per label        (C k )

2. choose the label with greatest number of items, take C k
items whose probabilty of being the current label is the highest
and label as the current selected label.


3. iterate through all the possible labels
Experiment Setup
●   Artificial Data
    ●   Comparison LP vs kNN (k=1)


●   Character recognition
    ●   Recognize handwritten digits
    ●   Images 16x16 pixels,gray scale
    ●   Recognizing 1,2,3.
    ●   256 dimensional vector
Results using LP on artificial data
Results using LP on artificial data




●   LP finds the structure in the data while KNN fails
P1NN
●   P1NN is a baseline for comparisons
●   Simplified version of LP




    1.During each iteration find the unlabeled instance nearest
    to a labeled instance and label it
    2. Iterate until all instances are labeled
Results using LP on Handwritten
                    dataSet
●   P1NN (BaseLine), 1NN (kNN)




    ●   Cne: Class mass normalization. Proportions from Labeled Data
    ●   Lbo: Label bidding with oracle class proportions
    ●   ML: most likely labels
Relation Extraction?
●   From natural language texts detect semantic
    relations among entities




Example:

B. Gates married Melinda French on January 1, 1994



    spouse(B.Gates, Melinda French)
Why LP to do RE?
                 Problems




  Supervised                  Unsupervised


                            Retrieves clusters of
Needs many                  relations with no
annotated data              label.
RE- Problem Definition
  ●   Find an appropiate label to an ocurrance of two
      entities in a context
Example:

….. B. Gates married Melinda French on January 1, 1994


Context
(Cpre)             Context     Entity 2
          Entity 1 (Cmid)                   Context
                               (e2)         (Cpos)
          (e1)


   Idea: if two ocurrances of entity pairs ahve similar
   Contexts, then they have same relation type
RE problem Definition - Features

●   Words: in the contexts
●   Entity Types: Person, Location, Org...
●   POS tagging: of Words in the contexts
●   Chunking Tag: mark which words in the
    contexts are inside chunks
●   Grammatical function of words in the contexts.
    i.e : NP-SBJ (subject)
●   Position of words:
    ●   First Word of e1      -is there any word in Cmid
                              -first word in Cpre,Cmid,Cpost...
    ●   Second Word of e1..   -second word in Cpre...
RE problem Definition - Labels
Experiment
●   ACE 2003 data. Corpus from Newspapers


●   Assume all entities have been identified already


●   Comparison between:
          –   Differents amount of labeled samples
              1%,10%,25,50%,75%,100%
          –   Different Similarity Functions
          –   LP, SVM and Bootstrapping
●   LP:
    ●   Similarity Function: Cosine, JensenShannon
    ●   Labeling Approach: Take the most likely class
    ●   Sigma: average similarity between labeled classes
Experiment
JensenShannon
-Similarity Measure

-Measure the distance between two probabilitiy functions

-JS is a smoothing of Kullback-Leibler divergence
                                  DK L   Kullback-Leibler
                                         divergence
                                    -not symmetric

                                     -not always has a
                                    finite value
Results
Classifying relation subtypes-
          SVM vs LP




       SVM with linear Kernel
Bootstrapping


             Train a Classifier

Seeds                             Classifier

        Update set of seeds whose
        confidence is high enough
Classifying relation types
  Bootstrapping vs LP




 Starting with 100 random seeds
Results
●   Performs well in general when there are few
    annotated data in comparison to SVM and kNN

●   Irrelevant dimensions can be identified by using
    LP

●   Looking at the structure of unlabeled data
    helps when there is few annotated data
Thank you

More Related Content

What's hot

GraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDBGraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDBArangoDB Database
 
Image segmentation with deep learning
Image segmentation with deep learningImage segmentation with deep learning
Image segmentation with deep learningAntonio Rueda-Toicen
 
Transformer in Computer Vision
Transformer in Computer VisionTransformer in Computer Vision
Transformer in Computer VisionDongmin Choi
 
Deep Learning for Graphs
Deep Learning for GraphsDeep Learning for Graphs
Deep Learning for GraphsDeepLearningBlr
 
Text classification & sentiment analysis
Text classification & sentiment analysisText classification & sentiment analysis
Text classification & sentiment analysisM. Atif Qureshi
 
サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会
サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会
サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会Kenyu Uehara
 
Feature Engineering
Feature EngineeringFeature Engineering
Feature EngineeringHJ van Veen
 
A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)Shuntaro Yada
 
「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料
「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料
「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料Ken'ichi Matsui
 
Dimensionality reduction with UMAP
Dimensionality reduction with UMAPDimensionality reduction with UMAP
Dimensionality reduction with UMAPJakub Bartczuk
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Universitat Politècnica de Catalunya
 
Nonnegative Matrix Factorization
Nonnegative Matrix FactorizationNonnegative Matrix Factorization
Nonnegative Matrix FactorizationTatsuya Yokota
 
Image-to-Image Translation pix2pix
Image-to-Image Translation pix2pixImage-to-Image Translation pix2pix
Image-to-Image Translation pix2pixYasar Hayat
 
Introduction to Transformers for NLP - Olga Petrova
Introduction to Transformers for NLP - Olga PetrovaIntroduction to Transformers for NLP - Olga Petrova
Introduction to Transformers for NLP - Olga PetrovaAlexey Grigorev
 
Visual Explanation of Ridge Regression and LASSO
Visual Explanation of Ridge Regression and LASSOVisual Explanation of Ridge Regression and LASSO
Visual Explanation of Ridge Regression and LASSOKazuki Yoshida
 

What's hot (20)

GraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDBGraphSage vs Pinsage #InsideArangoDB
GraphSage vs Pinsage #InsideArangoDB
 
Image segmentation with deep learning
Image segmentation with deep learningImage segmentation with deep learning
Image segmentation with deep learning
 
Transformer in Computer Vision
Transformer in Computer VisionTransformer in Computer Vision
Transformer in Computer Vision
 
Deep Learning for Graphs
Deep Learning for GraphsDeep Learning for Graphs
Deep Learning for Graphs
 
Text classification & sentiment analysis
Text classification & sentiment analysisText classification & sentiment analysis
Text classification & sentiment analysis
 
Restricted boltzmann machine
Restricted boltzmann machineRestricted boltzmann machine
Restricted boltzmann machine
 
Data Visualization With R
Data Visualization With RData Visualization With R
Data Visualization With R
 
サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会
サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会
サポートベクターマシン(SVM)の数学をみんなに説明したいだけの会
 
BERT
BERTBERT
BERT
 
Feature Engineering
Feature EngineeringFeature Engineering
Feature Engineering
 
Regularization
RegularizationRegularization
Regularization
 
A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)A Review of Deep Contextualized Word Representations (Peters+, 2018)
A Review of Deep Contextualized Word Representations (Peters+, 2018)
 
「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料
「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料
「ベータ分布の謎に迫る」第6回 プログラマのための数学勉強会 LT資料
 
Dimensionality reduction with UMAP
Dimensionality reduction with UMAPDimensionality reduction with UMAP
Dimensionality reduction with UMAP
 
Bert
BertBert
Bert
 
Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...Deep Learning for Computer Vision: Generative models and adversarial training...
Deep Learning for Computer Vision: Generative models and adversarial training...
 
Nonnegative Matrix Factorization
Nonnegative Matrix FactorizationNonnegative Matrix Factorization
Nonnegative Matrix Factorization
 
Image-to-Image Translation pix2pix
Image-to-Image Translation pix2pixImage-to-Image Translation pix2pix
Image-to-Image Translation pix2pix
 
Introduction to Transformers for NLP - Olga Petrova
Introduction to Transformers for NLP - Olga PetrovaIntroduction to Transformers for NLP - Olga Petrova
Introduction to Transformers for NLP - Olga Petrova
 
Visual Explanation of Ridge Regression and LASSO
Visual Explanation of Ridge Regression and LASSOVisual Explanation of Ridge Regression and LASSO
Visual Explanation of Ridge Regression and LASSO
 

Viewers also liked

Community Detection in Social Media
Community Detection in Social MediaCommunity Detection in Social Media
Community Detection in Social MediaSymeon Papadopoulos
 
Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...
Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...
Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...Spark Summit
 
Semi-Supervised Learning
Semi-Supervised LearningSemi-Supervised Learning
Semi-Supervised LearningLukas Tencer
 
Semi supervised learning
Semi supervised learningSemi supervised learning
Semi supervised learningAhmed Taha
 
Community detection in graphs
Community detection in graphsCommunity detection in graphs
Community detection in graphsNicola Barbieri
 
Language of Politics on Twitter - 03 Analysis
Language of Politics on Twitter - 03 AnalysisLanguage of Politics on Twitter - 03 Analysis
Language of Politics on Twitter - 03 AnalysisYelena Mejova
 
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...zukun
 
Semi-supervised classification for natural language processing
Semi-supervised classification for natural language processingSemi-supervised classification for natural language processing
Semi-supervised classification for natural language processingRushdi Shams
 
SocNL: Bayesian Label Propagation with Confidence
SocNL: Bayesian Label Propagation with ConfidenceSocNL: Bayesian Label Propagation with Confidence
SocNL: Bayesian Label Propagation with ConfidenceYuto Yamaguchi
 
MINING HEALTH EXAMINATION RECORDS A GRAPH-BASED APPROACH
MINING HEALTH EXAMINATION RECORDS  A GRAPH-BASED APPROACHMINING HEALTH EXAMINATION RECORDS  A GRAPH-BASED APPROACH
MINING HEALTH EXAMINATION RECORDS A GRAPH-BASED APPROACHNexgen Technology
 
GraphFrames: DataFrame-based graphs for Apache® Spark™
GraphFrames: DataFrame-based graphs for Apache® Spark™GraphFrames: DataFrame-based graphs for Apache® Spark™
GraphFrames: DataFrame-based graphs for Apache® Spark™Databricks
 
Semi-supervised Learning
Semi-supervised LearningSemi-supervised Learning
Semi-supervised Learningbutest
 
What is Agile Software Development?
What is Agile Software Development?What is Agile Software Development?
What is Agile Software Development?Blossom IO Inc.
 
Agile Software Development with Scrum – Introduction
Agile Software Development with Scrum – IntroductionAgile Software Development with Scrum – Introduction
Agile Software Development with Scrum – IntroductionBlackvard
 
[Dl輪読会]semi supervised learning with context-conditional generative adversari...
[Dl輪読会]semi supervised learning with context-conditional generative adversari...[Dl輪読会]semi supervised learning with context-conditional generative adversari...
[Dl輪読会]semi supervised learning with context-conditional generative adversari...Deep Learning JP
 
論文紹介 Semi-supervised Learning with Deep Generative Models
論文紹介 Semi-supervised Learning with Deep Generative Models論文紹介 Semi-supervised Learning with Deep Generative Models
論文紹介 Semi-supervised Learning with Deep Generative ModelsSeiya Tokui
 
Overview of Agile Methodology
Overview of Agile MethodologyOverview of Agile Methodology
Overview of Agile MethodologyHaresh Karkar
 
Agile Software Development Overview
Agile Software Development OverviewAgile Software Development Overview
Agile Software Development OverviewStewart Rogers
 
Hierarchical Label Propagation and Discovery for Machine Generated Email
Hierarchical Label Propagation and Discovery for Machine Generated EmailHierarchical Label Propagation and Discovery for Machine Generated Email
Hierarchical Label Propagation and Discovery for Machine Generated EmailKenji Esaki
 
Semi-supervised concept detection by learning the structure of similarity graphs
Semi-supervised concept detection by learning the structure of similarity graphsSemi-supervised concept detection by learning the structure of similarity graphs
Semi-supervised concept detection by learning the structure of similarity graphsSymeon Papadopoulos
 

Viewers also liked (20)

Community Detection in Social Media
Community Detection in Social MediaCommunity Detection in Social Media
Community Detection in Social Media
 
Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...
Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...
Extending Word2Vec for Performance and Semi-Supervised Learning-(Michael Mala...
 
Semi-Supervised Learning
Semi-Supervised LearningSemi-Supervised Learning
Semi-Supervised Learning
 
Semi supervised learning
Semi supervised learningSemi supervised learning
Semi supervised learning
 
Community detection in graphs
Community detection in graphsCommunity detection in graphs
Community detection in graphs
 
Language of Politics on Twitter - 03 Analysis
Language of Politics on Twitter - 03 AnalysisLanguage of Politics on Twitter - 03 Analysis
Language of Politics on Twitter - 03 Analysis
 
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
CVPR2010: Semi-supervised Learning in Vision: Part 3: Algorithms and Applicat...
 
Semi-supervised classification for natural language processing
Semi-supervised classification for natural language processingSemi-supervised classification for natural language processing
Semi-supervised classification for natural language processing
 
SocNL: Bayesian Label Propagation with Confidence
SocNL: Bayesian Label Propagation with ConfidenceSocNL: Bayesian Label Propagation with Confidence
SocNL: Bayesian Label Propagation with Confidence
 
MINING HEALTH EXAMINATION RECORDS A GRAPH-BASED APPROACH
MINING HEALTH EXAMINATION RECORDS  A GRAPH-BASED APPROACHMINING HEALTH EXAMINATION RECORDS  A GRAPH-BASED APPROACH
MINING HEALTH EXAMINATION RECORDS A GRAPH-BASED APPROACH
 
GraphFrames: DataFrame-based graphs for Apache® Spark™
GraphFrames: DataFrame-based graphs for Apache® Spark™GraphFrames: DataFrame-based graphs for Apache® Spark™
GraphFrames: DataFrame-based graphs for Apache® Spark™
 
Semi-supervised Learning
Semi-supervised LearningSemi-supervised Learning
Semi-supervised Learning
 
What is Agile Software Development?
What is Agile Software Development?What is Agile Software Development?
What is Agile Software Development?
 
Agile Software Development with Scrum – Introduction
Agile Software Development with Scrum – IntroductionAgile Software Development with Scrum – Introduction
Agile Software Development with Scrum – Introduction
 
[Dl輪読会]semi supervised learning with context-conditional generative adversari...
[Dl輪読会]semi supervised learning with context-conditional generative adversari...[Dl輪読会]semi supervised learning with context-conditional generative adversari...
[Dl輪読会]semi supervised learning with context-conditional generative adversari...
 
論文紹介 Semi-supervised Learning with Deep Generative Models
論文紹介 Semi-supervised Learning with Deep Generative Models論文紹介 Semi-supervised Learning with Deep Generative Models
論文紹介 Semi-supervised Learning with Deep Generative Models
 
Overview of Agile Methodology
Overview of Agile MethodologyOverview of Agile Methodology
Overview of Agile Methodology
 
Agile Software Development Overview
Agile Software Development OverviewAgile Software Development Overview
Agile Software Development Overview
 
Hierarchical Label Propagation and Discovery for Machine Generated Email
Hierarchical Label Propagation and Discovery for Machine Generated EmailHierarchical Label Propagation and Discovery for Machine Generated Email
Hierarchical Label Propagation and Discovery for Machine Generated Email
 
Semi-supervised concept detection by learning the structure of similarity graphs
Semi-supervised concept detection by learning the structure of similarity graphsSemi-supervised concept detection by learning the structure of similarity graphs
Semi-supervised concept detection by learning the structure of similarity graphs
 

Similar to Label propagation - Semisupervised Learning with Applications to NLP

Relaxed Utility Maximization in Complete Markets
Relaxed Utility Maximization in Complete MarketsRelaxed Utility Maximization in Complete Markets
Relaxed Utility Maximization in Complete Marketsguasoni
 
Fractional Calculus
Fractional CalculusFractional Calculus
Fractional CalculusVRRITC
 
Weatherwax cormen solutions
Weatherwax cormen solutionsWeatherwax cormen solutions
Weatherwax cormen solutionskirankoushik
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksStratio
 
Estimation of the score vector and observed information matrix in intractable...
Estimation of the score vector and observed information matrix in intractable...Estimation of the score vector and observed information matrix in intractable...
Estimation of the score vector and observed information matrix in intractable...Pierre Jacob
 
05 history of cv a machine learning (theory) perspective on computer vision
05  history of cv a machine learning (theory) perspective on computer vision05  history of cv a machine learning (theory) perspective on computer vision
05 history of cv a machine learning (theory) perspective on computer visionzukun
 
Doering Savov
Doering SavovDoering Savov
Doering Savovgh
 
NIPS2009: Sparse Methods for Machine Learning: Theory and Algorithms
NIPS2009: Sparse Methods for Machine Learning: Theory and AlgorithmsNIPS2009: Sparse Methods for Machine Learning: Theory and Algorithms
NIPS2009: Sparse Methods for Machine Learning: Theory and Algorithmszukun
 
Sienna 3 bruteforce
Sienna 3 bruteforceSienna 3 bruteforce
Sienna 3 bruteforcechidabdu
 
Classification of-signals-systems-ppt
Classification of-signals-systems-pptClassification of-signals-systems-ppt
Classification of-signals-systems-pptMayankSharma1126
 
Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)
Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)
Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)STAIR Lab, Chiba Institute of Technology
 
11.the univalence of some integral operators
11.the univalence of some integral operators11.the univalence of some integral operators
11.the univalence of some integral operatorsAlexander Decker
 
The univalence of some integral operators
The univalence of some integral operatorsThe univalence of some integral operators
The univalence of some integral operatorsAlexander Decker
 
Transactional Data Mining
Transactional Data MiningTransactional Data Mining
Transactional Data MiningTed Dunning
 
P805 bourgeois
P805 bourgeoisP805 bourgeois
P805 bourgeoiskklub
 
On probability distributions
On probability distributionsOn probability distributions
On probability distributionsEric Xihui Lin
 

Similar to Label propagation - Semisupervised Learning with Applications to NLP (20)

Relaxed Utility Maximization in Complete Markets
Relaxed Utility Maximization in Complete MarketsRelaxed Utility Maximization in Complete Markets
Relaxed Utility Maximization in Complete Markets
 
Fractional Calculus
Fractional CalculusFractional Calculus
Fractional Calculus
 
Weatherwax cormen solutions
Weatherwax cormen solutionsWeatherwax cormen solutions
Weatherwax cormen solutions
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural Networks
 
Neural network and mlp
Neural network and mlpNeural network and mlp
Neural network and mlp
 
Estimation of the score vector and observed information matrix in intractable...
Estimation of the score vector and observed information matrix in intractable...Estimation of the score vector and observed information matrix in intractable...
Estimation of the score vector and observed information matrix in intractable...
 
05 history of cv a machine learning (theory) perspective on computer vision
05  history of cv a machine learning (theory) perspective on computer vision05  history of cv a machine learning (theory) perspective on computer vision
05 history of cv a machine learning (theory) perspective on computer vision
 
Doering Savov
Doering SavovDoering Savov
Doering Savov
 
Algo complexity
Algo complexityAlgo complexity
Algo complexity
 
NIPS2009: Sparse Methods for Machine Learning: Theory and Algorithms
NIPS2009: Sparse Methods for Machine Learning: Theory and AlgorithmsNIPS2009: Sparse Methods for Machine Learning: Theory and Algorithms
NIPS2009: Sparse Methods for Machine Learning: Theory and Algorithms
 
Sienna 3 bruteforce
Sienna 3 bruteforceSienna 3 bruteforce
Sienna 3 bruteforce
 
Classification of-signals-systems-ppt
Classification of-signals-systems-pptClassification of-signals-systems-ppt
Classification of-signals-systems-ppt
 
Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)
Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)
Higher-order Factorization Machines(第5回ステアラボ人工知能セミナー)
 
11.the univalence of some integral operators
11.the univalence of some integral operators11.the univalence of some integral operators
11.the univalence of some integral operators
 
The univalence of some integral operators
The univalence of some integral operatorsThe univalence of some integral operators
The univalence of some integral operators
 
Transactional Data Mining
Transactional Data MiningTransactional Data Mining
Transactional Data Mining
 
P805 bourgeois
P805 bourgeoisP805 bourgeois
P805 bourgeois
 
On probability distributions
On probability distributionsOn probability distributions
On probability distributions
 
Linear regression
Linear regressionLinear regression
Linear regression
 
Nokton theory-en
Nokton theory-enNokton theory-en
Nokton theory-en
 

More from David Przybilla

Reproducible datascience [with Terraform]
Reproducible datascience [with Terraform]Reproducible datascience [with Terraform]
Reproducible datascience [with Terraform]David Przybilla
 
Transition Based Dependency Parsing
Transition Based Dependency ParsingTransition Based Dependency Parsing
Transition Based Dependency ParsingDavid Przybilla
 
Python in the land of serverless
Python in the land of serverlessPython in the land of serverless
Python in the land of serverlessDavid Przybilla
 
Apache Spark - Introduccion a RDDs
Apache Spark - Introduccion a RDDsApache Spark - Introduccion a RDDs
Apache Spark - Introduccion a RDDsDavid Przybilla
 
Procesamiento de Lenguaje Natural
Procesamiento de Lenguaje NaturalProcesamiento de Lenguaje Natural
Procesamiento de Lenguaje NaturalDavid Przybilla
 
Automatic generation of domain models for call centers
Automatic generation of domain models for call centersAutomatic generation of domain models for call centers
Automatic generation of domain models for call centersDavid Przybilla
 

More from David Przybilla (7)

Reproducible datascience [with Terraform]
Reproducible datascience [with Terraform]Reproducible datascience [with Terraform]
Reproducible datascience [with Terraform]
 
Transition Based Dependency Parsing
Transition Based Dependency ParsingTransition Based Dependency Parsing
Transition Based Dependency Parsing
 
Python in the land of serverless
Python in the land of serverlessPython in the land of serverless
Python in the land of serverless
 
Terraforming
Terraforming Terraforming
Terraforming
 
Apache Spark - Introduccion a RDDs
Apache Spark - Introduccion a RDDsApache Spark - Introduccion a RDDs
Apache Spark - Introduccion a RDDs
 
Procesamiento de Lenguaje Natural
Procesamiento de Lenguaje NaturalProcesamiento de Lenguaje Natural
Procesamiento de Lenguaje Natural
 
Automatic generation of domain models for call centers
Automatic generation of domain models for call centersAutomatic generation of domain models for call centers
Automatic generation of domain models for call centers
 

Recently uploaded

Protein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptxProtein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptxvidhisharma994099
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxraviapr7
 
A gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceA gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceApostolos Syropoulos
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxraviapr7
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17Celine George
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptxSlides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptxCapitolTechU
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17Celine George
 
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...Dr. Asif Anas
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptxSandy Millin
 
How to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using CodeHow to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using CodeCeline George
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxraviapr7
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...M56BOOKSTORE PRODUCT/SERVICE
 
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSyedNadeemGillANi
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 

Recently uploaded (20)

Protein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptxProtein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptx
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptx
 
A gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceA gentle introduction to Artificial Intelligence
A gentle introduction to Artificial Intelligence
 
Prelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quizPrelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quiz
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptx
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptxSlides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17
 
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
Unveiling the Intricacies of Leishmania donovani: Structure, Life Cycle, Path...
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
 
March 2024 Directors Meeting, Division of Student Affairs and Academic Support
March 2024 Directors Meeting, Division of Student Affairs and Academic SupportMarch 2024 Directors Meeting, Division of Student Affairs and Academic Support
March 2024 Directors Meeting, Division of Student Affairs and Academic Support
 
How to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using CodeHow to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using Code
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptx
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
 
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdfPersonal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
 

Label propagation - Semisupervised Learning with Applications to NLP

  • 1. Label Propagation Seminar: Semi-supervised and unsupervised learning with Applications to NLP David Przybilla davida@coli.uni-saarland.de
  • 2. Outline ● What is Label Propagation ● The Algorithm ● The motivation behind the algorithm ● Parameters of Label Propagation ● Relation Extraction with Label Propagation
  • 3. Label Propagation ● Semi-supervised ● Shows good results when the amount of annotated data is low with respect to the supervised options ● Similar to kNN
  • 4. K-Nearest Neighbors(KNN) ● Shares similar ideas with Label Propagation ● Label Propagation (LP) uses unlabeled instances during the process of finding out the labels
  • 5. Idea of the Problem Similar near Unlabeled Instances should have similar Labels L=set of Labeled Instances U =set of Unlabeled Instances We want to find a function f such that:
  • 6. The Model ● A complete graph ● Each Node is an instance ● Each arc has a weight T xy ● T xy is high if Nodes x and y are similar.
  • 7. The Model ● Inside a Node: Soft Labels
  • 8. Variables - Model ● T is a matrix, holding all the weights of the graph N 1 ... N l = Labeled Data TllTlu N l+1 .. N n=Unlabeled Data T u lT u u Tll Tlu T ul T uu
  • 9. Variables - Model ● Y is a matrix, holding the soft probabilities of each instance YN a n , R b is the probability of a being labeled as R b YL YU The problem to solve R1 , R 2 ... R k each of the possible labels N 1 , N 2 ... N n each of the instances to label
  • 10. Algorithm Y will change in each iteration
  • 11. How to Measure T? Distance Measure Euclidean Distance Important Parameter (ignore it at the moment) we will talk about this later
  • 12. How to Initialize Y? 0 ● How to Correctly set the values of Y ? ● Fill the known values (of the labeled data) ● How to fill the values of the unlabeled data? → The initialization of this values can be arbitrary. ● Transform T into T' (row normalization)
  • 13. Propagation Step ● During the process Y will change 0 1 k Y → Y → ... → Y ● Update Y during each iteration
  • 14. Convergence During the iteration Clamped Yl ̄ T l l T̄l u Yl = Yu T̄u l T̄ u u Yu Assumming we iterate infinite times then: 1 Y =T U ̄uu Y 0+ T ul Y L u ̄ 2 Y =T U ̄uu ( T̄uu Y 0 + T ul Y L )+T ul Y L u ̄ ̄ ...
  • 15. Convergence ̄ Since T is normalized and ̄ is a submatrix of T: Doing it n times will lead to: Converges to Zero
  • 16. After convergence After convergence one can find by solving: =
  • 17. Optimization Problem w i j : Similarity between i j F should minimize the energy function f (i ) and f ( j) should be similar for a high w i j in order to minimize
  • 18. The graph laplacian Let D be a diagonal matrix where T̄i j Rows are normalized so: D= I The graph laplacian is defined as : ̄ T since f :V → R Then we can use the graph laplacian to act on it So the energy function can be rewritten in terms of
  • 19. Back to the optimization Problem Energy can be rewritten using laplacian F should minimize the energy function. ̄ Δuu =( D uu −T uu) ̄ Δuu =( I −T uu) ̄ Δ ul =( Dul − T ul ) ̄ Δ ul =−T ul
  • 20. Optimization Problem ̄ Δuu =( D uu −T uu) Delta can be rewritten in terms of ̄ T ̄ Δ uu=( I − T uu) ̄ Δ ul =( Dul − T ul ) ̄ f u =( I −T uu)T ul f l ̄ Δ ul =−T ul The algorithm converges to the minimization of the Energy function
  • 21. Sigma Parameter Remember the Sigma parameter? ● It strongly influences the behavior of LP. ● There can be: ● just one σ for the whole feature vector ● One σ per dimension
  • 22. Sigma Parameter ● What happens if σ tends to be: – 0: ● The label of an unknown instance is given by just the nearest labeled instance – Infinite ● All the unlabaled instances receive the same influence from all labeled instances. The soft probabilities of each unlabeled instance is given by the class frecuency in the labeled data ● There are heuristics for finding the appropiate value of sigma
  • 23. Sigma Parameter - MST Label1 Label2 This is the minimum arc connecting two components with differents labels (min weight (arc)) σ= 3 Arc connects two components with different label
  • 24. Sigma Parameter – Learning it How to learn sigma? ● Assumption : A good sigma will do classification with confidence and thus minimize entropy. How to do it? ● Smoothing the transition Matrix T ● Finding the derivative of H (the entropy) w.r.t to sigma When to do it? ● when using a sigma for each dimension can be used to determine irrelevant dimensions
  • 25. Labeling Approach ● Once Yu is measured how do we assign labels to the instances? Yu ● Take the most likely class ● Class mass Normalization ● Label Bidding
  • 26. Labeling Approach ● Take the most likely class ● Simply, look at the rows of Yu, and choose for each instance the label with highest probability ● Problem: no control on the proportion of classes
  • 27. Labeling Approach ● Class mass Normalization ● Given some class proportions P 1 , P 2 ... P k ● Scalate each column C to Pc ● Then Simply, look at the rows of Yu, and choose for each instance the label with highest probability
  • 28. Labeling Approach ● Label bidding ● Given some class proportions P 1 , P 2 ... P k 1.estimate numbers of items per label (C k ) 2. choose the label with greatest number of items, take C k items whose probabilty of being the current label is the highest and label as the current selected label. 3. iterate through all the possible labels
  • 29. Experiment Setup ● Artificial Data ● Comparison LP vs kNN (k=1) ● Character recognition ● Recognize handwritten digits ● Images 16x16 pixels,gray scale ● Recognizing 1,2,3. ● 256 dimensional vector
  • 30. Results using LP on artificial data
  • 31. Results using LP on artificial data ● LP finds the structure in the data while KNN fails
  • 32. P1NN ● P1NN is a baseline for comparisons ● Simplified version of LP 1.During each iteration find the unlabeled instance nearest to a labeled instance and label it 2. Iterate until all instances are labeled
  • 33. Results using LP on Handwritten dataSet ● P1NN (BaseLine), 1NN (kNN) ● Cne: Class mass normalization. Proportions from Labeled Data ● Lbo: Label bidding with oracle class proportions ● ML: most likely labels
  • 34. Relation Extraction? ● From natural language texts detect semantic relations among entities Example: B. Gates married Melinda French on January 1, 1994 spouse(B.Gates, Melinda French)
  • 35. Why LP to do RE? Problems Supervised Unsupervised Retrieves clusters of Needs many relations with no annotated data label.
  • 36. RE- Problem Definition ● Find an appropiate label to an ocurrance of two entities in a context Example: ….. B. Gates married Melinda French on January 1, 1994 Context (Cpre) Context Entity 2 Entity 1 (Cmid) Context (e2) (Cpos) (e1) Idea: if two ocurrances of entity pairs ahve similar Contexts, then they have same relation type
  • 37. RE problem Definition - Features ● Words: in the contexts ● Entity Types: Person, Location, Org... ● POS tagging: of Words in the contexts ● Chunking Tag: mark which words in the contexts are inside chunks ● Grammatical function of words in the contexts. i.e : NP-SBJ (subject) ● Position of words: ● First Word of e1 -is there any word in Cmid -first word in Cpre,Cmid,Cpost... ● Second Word of e1.. -second word in Cpre...
  • 39. Experiment ● ACE 2003 data. Corpus from Newspapers ● Assume all entities have been identified already ● Comparison between: – Differents amount of labeled samples 1%,10%,25,50%,75%,100% – Different Similarity Functions – LP, SVM and Bootstrapping ● LP: ● Similarity Function: Cosine, JensenShannon ● Labeling Approach: Take the most likely class ● Sigma: average similarity between labeled classes
  • 40. Experiment JensenShannon -Similarity Measure -Measure the distance between two probabilitiy functions -JS is a smoothing of Kullback-Leibler divergence DK L Kullback-Leibler divergence -not symmetric -not always has a finite value
  • 42. Classifying relation subtypes- SVM vs LP SVM with linear Kernel
  • 43. Bootstrapping Train a Classifier Seeds Classifier Update set of seeds whose confidence is high enough
  • 44. Classifying relation types Bootstrapping vs LP Starting with 100 random seeds
  • 45. Results ● Performs well in general when there are few annotated data in comparison to SVM and kNN ● Irrelevant dimensions can be identified by using LP ● Looking at the structure of unlabeled data helps when there is few annotated data