Successfully reported this slideshow.
Upcoming SlideShare
×

# NODES 2020 extended - Manifolds in semi-supervised learning

• Full Name
Comment goes here.

Are you sure you want to Yes No

• Be the first to like this

### NODES 2020 extended - Manifolds in semi-supervised learning

1. 1. MANIFOLDS IN SEMI-SUPERVISED LEARNING Monojit Basu Director, TechYugadi IT Solutions & Consulting, Bangalore EXTENDED
2. 2. 2 Outline ● Semi-supervized Learning and Graph-based Algorithms ● Data Distribution on Manifold and Multi-manifold ● Classification Algorithms with Manifold Regularization ● Implementation Hints ● Closing Remarks
3. 3. 3 Outline ● Semi-supervized Learning and Graph-based Algorithms ● Data Distribution on Manifold and Multi-manifold ● Classification Algorithms with Manifold Regularization ● Implementation Hints ● Closing Remarks
4. 4. 4 Semi-supervized Learning: Overview ● Training Samples consist of data with and without class label ● Images with and without captions ● Text with and without tags, .. ● Model is built with both labeled and unlabeled data Prob(y|x) Prob(x) ● Smoothness Property: If two data points are close, their labels should be similar Label Data Based on labeled samples Based on both labeled and unlabeled samples
5. 5. 5 Graph-based Algorithms For SSL ● There are many many ways of exploiting smoothness property ● A simplistic baseline approach is self-training (not graph-based) ● Graph-based Algorithms are particularly effective ● Label Propagation ● Random-Walk ● Min-Cut ● Density-based Distances ● Local and Global Consistency ● Using Graph Kernels, ..
6. 6. 6 Label Propagation ● Generates a weighted graph where edges between similar neighbours have higher weights (Zhu and Ghahramani, 2002) ● Defines a transition matrix: ● Tij = probability of node i ‘jumping’ into node j, that is, taking up j’s label ● Repeatedly multiplies the current label matrix with the transition matrix (which itself gets updated) ● Until labels on all nodes stabilize (convergence) ● In effect labels propagate from labeled to unlabeled nodes 1 1 1 00 0 unlabeled
7. 7. 7 Outline ● Semi-supervized Learning and Graph-based Algorithms ● Data Distribution on Manifold and Multi-manifold ● Classification Algorithms with Manifold Regularization ● Implementation Hints ● Closing Remarks
8. 8. 8 Manifold Structures ● Data (nodes) are distributed over low and high density regions ● Two nodes that are geometrically close may not be similar ● Or equivalently, the geometry / distance measure should be redefined ● Euclidean distances and weights based on them may not work ● Such data is said to lie on a manifold ● Although not necessary, manifold structures are often observed with high-dimensional data ● More complex scenario: data may not lie on a single manifold ● This is called multi-manifold structure
9. 9. 9 Single Manifold Structures SWISS ROLL TWO MOONS
10. 10. 10 Multi-manifold Structures \$ Dollar Symbol Surface Sphere
11. 11. 11 Outline ● Semi-supervized Learning and Graph-based Algorithms ● Data Distribution on Manifold and Multi-manifold ● Classification Algorithms with Manifold Regularization ● Implementation Hints ● Closing Remarks
12. 12. 12 Manifold Regularization ● This is the technical term for semi-supervized classification of data distributed on a (single) manifold (Belkin et al., 2006) ● Key is to establish connectivity between similar nodes by staying along a high-density region ● Mathematically it involves ● Computing a matrix L derived from the ordinary weight matrix W ● Taking the top n eigenvalues of L ● Computing an indicator function using the dot product of a data point with the eigenvalues ● It is based on a theory known as Kernel Hilbert Spaces
13. 13. 13 Maniford Regularization (Schematic) DATA W L=D-W Eigen(L) dotxData Point >0 +ve -ve CLASS LABELS
14. 14. 14 Multi-manifold Regularization ● This is the technical term for semi-supervized classification of data distributed on a multi-manifold (Goldberg et al., 2009) ● Single manifold algorithm still starts with Euclidean distances, but reformulates steps based on the derived matrix L ● Multi-manifold algorithm straight away changes distance metrics ● It is based on Hellinger distances H, and ● A Mahalnabis k-nearest neighbor graph computed from H ● Complete algorithm is much longer, involving spectral clustering and self-training on each cluster
15. 15. 15 Multi-manifold Regularization (Schematic) DATA Σs Sample Cov. Mat. H kNN graph Spectral Clustering Self-trained Clusters
16. 16. 16 Multi-view Semi-supervised Learning ● Multi-view learning involves two or more independent projections for each data point ● Classic Example: web-page classification using ● Bag of words ● Links to other web-pages ● Instead of representing data as (X, y) where y is class label, it may be represented as (X1, X2, y), where Xi are views ● Somewhat related to multimodal learning (like video and audio)
17. 17. 17 Multi-view Manifold Regularization ● Can manifold regularization be extended to multi-view data ● Yes, algorithms exist, based on strong mathematical foundations, like Sindhwani and Rosenberg, 2008 ● There is actually a generic pattern for multi-view semi- supervized learning, called co-training ● Sindhwani et al., extends co-training with an algorithm called co-regularization ● It reduces the problem to a convex optimization to minimize a loss function ● The total loss function depends on individual class predictors for each view, and a couple of regularization hyperparameters
18. 18. 18 Outline ● Semi-supervized Learning and Graph-based Algorithms ● Data Distribution on Manifold and Multi-manifold ● Classification Algorithms with Manifold Regularization ● Implementation Hints ● Closing Remarks
19. 19. 19 Python Implementation ● An implementation of some of these algorithms in Python 3.x is published on github: https://github.com/techyugadi/manifold_ssl ● These algorithms offer an interface similar to scikit-learn ● There are some programs to generate synthetic data and also use the MNIST handwritten digits data ● Note: scikit-learn as of now supports only label propagation algorithm for semi-supervized learning ● R package has more algorithms but not maifold regularization ● This is early-access release, more algorithms to be published !
20. 20. 20 Outline ● Semi-supervized Learning and Graph-based Algorithms ● Data Distribution on Manifold and Multi-manifold ● Classification Algorithms with Manifold Regularization ● Implementation Hints ● Closing Remarks
21. 21. 21 Summary ● Manifold regularization is an improvement over the standard label propagation algorithm for semi-supervised learning ● It may lead to better results when data is distributed over a manifold or multi-manifold ● This class of algorithms cover a wide range of scenarios, including multi-view datasets ● These algorithms can be implemented in Python using common numpy and linear algebra packages (see github)
22. 22. 22 References ● Zhu and Ghahramani, 2002: Learning from Labeled and Unlabeled Data with Label Propagation ● Belkin, Niyogi and Sindhwani, 2006: Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples ● Sindhwani and Rosenberg, 2008: An RKHS for Multi-View Learning and Manifold Co-Regularization ● Goldberg, Zhu, Singh, Xu and Nowak, 2009: Multi-Manifold Semi-Supervised Learning
23. 23. 23 THANK YOU monojit@techyugadi.com