Curse of Dimensionality and Big Data

Stephane.Marchand-Maillet@unige.ch – University of Geneva – KEYSTONE Summer School – © July 2015 - 1
Curse of Dimensionality
and
Big Data
Stephane Marchand-Maillet
Viper group
University of Geneva
Switzerland

• Are you familiar with vector spaces?
– Dimension, projection
• Are you familiar with statistics?
– Mean, variance, Gaussian distribution
• Are you familiar with linear algebra?
– Matrix, inner product
• Are you familiar with indexing?
– Principle, methods
• Do you realise all the above is one and the same
thing?
– That’s what we’ll see 
– I hope it will not be just trivial…
Quick get-to-know (profiling  )

• To provide you with an overview of
– Basics of data modelling
– Potential issues with high-dimensional data
– Large-scale indexing techniques
• To create bridges between basic techniques
– For better intuition
– To understand what is the information we
manipulate
– To understand what approximations are made
• To start you on doing your own data modelling
Objectives of the tutorial

Outline
• Motivation and context
• Large-scale high-dimensional data
• Fighting the dimensionality
• Fighting large-scale volumes
4
Note: Several illustrations from within these slides have been borrowed from the Web, including Wikipedia or teaching
material. Please do not reproduce without permission from the respective authors. When in doubt, don't.

Data Production
• Growth of Data
– 1,250B GB (=1.2EB) of data generated in
2010.
– Data generation growing at a rate of 58%
per year
• Baraniuk, R., “More is Less: Signal Processing and the
Data Deluge”, Science, V331, 2011.
1 exabyte (EB) = 1,073,741,824 gigabytes
0
2000
4000
6000
8000
10000
2010 2011 2012 2013 2014
DataSize(EB)
Data Generation Growth
http://www.intel.com/conte
nt/www/us/en/communicati
ons/internet-minute-
infographic.html
http://www.ritholtz.com/blog
/2011/12/financial-industry-
interconnectedness/
Internet
Scientific
Industry
Data
By Sverre Jarp,
By Felix'Schürmann
© Copyright attached

A digital world

[Picture from: http://www.intel.com/content/www/us/en/communications/internet-minute-infographic.html]
Data communication

User “productivity”
[Picture from: http://www.go-gulf.com/wp-content/themes/go-gulf/blog/online-time.jpg - Feb 2012]

Motivation
• Decision making requires informed
choices
• The information is often not easy to
manage and access
• The information is often
overwhelming
– « Big Data » trend
We need to bring a structure to the raw data
• Document (data) representation
• Similarity measurements
• Further analysis: mining, retrieval, learning

Information management process
Raw documents
Representation space
(visualisation)
Document features
User interaction
Feature extraction
“Appropriate”
mapping
“Decision”
process

Example: text
Text documents
Feature extraction
“Appropriate”
mapping
User interaction“Decision”
process
“Word” occurrences

Example: Images
Images
Feature extraction
“Appropriate”
mapping
User interaction“Decision”
process
Photo collage
Filtering
Color histogram

Also...
• Any type of media: webpage, audio, video,
data,...
• Objects, based on their characteristics
• People in social networks
• Concepts: processes, states, ... Etc
 Anything for which “characteristics” may be
measured

The key is distance
• Features help characterizing 1 document (summary)
• Features help comparing 2 documents
• How can they help structuring a collection?

Most often back to the local neighbours
- Information retrieval
- Similarity query
- Machine learning
- Generalization
- Data mining
- Discover continuous patterns
Distance measurements

However
Two main issues:
• High-dimensional data
– «Curse of dimensionality»
• Large data
– «Big data»

• Raw data (the documents) carries information
• Computer essentially perform additions
• We need to represent the data somehow to
provide the computer with as much as
possible faithful information
• The representation is an opportunity for us to
transfer some prior (domain) knowledge as
design assumptions
If this (data modelling) step is flawed, the
computer will work with random information
Representation spaces (intuition)

Given a set C of N documents di, mapped onto a
set X of points xi of a M-dimensional vector
space RM
• To index and organise (exploit) this collection,
we must understand its underlying structure
We study its geometrical properties
Notion of distance, neighbourhood
We study its statistical properties
Density, generative law
Both are the same information!
Approach

Terminology
Given a set C of N documents di, mapped onto a
set X of points xi of a M-dimensional vector
space RM
Two main issues:
– M increases
• Large data
– N increases

• C={d1,d2,…,dN} a collection of documents
– For each document, perform feature extraction f
– di is represented by its feature vector xi in RM
– xi is the view of di from the computer perspective
– f: C  X = {x1,x2,…,xN}
• Examples
– Images: xi is a 128-bin color histogram: M=128
– Text: xi measures the occurrence of each word of
the dictionary: M=50’000
Representation spaces

We have
We want to create an order or a structure over X
– We define a topology on the representation space
We study distances
We study neighborhoods (kNN)
Representation spaces
M
iN RxxxxX  },...,,{ 21
M
M Reee ofbasis},...,,{ 21

Norms and distances
• Norm
– norm of x, vector of RM
– if the norm derives from an inner
product
Exple:
• Distance (metric)
• Norm and distance
M
iN RxxxxX  },...,,{ 21
x
xxxxx T
 ,
2
 

M
i
i
M
i
ii xxyxyx
1
2
1
.,

 RXXd :
xxxd  0),(
yxyxdyxd ,),(),( 
yzydyxdzxd  ),(),(),(
yxyxd ),(

• Examples of norms (distances)
– Minkowsky (Lp norms)
• p=1 : norm L1
• p=2: L2 norm (Euclidean)
• : norm
• Unit ball for distance d(.,.)
Norms and distances M
iN RxxxxX  },...,,{ 21
pM
i
p
ip
xx
1
1






 


M
i
ixx
1
1
 ii xx max
p L
(open)}1),(s.t{)(
(closed)}1),(s.t{)(


yxdyxB
yxdyxB
d
d
1
2

Ilustrations: http://www.viz.tamu.edu/faculty/ergun
Wikipedia
)()()(),(),(),(
2
2
2
2
2
yxyxyxyxyxyxdyxd T
E 

• Generalised Euclidean distance
• Mahalanobis distance
Norms and distances
2
1
)(
1
),( ii
M
i i
G yx
w
yxd  
)0;0(s.d.p  xAxxRA TMxM
)()(),( 12
yxAyxyxd T
A  
2Idif ddA A 
GAi ddwiagA  )(dif

• Hausdorff distance (set distance)
X, Y sous ensembles de C
Norms and distances
)),(infsup),,(infsupmax(),( yxdyxdYXd
yyXxXxYy
H


(Illustration: Wikipedia)

• Unit masses at positions xi
• Center of mass
• Inertia wrt point a:
• Inertia wrt subspace F:
• Huygens theorem:
Physics and statistics M
iN RxxxxX  },...,,{ 21

i
ix
N
g
1

i
ia xadXI ),()( 2
),()()( 2
gadXIXI ag 
Physics Statistics
Mass(xi) Probability P(xi)
Centre of mass g Mean mEX
Inertia Ig Variance s2=V(X)

i
iF xFdXI ),()( 2

• Relation between standard deviation and
distribution around the mean
– : at least half of the values are between
– Gaussian distribution N(0,1) :
• Centred variable:
Chebichev inequality
1;0)(;
)(
*
**


 X
X
XE
XEX
X s
s
2
1
)(
n
nXP  sm
2n ]2,2[ smsm 
9973.0)3( XP
Illustration: Wikipedia

Markov inequality
• Upper bound of the cumulative distribution
• Useful for proofs and upper bounds
0
)(
)(  a
a
XE
aXP

n random variables (X1,…,Xn) such that E(Xi)=m
then
is an « estimator » for m
and if V(Xi)=s2
Weak law of large numbers


n
i
iX
n
XNn
1
* 1
mm p
n
n
XEXP


 )(00)(lim
2
)( sp
n
XV



• N uniform draws U([0,1])
•
Simulation: exponential distribution
n=10 n=100 n=1’000 n=3’000
n=10’000 Mean Standard deviation
3)ln(
1
 

UX

X such that E(X)=m et V(X)=s2
X1,…,Xn random variables iid with X
Then, Zn converges (in probability) to N(0,1)
Central Limit Theorem
dxebZaP
b
a
x
n
n 


 2
2
2
1
)(lim
s
)(
1
1
*
m
s
 
X
n
ZX
n
XNn n
n
i
i

• n uniform draws U([-0.5,0.5])
• Average n distributions: n draws of
Simulation: Normal distribution
X
n=1 n=4n=3n=2
n=100
Zn: Mean Zn: standard deviation

• X random variable whose mean m to be estimated
– Exple: « Diameter »
• Xi population
– Exple: « Apples »
• xi : measures
– Exple: « measured diameters »
 (mean of measures) tend to X
(by the Weak Law of Large Numbers)
• The Central Limit Theorem says that the error on the
estimate of m (Zn) follow a normal law N(0,1)
Zn is a random variable representing the error carried
by
Interpretation
X
X
nZ mm

• In vector spaces, distances are essentially measured using
difference of coordinates
• Statistical distribution may be considered as statistical objects with
inter-distances (similarity)
• However, it would not be relevant to compare their intrinsic values
directly. We rather use Divergences
• The most known/used divergence: KL-Divergence (Kullback Liebler)
– Given two distributions P and Q, the KL divergence between P and Q is
the measure of how much information is lost when Q is used to
approximate P
– The discrete formulation of the KL divergence is
– DKL is non-symmetric, it can be symmetrised (to better approach a
distance) as
A quick note on divergences

i
KL
iQ
iP
iPQPD
)(
)(
ln)()(
2
)()(
)(
PQDQPD
QPD KLKL 


Topology (very loose intuition)
• A topology is built based on neighbourhood
• The neighbourhood is the base for the definition of continuity
• Continuity implies some assumption of the propagation of a function
(some smoothness)
In our context
• We are given data points (localised scattered information)
• We need to gain some “smoothness”
• We will propagate the information “around” our data points
• Distance identifies neighbourhoods
• We somehow “interpolate” (spread) information between data
points
• Because that our “best guess”!
• Everything depends on the fact of having informative characteristics
to localise our similar documents (di) as neighbouring points (xi)

One of the main problems in Data Analysis
• Given a query point
• Find its neighbourhood (vicinity) V
k-NN (nearest neighbour)
is the nearest (k-)neighbour
is the farthest k-neighbour
-NN >0, fixed (range query)
Nearest neighbours
M
Rq
*
Nk 
 },...,{),(),(s.t,...,, 121 kjiiii iijxqdxqdxxxV lk

 kxqdxxxV lk iiii  ),(s.t,...,, 21
1ix
kix

Voronoi diagram
ci: Voronoi cell associated to point xi
Delaunay Graph
D=(C,E) : points xi are the vertices of D
(xi,xj) is an edge if ci and cj have a common edge
The graph connects neighbouring cells
Space partitioning
M
iN RxxxxX  },...,,{ 21
 ijyxdyxdRyc ji
M
i  ),(),(t.q.
Ilustrations: http://www.wblut.com
Wikipedia
ci
xi
xj
(xi,xj)

We, as human, are experts in 1D, 2D, 3D, a bit less
in 4D (time) and less so afterwards
In high dimensions (eg 20 is enough), counter
intuitive properties appear
Eg:
• Sparsity
• Concentration of distances
• Relation to kNN: hubness
which we try to model here, to better understand
why things go wrong (or not as good)
Curse of Dimensionality

• M is the dimension of the space (and the data)
– Measures, characteristics, …
• X is therefore the sample data of a M-
dimensional space
What if M increases?
– Influence on geometric measures (distances, k-NN)
– Influence on statistical distributions
« Curse of dimensionality »
Richard Ernest Bellman (1961). Adaptive control
processes: a guided tour. Princeton University Press.
High dimensionality M
iN RxxxxX  },...,,{ 21

Imagine a data sample in [a,b]M
We quantify every dimension with k bins
To estimate the distribution we require n samples in each
bin in average
• M=1: N~k.n
• M=2: N~n.k2
…
• M: N~n.kM
Exple:
k=10, n=10, M=6 => N ~ 10’000’000 samples required
High dimensionality

Curse of dimensionality
• Sparsity
– N samples
– M dimensions
– k quantization steps
n samples per bin
or
to maintain n constant
41
M
k
N
n ~
M
kN ~

42
Mki
k
N
xPE ~))bin(( 
• Consequences:
– With finite sample size (limited data collection), most
of the cells are empty if the feature dimension is too
high
– The estimation of probability density is unreliable

• Gaussian distribution
43
M
XP )9973.0()3( 
M
1 99.7%
10 97.3%
100 76.3%
500 25.8%
1000 6.7%
)3( XP

Neighbourhood structure
• S a ball around a point (radius r, dimension M)
• C a cube around a point [-r,+r]M
0
)2/(2)(
)(
ratio
)2()(
)2/(
2
)(
1
2/
2/
 







M
M
M
C
S
M
C
MM
S
MMMV
MV
rMV
MM
r
MV



• Most of the neighbours of the centre are «in the
corners of the cube»
• Empty space: each point (center) sees its neighbours
far away
0
)(
)(
  M
C
S
MV
MV
0))((  M
i SxPE

• S a ball around a point (radius r, dimension M)
• C enclosed cube: side a
?
)2/(2)(
)(
ratio 12/1
2/
 

 

M
MM
M
C
S
MMMV
MV 
M
r
aM
a
r
2
2

M
C
M
r
MV 






2
)(
)2/(
2
)(
2/
MM
r
MV
MM
S




Dmax and Dmin are smallest and largest neighbour
distances
High-dimensional k-NN
Dmax Dmin
Beyer, K., Goldstein, J.,
Ramakrishnan, R., and Shaft, U.
(1999). When is“nearest neighbor”
meaningful? In Proceedings of the
7th International Conference on
Database Theory, pages 217–235
01
)(
Plimnthe0
)(
limif
min
minmax
M







D
DD











M
kM
kM
XE
X
V
Thm [Beyer et al, 1999]

Loss of contrast:
High-dimensional k-NN
Dmax Dmin
 Computational imprecision prevents
relevance
 Noise is taking over
 -NN: all or nothing
 k-NN: random draw
0
)(
min
minmax


D
DD M
P

Loss of contrast 2
/])]1,0([[ MU M
Dimension
Norm

Loss of contrast 2
/)]1,0([ MN M
Dimension
Norm

• Consequences
– Database index based on metric distances
• K-d-tree
• VP-tree
have to perform exhaustive search
“Every enclosing rectangle encloses everything”
High dimensional k-NN
Illustrations: Peter N. Yianilos.
Data Structures and Algorithms
for Nearest Neighbor Search in
General Metric Spaces. Fourth
ACM-SIAM Symposium on
Discrete Algorithms, 1993

In M dimension, the unit hypercube has as
diagonal u=[1 1 … 1]T, then
High dimensional diagonals

M
Mu 2
u
e1
Dimension

In M dimension, the unit hypercube has as
0
1
),cos()cos( 1
1


MT
M
Mu
eu
eu

u
e1

In M dimensions, the unit hypercube has as
• In high dimensions, all (2M-1) diagonal vectors are
orthogonal to the basis vectors
• High dimensional spaces have an exponential
number of dimensions
• Everything along the diagonals is projected onto
the origin

2




M
M

Given a Gaussian distribution in a M-dimensional space
N(mM,SM), what is the density of samples of radius r?
With no loss of generality we study the centered
distribution N(0,IM)
Gaussian distribution
]r-dr,r+dr[

Gaussian distribution ]r-dr,r+dr[
MM
M
rVM
M
rE
M
X
M
r
rMXX
rrXP
NX,...,X(XX
M
i
i
M
i
i
T
iM
22
)(1
1
)(
1
~
1
.
)),...,((ofestimation
)1,0(~)
2
22
2
1
22
2
1
22
2
1










kVkE
XVabaXV
bXaEbaXE
2)(;)(
)(2)(
)()(
22





]r-dr,r+dr[
MM
M
rVM
M
rE
rrXP
NX,...,X(XX
T
iM
22
)(1
1
)(
)),...,((forestimation
)1,0(~)
2
22
1



« Gaussian egg »
Dimension
0
1
)),...,(( T
rrXP 
)( rXP 
)( rXP 

MM
M
rVM
M
rE
rrXP
NX,...,X(XX
T
iM
22
)(1
1
)(
)),...,((forestimation
)1,0(~)
2
22
1



« Gaussian egg »
Dimension
)),...,(( T
rrXP 
)( rXP 
)( rXP 
For a M-dimensional Normal distribution of mean 0
and s.d 1, the expected distribution marginalised
over concentric spheres has a mean of 1 and a
variance converging to 0
Intuition: The volume of the sphere tends to 0 goes
against the high density at the centre

Empirical evidence (10’000 samples)
Dimension
Bins on [0,2]

Empirical evidence (10’000 samples)
)( sXP
)),...,(( T
XP ss
)( sXP
Probability
Dimension

Empirical evidence (10’000 samples)Probability(cumulative)
Dimension
)),...,(( T
XP ss
)( sXP
)( sXP

Consequences
• Loss of contrast: the relative spread of points
is not seen accurately
• Conversely: using high dimensional Gaussian
distributions to model the data may not be as
accurate

• We want to characterise the number of times
a sample appears in the k-NN of another
sample:
The distribution of Nk is skewed to the left. A
small number of samples appear in the
neighbourhood of many samples
Hubs



 

i
ikk
ik
ik
xPxN
xx
xP
)()(
otherwise0
)(NNif1
)(

20-NN M=100 (1000 samples) (50bins)
Bin
Frequency

Hubness
Dimension
Bin

When using the cosine distance as similarity
measure
Centering the data helps reducing the hubness
Hubs: centering
yx
yx
yxd
T
1),(cos
I Suzuki et al. Centering Similarity Measures to Reduce Hubs.2013 Conf. on Empirical Methods in NLP.

Lesson
Although data points may be uniformly
distributed, the Lp norms being sums of
coordinate distances,
the computed distances are corrupted by the
excess of uniformative dimensions
As a result, points appear non uniformely
distributed
pM
i
p
ip
xx
1
1






 

Summary
Two main issues:
– Making distance measurements unreliable
– Making statistical estimation inaccurate
• Large data
– «Big data»
 Reduction of dimension

Dimension reduction: principle
• Given a set of data in a M-dimensional space,
we seek an equivalent representation of lower
dimension
• Dimension reduction induces a loss. What to
sacrifice? What to preserve?
– Preserve local: neighbourhood, distances
– Preserve global: distribution of data, variance
– Sacrifice local: noise
– Sacrifice global: large distances
– Map linearly
– Unfold data

Some example techniques:
• SFC: preserve neighbourhoods
• PCA: preserve global linear structures
• MDS: preserve linear neighbourhoods
• IsoMAP: Unfold neighbourhoods
• SNE family: unfold statistically
Not studied here (but also valid):
• SOM (visualisation), LLE, Random projections
(hashing)
Dimension reduction

Space-filling curves
• Definition:
– A continuous curve which passes through
every point of a closed n-cell in Euclidean n-
space Rn is called a space filling curve (SFC)
• The idea is to map the complete space
onto a simple index: a continuous line
– Directly implies an order on the dataset

Application of SFC
• Mapping multi-dimensional space to one dimensional sequence
• Applications in computer science:
– Database multi-attribute access
– Image compression
– Information visualization
– ……
Various types
• Non-recursive
– Z-Scan Curve
– Snake Scan Curve
• Recursive
– Hilbert Curve
– Peano Curve
– Gray Code Curve

Hilbert curve Ilustrations: Wikipedia

Peano Curve Ilustrations: Wikipedia

SFC
• In our case, the idea is to use SFC to
“explore” local neighborhoods, hoping
that neighborhoods will appear
“compact” on the curve
• Hence we study such mapping for SFC

Visualizing 4D Hyper-Sphere Surface
• Z-Curve  Hilbert Curve
[Illustrations from the lecture “SFC in Info Viz”, Jiwen Huo, Uni Waterloo, CA]

• Points can be identified as vectors from the
origin
• Orthogonal projection
• x gets projected in x* onto u (which we take of
unit length to represent the subspace Fu)
Projection
x
uo
x*
x-x*
 Fu

Projection
 
uuxxxxFxd
uoF
yxdx
uuxx
u
u
uy
,),(
),(minarg
,
*
2*
*







x
uo
x*
x-x*


• x* is the part of x that lives in Fu (eg subspace of
interest)
• x-x* is the residual (what is not represented)
• x and x-x* are orthogonal (they represent
complementary information)
• Point x* is the closest point from Fu to x (minimal
loss, maximal representation)
Interpretation
x
uo
x*
x-x*
 Fu

• Given a set of data in a M dimensional space,
we seek a subspace of lower dimension onto
which to project our data
• Criterion: preserve most inertia of the dataset
• Consequence: project and minimize residuals
• We construct incrementally a new subspace
by successive projections
– X is projected onto ui, find an orthogonal ui+1 to
project the residual
– At most M ui s can be found, we then select the
most representative (preserving most inertia)
Principal Component Analysis (PCA)

• The data is centred around its mean to get
minimal global inertia
• We then look for u1 the direction capturing
most inertia (minimizing the global sum of
residuals)
PCA
),(minarg 2
1 u
i
i
u
Fxdu 
m ii xx
M
iN RxxxxX  },...,,{ 21
uxxuxxuuxxuuxxFxd i
T
i
T
i
T
iii
T
iiui  ),(),(),(2
)(maxarg),(minarg
1
2
1 uutrFxdu T
u
u
i
i
u
S


uuuu
u
L
uu
L T


SS





221

• PCA incrementally finds “best fit” subspaces, orthogonal to
each other (minimize residuals)
• PCA incrementally finds directions of max variance
(maximize trace of the cov matrix)
• PCA transforms the data linearly (eigen decomposition) to
align it with its axis of max variance (and make the
covariance matrix diagonal)
• The reduction of dimension is made by selecting
eigenvectors corresponding to the (m<<M) largest
eigenvalues
• Because of the max variance criterion, PCA sees the dataset
as a set of data draw from a centred distribution penalised
by their deviation (distance) to the centre: a Normal
distribution
 PCA is a linear transformation adapted to non clustered data
PCA

PCA
[Illustration Wikipedia]

• “Discriminant”  Supervised (xi,yi), yis
“labels”
• Simple case: 2 classes
– We seek u such that the projections of xis (xi*)
onto Fu is best linearly separated
Linear Discriminant Analysis (LDA)
u
u

• Intuitively: max inter-class distance
– u parallel to the original m1-m2
• Fisher criterion adds min intra-class spread (s2)
• Fisher criterion
LDA
*
2
*
1
1
maxarg mm 
u
u
2*
2
2*
1
1
minarg ss 
u
u
2*
2
2*
1
*
2
*
1
1
maxarg
ss
mm



u
u

• Both inter- and intra-class criterion can be
generalised to multi-class
• Criteria consider classes as one Gaussian
distribution N(mj,sj) each
• Resolved by solving an eigensystem
Linear solution
• Can be used for supervised projection onto a
reduced set of dimensions
LDA

Given dij a set of inter-distances between points of a
supposed M-dimensional set X (M unknown),
• We seek points X* in a m-dimensional space (m
given) such that dij(X*) approximates dij
• We define stress functions:
which are optimised by majorization
Note: weighting by dij may help privileging local
structures (less penalty on small distance values)
Multi Dimensional Scaling (MDS)







ji
ij
ji
ijij
Y Yd
Yd
X
m
2
2
))((
))((
minarg*
d

Shepard diagram: plot dij against dij(X*)
• Ideally along the diagonal (or highly
correlated)
• Helps seeing where the discrepancy appears
MDS
[Illustration from I. Borg & PJF Groenen. Modern Multidimensional Scaling. Springer 2005]

• Recall:
• This implies that if D is an interdistance matrix
– D is symmetric
–
A quick note on “distance” matrices

 RXXd :
xxxd  0),(
yxyxdyxd ,),(),( 
)0;0(s.d.pis  xDxxD T

• Euclidean distances say that the shortest distance
between two points is along a straight line (any
diversion increases the distance value)
• This also says that if y is close to x and z, then x and z
should be reasonably close to each other
• This may not always be true
– Social nets : if y is friend with x and z, it says nothing about
the social distance between x and z (may be large)
– Data Manifold: if the data lies on a complex manifold, the
straight line is irrelevant
Non Euclidean distances

• A local neighbourhood graph (eg 5-NN graph) is built to create
a topology and ensure continuity
• Distances are replaced by geodesics (paths on the
neighbourhood graph)
• MDS is applied on this interdistance matrix (eg with m=2)
IsoMap (non Euclidean)
[Illustration from http://isomap.stanford. edu]

• Locally Euclidean neighbourhoods are
considered
– Requires a good (dense, uniform) data distribution
– Choice of the neighbourhood size to ensure
connectivity and avoid infinite distances
• Powerful to “unfold” the manifold
IsoMap

• Deterministic distance-based neighbourhoods,
which may contain noise or outlying values, are
replaced by a stochastic view
• Distances are then taken between probability
distributions
• The embedding is made “in probability”
• Given X in M-dimensional space, and m
– pj|i is the probability of xi to pick xj as a neighbour in
M-dimensional space
– qj|i is the probability of xi* to pick xj* as a neighbour in
m-dimensional space
– Do so that q stays “close” to p (divergence)
Stochastic Neighbourhood Embedding (SNE)

• X* is found by minimizing F(X’) using gradient-based
optimisation
• The definition of P and Q relax the rigid constraints
found using distances
• The exponential decay of likelyhood favors local
structures
• t-SNE uses a Student t-distribution in the low
dimensional space
SNE







k
xxd
xxd
ij
k
xxd
xxd
ij
ki
ji
i
ki
i
ji
e
e
q
e
e
p ),(
),(
|
2
),(
2
),(
| **2
**2
2
2
2
2
s
s
ij
ij
i j
ij
i
iiKL
q
p
pQPDXF
|
|
| log)()'(  

• MNIST dataset
t-SNE example
[Illustration from L. van der Maaten’s website]

Traces of our everyday activities can be:
• Captured, exchanged (production,
communication)
• Aggregated, Stored
• Filtered, Mined (Processing)
The “V”’s of Big Data:
• Volume, Variety, Velocity (technical)
• and hopefully... Value
Raw data is almost worthless, the added value is in
juicing the data into information (and knowledge)
Big Data

However
Two main issues:
– Making distance measurements unreliable
– Making statistical estimation inaccurate
• Large data
– «Big data»
– Could compensate for sparsity problems
– But hard to manage efficiently

Solutions
Two main issues
– Reduce the dimension
– Indexing for solving the kNN problem efficiently
• Large data
– Reduce the volume
– Filter, compress, cluster,…

Indexing structures
…+
M-tree
Tries
Suffix array
Suffix Tree
Inverted files
LSH…
Illustration: Wikipedia

Main ideas:
• A point is described by its neighbourhood
• The neighbourhood of a point encodes its
position
• We use only neighboring landmarks
– To be fast
• We don’t keep distances, just ranks
– To be faster
Permutation-based Indexing

L(x1, R)= (𝑟1, 𝑟2, 𝑟3, 𝑟4, 𝑟5) L(x2, R)=(𝑟1, 𝑟2, 𝑟3, 𝑟4, 𝑟5) L(x3, R)= (𝑟5, 𝑟3, 𝑟2, 𝑟4, 𝑟1)
n=5:
D={x1, . . . , x 𝑁}, N objects,
R = {𝑟1, . . . , 𝑟 𝑛} ⊂ D, n references
Each 𝑜𝑖 is identified by an ordered list:
L(x𝑖, R)= {𝑟𝑖1, . . . , 𝑟𝑖𝑛} such that
d(x𝑖, 𝑟𝑖𝑗) ≤ d(x𝑖, 𝑟𝑖𝑗+1 ) ∀j = 1, . . . , n − 1
x
y
z
1x2x
3x
4x
5x
r1
r2
r3
r4
r5

Indexing: Building ordered lists
Querying (kNN):
• Build the query ordered list
• Compare it with points ordered lists
Using the Spearman Footrule Distance:
Solving kNN: “I see what you see if I am close to you”
 
j
ririSFD
rank
i jj
RxLRqLxqdxqd || ),(),(),(),(
x
y
z
1x2x
3x
4x
5x
r1
r2
r3
r4
r5

PBI in practice
Given a query point q, we seek objects xi such
that L(xi,R) ~ L(q,R)
• We use inverted files to (pre-)select objects
such that L(xi,R)|rj ~ L(q,R)|rj
• We prune the lists with the assumption that
only local neighborhood is important
• We quantize the lists for easier indexing
• … (still an active development)

Efficiency of PBI
• Still uses distances for creating lists
• Issues with ordering due to the curse of
dimensionality
However
• The choice of reference points (location,
number) may be optimised
• Empirical performance are good

Optimising PBI

PBI: Encoding Model
𝒓 𝟏
𝒓 𝟐
𝒓 𝟑
𝒓 𝟒
𝒓 𝟓
𝜹 𝟏𝟐 𝜹 𝟐𝟑
𝛿24
𝛿43
𝛿14
𝛿15
𝛿53
𝛿45
𝐿 𝑜, 𝑅 = (1,2)
𝐿 𝑜, 𝑅 = (1,4)
𝐿 𝑜, 𝑅 = (1,5)
𝐿 𝑜, 𝑅 = (5,1)
𝐿 𝑜, 𝑅 = (4,5)
𝐿 𝑜, 𝑅 = (4,2)
𝐿 𝑜, 𝑅 = (3,4)
𝐿 𝑜, 𝑅 = (3,2)
𝐿 𝑜, 𝑅 = (2,3)
𝐿 𝑜, 𝑅 = (2,4)
𝐿 𝑜, 𝑅 = (3,5)
𝐿 𝑜, 𝑅 = (5,3)
𝐿 𝑜, 𝑅 = (5,4)
𝐿 𝑜, 𝑅 = (2,1)
𝐿 𝑜, 𝑅 = (4,1)
107

Optimising PBI

Map-Reduce principle
Two-step parallel processing of data:
• Map the data properties (values) onto
respective keys (data)
– (key,value) pairs
• Reduce the list of values for each of the keys
– (key, list of values)
– Process the list

Map Reduce – Word Count example
[Illustration: http://blog.trifork.com/2009/08/04/introduction-to-hadoop/]
• Keys: stems
• Values: occurrence (1)
• Reducing: sum (frequency)

MapReduce
• The MapReduce programming interface brings a
simple and powerful interface for data
parallelization, by keeping the user away from the
communications and exchange of data.
1. Mapping
2. Shuffling
3. Merging
4. Reducing

Distributed inverted files
• Data size: 36GB of XML data.
• Hadoop: 40 minutes.
• The best ratio between the mappers and reducers is
found to be:

• Host:
– Computer hosts the GPU card.
• Device:
– GPU
• Kernel:
– Function runs thousands of threads in
parallel
• Grids:
– Two or three-dimensional of blocks.
• Blocks:
– Consists of an upper limit of threads 512 or 1024.
• Memory:
– Local memory (Fast and Small (KB)).
– Global memory (Slow and Big (GB)).
GPU Architecture

PDPS PIOFPDSS
𝑂
𝑁𝑛
𝑃
+ 𝑁(𝑛 log 𝑛 + 𝑛) + 𝑡1
𝑂
𝑁(2𝑛 + 𝑛 log 𝑛)
𝑃
+ 𝑡2
= 𝑠 × (𝑁𝑙× 𝑚 + 𝑛 × 𝑚
+2(𝑁𝑙 × 𝑛))
= 𝑠 × (𝑁𝑙× 𝑚 + 𝑛 × 𝑚
+ 𝑁𝑙 × 𝑛 + (𝑁𝑙 × 𝑛))
= 𝑠 × (𝑁𝑙× 𝑚 + 𝑛 × 𝑚
+(𝑁𝑙 × 𝑛)
Complexity:
Memory:
𝑂
𝑁(2𝑛 + 𝑛 log 𝑛)
𝑃
+ 𝑡2
PIOF does the sorting while it calculate the distances!
Permutation Based Indexing on GPU

• Indexing looks at organising neighborhoods to
avoid exhaustive search
• Indexing may be tailored to the issue in
question
– Inverted files for text search
– Spatial indexing for neighbourhood search
Summary

• Hashing
– LSH, Random projections,
• Outlier detection
– Including in high-dimensional spaces
• Classification, regression
– With sparse data
Were not studied here…

Conclusion
“Distance is key”
– Defines the neighbourhood of points
– Defines the standard deviation around the mean
– Defines the notion of similarity
However
– Distance may have a non-intuitive behavior
– Distance may not be strictly needed
• Stochastic model for neighbourhoods (SNE)
• Ranking approach for neighbourhoods (PBI)

References
Big Data and Large-scale data
– Mohammed, H., & Marchand-Maillet, S. (2015). Scalable Indexing for
Big Data Processing. Chapman & Hall.
– Marchand-Maillet, S., & Hofreiter, B. (2014). Big Data Management
and Analysis for Business Informatics. Enterprise Modelling and
Information Systems Architectures (EMISA), 9.
– M. von Wyl, H. Mohamed, E. Bruno, S. Marchand-Maillet, “A parallel
cross-modal search engine over large-scale multimedia collections
with interactive relevance feedback” in ICMR 2011 - ACM International
Conference on Multimedia Retrieval.
– H. Mohamed, M. von Wyl, E. Bruno and S. Marchand-Maillet,
“Learning-based interactive retrieval in large-scale multimedia
collections” in AMR 2011 - 9th International Workshop on Adaptive
Multimedia Retrieval.
– von Wyl, M., Hofreiter, B., & Marchand-Maillet, S. (2012).
Serendipitous Exploration of Large-scale Product Catalogs. In 14th IEEE
International Conference on Commerce and Enterprise Computing
(CEC 2012), Hangzhou, CN.
More at http://viper.unige.ch/publications

References
Large-scale Indexing
– Mohamed, H., & Marchand-Maillet, S. (2015). Quantized Ranking for Permutation-
Based Indexing. Information Systems.
– Mohamed, H., Osipyan, H., & Marchand-Maillet, S. (2014). Multi-Core (CPU and
GPU) For Permutation-Based Indexing. In Proceedings of the 7th Internation
Conference on Similarity Search and Applications (SISAP2014), Los Cabos, Mexico.
– H. Mohamed and S. Marchand-Maillet “Parallel Approaches to Permutation-Based
Indexing using Inverted Files” in SISAP 2012 - 5th International Conference on
Similarity Search and Applications .
– H. Mohamed and S. Marchand-Maillet “Distributed Media indexing based on MPI
and MapReduce” in CBMI 2012 - 10th Workshop on Content-Based Multimedia
Indexing.
– H. Mohamed and S. Marchand-Maillet “Enhancing MapReduce using MPI and an
optimized data exchange policy”, P2S2 2012 - Fifth International Workshop
onParallel Programming Models and Systems Software for High-End Computing.
– Mohamed, H., & Marchand-Maillet, S. (2014). Distributed media indexing based on
MPI and MapReduce. Multimedia Tools and Applications, 69(2).
– Mohamed, H., & Marchand-Maillet, S. (2013). Permutation-Based Pruning for
Approximate K-NN Search. In DEXA, Prague, CZ.

References
Large data analysis – Manifold learning
– Sun, K., Morrison, D., Bruno, E., & Marchand-Maillet, S. (2013).
Learning Representative Nodes in Social Networks. In 17th Pacific-Asia
Conference on Knowledge Discovery and Data Mining, Gold Coast, AU.
– Sun, K., Bruno, E., & Marchand-Maillet, S. (2012). Unsupervised
Skeleton Learning for Manifold Denoising and Outlier Detection. In
International Conference on Pattern Recognition (ICPR'2012), Tsukuba,
JP.
– Sun, K., & Marchand-Maillet, S. (2014). An Information Geometry of
Statistical Manifold Learning. In Proceedings of the International
Conference on Machine Learning (ICML 2014), Beijing, China.
– Wang, J., Sun, K., Sha, F., Marchand-Maillet, S., & Kalousis, A. (2014).
Two-Stage Metric Learning. In Proceedings of the International
Conference on Machine Learning (ICML 2014), Beijing, China.
– Sun, K., Bruno, E., & Marchand-Maillet, S. (2012). Stochastic Unfolding.
In IEEE Machine Learning for Signal Processing Workshop (MLSP'2012),
Santander, Spain.

Curse of Dimensionality and Big Data

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Curse of Dimensionality and Big Data

Similar to Curse of Dimensionality and Big Data (20)

Recently uploaded

Recently uploaded (20)

Curse of Dimensionality and Big Data