Declarative Programming for Statistical ML: The democratization of complex data does not mean dropping the data on everyone’s desk and saying, “good luck”! It means to make machine learning methods usable in such a way that people can easily instruct machines to have a “look” at complex data and help them to understand and act on it. Existing statistical relational learning and probabilistic programming languages only provide partial solutions; most of them do not support convex optimization commonly used in machine learning.
In this talk I will present RELOOP, a declarative mathematical programming language embedded into Python. It allows the user to specify mathematical programs before she knows what individuals are in the domain and, therefore, before she knows what variables and constraints exist. It facilitates the formulation of abstract, general knowledge. And, it reveals the rich logical structure underlying many machine learning problems to the solver and, turn, may make it go faster.
With this, people can start to rapidly develop statistical machine learning approaches for complex data. For instance, adding just three lines of RELOOP code makes a linear support vector machines aware of any underlying network that connects the objects to be classified.
Joint work with Martin Mladenov and many others.
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Kristian Kersting, Associate Professor for Computer Science, TU Dortmund University, Germany at MLconf SEA - 5/20/16
1. Declarative Programming
for Statistical ML
Kristian
Kersting
Martiin
Mladenov
TUD
Babak
Ahmadi
PicoEgo
Amir
Globerson
HUJI
Martin
Grohe
RWTH
Sriraam
Natarajan
U. Indiana
Leonard
Kleinhans
TUD
Danny
Heinrich
TUD
andmany
more…
Pavel
Tokmakov
INRIA
Grenoble
2. Is there a “-01” flag for
Statistical ML?
Kristian
Kersting
Martiin
Mladenov
TUD
Babak
Ahmadi
PicoEgo
Amir
Globerson
HUJI
Martin
Grohe
RWTH
Sriraam
Natarajan
U. Indiana
Leonard
Kleinhans
TUD
Danny
Heinrich
TUD
andmany
more…
Pavel
Tokmakov
INRIA
Grenoble
4. There is an arms race to “deeply”
understand data
Kristian Kersting - Declarative Programming for Statistical ML
5. Kristian Kersting - Declarative Programming for Statistical ML
Take your spreadsheet …
Features
Objects
6. Latent Dirichlet AllocationMatrix Factorization
Features
Objects
… and apply some ML
Gaussian Processes
Decision Trees/Boosting
Autoencoder/Deep Learning
and many more …
Kristian Kersting - Declarative Programming for Statistical ML
Support Vector Machines
7. IS IT REALLY THAT SIMPLE?
Kristian Kersting - Declarative Programming for Statistical ML
9. Guy van den Broeck
UCLA
card
(1,d2)
card
(1,d3)
card
(1,pAce)
card
(52,d2)
card
(52,d3)
card
(52,pAce)
…
…
…
…
10. Guy van den Broeck
UCLA
card
(1,d2)
card
(1,d3)
card
(1,pAce)
card
(52,d2)
card
(52,d3)
card
(52,pAce)
…
…
…
…
11. Guy van den Broeck
UCLA
No independencies.
Fully connected.
22704 states
card
(1,d2)
card
(1,d3)
card
(1,pAce)
card
(52,d2)
card
(52,d3)
card
(52,pAce)
…
…
…
…
Kristian Kersting - Declarative Programming for Statistical ML
12. Guy van den Broeck
UCLA
A machine will not solve
the problem
card
(1,d2)
card
(1,d3)
card
(1,pAce)
card
(52,d2)
card
(52,d3)
card
(52,pAce)
…
…
…
…
14. Kristian Kersting - Declarative Programming for Statistical ML
Symmetry-Aware Message Passing
Compress the model
Run message passing
inference on the
smaller model
[Singla, Domingos AAAI’08; Kersting, Ahmadi, Natarajan UAI’09; Ahmadi, Kersting, Mladenov, Natarajan MLJ’13;
Mladenov, Globerson, Kersting AISTATS `14, UAI `14; Mladenov, Kersting UAI´15; ...]
Big model
small model
15. Kristian Kersting - Declarative Programming for Statistical ML
De Raedt, Kersting, Natarajan, Poole
“Statistical Relational Artificial Intelligence”,2016
… the study and design of intelligent agents that act in noisy
worlds composed of objects and relations among the objects
Statistical Relational AI
ScalingUncertainty
Logic
Graphs
Trees
Mining
And
Learning
[Getoor, Taskar MIT Press ’07; De Raedt, Frasconi, Kersting, Muggleton, LNCS’08; Domingos, Lowd Morgan Claypool
’09; Natarajan, Kersting, Khot, Shavlik Springer Brief’15; Russell CACM 58(7): 88-97 ’15]
16. BUT WAIT A MINUTE! WE WANT
TO USE SOME ML, NOT JUST
GRAPHICAL MODELS!
Kristian Kersting - Declarative Programming for Statistical ML
Latent Dirichlet Allocation
Matrix Factorization
Gaussian Processes
Decision Trees/Boosting
Autoencoder/Deep Learning
and many more …
Support Vector Machines
17. Let’s say we want to classify
publications that cite each other
18. This is a quadratic program. If you replace
l2- by l1-,l∞-norm you get a linear program
Standard ML approach:
Support Vector Machines
Kristian Kersting - Declarative Programming for Statistical ML
[Vapnik ´79; Bennett´99; Mangasarian´99; Zhou, Zhang, Jiao´02, ... ]
19. Write down the problem in „paper form“. The machine
then compiles automatically into algebraic solver form.
Statistical Machine Learning via
Declarative Programming
[Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16]
Logically parameterized variable
(set of ground variables)
Logically parameterized constraint
Logically parameterized objective
Data stored
externally
20. Program1
Data1
Program2
Data2
Program3
Data3
...
Kristian Kersting - Declarative Programming for Statistical ML
[Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16]
Captures the essence of a problem and
can be reused for several problems
21. MP1
Declarative
Program
MP2 MPn
Data1 Data2 Datan
...
Kristian Kersting - Declarative Programming for Statistical ML
[Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16]
Captures the essence of a problem and
can be reused for several problems
23. Simply program some additional constraints
[Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16]
Kristian Kersting - Declarative Programming for Statistical ML
25. Finally, the „-O1“ flag
[Kersting, Mladenov, Tokmakov AIJ 2015, Mladenov, Kleinhans, Kersting 2016]
Kristian Kersting - Declarative Programming for Statistical ML
(1) Reduce the QP via symmetries
(2) Run any solver on the reduced QP
26. … and the “-02” flag
Kristian Kersting - Declarative Programming for Statistical ML
Algebraic
Decision Diagrams
Formulae
parse trees
Matrix Free
Optimization
( )+
= Optimization with 60 Millions of non-zeros
with 12 minutes per log-barrier iteration and
actually sublinear in the number of non-zeros
27. HIGH-LEVEL LANGUAGES
FOR MACHINE LEARNING
AND OPTIMIZATION ARE A
STEP TOWARDS THE …
Conclusions
Kristian Kersting - Declarative Programming for Statistical ML
28. DEMOCRATIZATION OF
MACHINE LEARNING
Kristian Kersting - Declarative Programming for Statistical ML
Reduces the level of expertise necessary to
build optimization applications, makes models
faster to write and easier to communicate
Facilitate the construction of sophisticated
models with rich domain knowledge
Speed up solvers by exploiting language
properties, compression, and compilation