The document summarizes takeaways from various talks and presentations at the ICML 2019 conference. It discusses topics like safe machine learning and biases in algorithms, active learning techniques, attention mechanisms in deep learning, differential privacy in census data, time series forecasting methods, Hawkes processes, Shapley values for explainability and data valuation, topological data analysis, optimal transport, applications of machine learning in robotics, Gaussian processes, learning from noisy labels, interpretability methods in NLP, and the GluonTS library for probabilistic time series modeling.

1. Takeaways from ICML 2019
Hong Kong Machine Learning Meetup
Season 1 Episode 12 – Season Finale
Gautier Marti
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 1 / 41

2. Table of contents
1 Day 1 - Tutorials
2 Day 2 - U.S. Census, Time Series, Hawkes Processes, Shapley values,
Topological Data Analysis, Optimal Transport for Graphs
3 Day 3 - Robotics, Gaussian Processes, Learning with noisy labels
4 Day 4 - Interpretability, Natural Language Processing
5 Day 5 - Workshop Time Series
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 2 / 41

3. Disclaimer
I was sent to the ICML 2019 conference by my employer Shell Street Labs.
However, the opinions expressed in this presentation are my own and do
not reflect in any ways the view of my employer.
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 3 / 41

4. Section 1
Day 1 - Tutorials
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 4 / 41

5. Subsection 1
Safe Machine Learning
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 5 / 41

6. Safe Machine Learning
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 6 / 41

7. COMPAS - An example of ML biases & misspecification
We show, however, that the widely used commercial risk
assessment software COMPAS is no more accurate or fair than
predictions made by people with little or no criminal justice
expertise.
The accuracy, fairness, and limits of predicting recidivism
https://advances.sciencemag.org/content/4/1/eaao5580
Our analysis of Northpointe’s tool, called COMPAS (which
stands for Correctional Offender Management Profiling for
Alternative Sanctions), found that black defendants were far
more likely than white defendants to be incorrectly judged to be
at a higher risk of recidivism, while white defendants were more
likely than black defendants to be incorrectly flagged as low risk.
How We Analyzed the COMPAS Recidivism Algorithm
https://www.propublica.org/article/
how-we-analyzed-the-compas-recidivism-algorithm
GitHub: https://github.com/propublica/compas-analysis
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 7 / 41

8. RL in the Wild - Other examples of misspecifications
Reinforcement learning algorithms can break in surprising,
counterintuitive ways.
Faulty Reward Functions in the Wild
https://openai.com/blog/faulty-reward-functions/
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 8 / 41

9. Robustness - Adversarial attacks
Robust Physical-World Attacks on
Deep Learning Visual Classifica-
tion
https://arxiv.org/pdf/
1707.08945.pdf
Fooling automated surveillance
cameras: adversarial patches to
attack person detection
https://arxiv.org/pdf/
1904.08653.pdf
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 9 / 41

10. Subsection 2
Active Learning: From Theory to Practice
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 10 / 41

11. Active Learning: From Theory to Practice
Slides for the tutorial: http://nowak.ece.wisc.edu/ActiveML.html
Active Learning tries to answer the question:
Can we train machines with less labeled data and less human
supervision?
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 11 / 41

12. Active Learning
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 12 / 41

13. Rethinking classical model generalization
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 13 / 41

14. Subsection 3
A Tutorial on Attention in Deep Learning
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 14 / 41

15. A Tutorial on Attention in Deep Learning
Slides: http://alex.smola.org/talks/ICML19-attention.pdf
https://www.d2l.ai/
https://github.com/d2l-ai
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 15 / 41

16. Section 2
Day 2 - U.S. Census, Time Series, Hawkes Processes,
Shapley values, Topological Data Analysis, Optimal
Transport for Graphs
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 16 / 41

17. Subsection 1
Differential privacy
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 17 / 41

18. U.S. Census & Differential privacy
Good related read: https://www.sciencemag.org/news/2019/01/
can-set-equations-keep-us-census-data-private
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 18 / 41

19. Subsection 2
Time Series
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 19 / 41

20. Deep Factors for Forecasting
https://arxiv.org/pdf/1905.12417.pdf
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 20 / 41

21. Hawkes Processes
ICML 2018 tutorial: http://learning.mpi-sws.org/tpp-icml18/
http://proceedings.mlr.press/v97/trouleau19a/trouleau19a.pdf
cf. tick for practitioners: https://github.com/X-DataInitiative/tick
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 21 / 41

22. Subsection 3
Shapley values: Explainability & Data Valuation
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 22 / 41

23. Shapley values
A new trend in ML based on:
Shapley, Lloyd S. A value for n-person games. Contributions to the
Theory of Games 2.28 (1953). 158 citations.
φi (v) =
S⊆N{i}
|S|!(N − |S| − 1)!
N!
(v(S ∪ {i}) − v(S))
Recent applications:
Explainability: A Unified Approach to Interpreting Model Predictions
http://papers.nips.cc/paper/
7062-a-unified-approach-to-interpreting-model-predictions.
pdf
Data valuation: Data Shapley: Equitable Valuation of Data for
Machine Learning
http://proceedings.mlr.press/v97/ghorbani19c/ghorbani19c.pdf
Since computationally intensive, many papers try to approximate these
values fast...
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 23 / 41

24. Subsection 4
Topological Data Analysis
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 24 / 41

25. Topological Data Analysis
Already a tutorial at ICML 2013: Topological Data Analysis and Machine
Learning http://www2.stat.duke.edu/~sayan/Primoz/ICML.pdf
A round-up of TDA papers at ICML 2019:
https://bastian.rieck.me/blog/posts/2019/icml_tda_roundup/
(6 TDA-related papers)
Take-home message: TDA can provide robust features to Machine
Learning models.
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 25 / 41

26. Subsection 5
Optimal Transport
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 26 / 41

27. Optimal Transport
Trending in the ML community (at least 5 ICML 2019 papers) since
Cuturi’s 2013 NIPS paper: Sinkhorn Distances: Lightspeed Computation
of Optimal Transport, which made Optimal Transport for Machine
Learning possible in practice. Way too slow before!
Theoretical contributions:
On Efficient Optimal Transport: An Analysis of Greedy and
Accelerated Mirror Descent Algorithms
Methodology:
Optimal Transport for structured data with application on graphs
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 27 / 41

28. Section 3
Day 3 - Robotics, Gaussian Processes, Learning with
noisy labels
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 28 / 41

29. Subsection 1
Machine Learning with Application to Robotics
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 29 / 41

30. Machine Learning with Application to Robotics
Solving complex PDEs and other stochastic control problems in real time
is not really feasible as of today. Machine Learning (supervised learning of
trajectories, learning from demonstration, etc.) can help robotics.
Recorded talk:
https://www.facebook.com/icml.imls/videos/2368059266588651/
Lab: http://lasa.epfl.ch/
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 30 / 41

31. Subsection 2
Gaussian Processes
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 31 / 41

32. Gaussian Processes
One flagship project: The Automatic Statistician
https://www.automaticstatistician.com/index/
Discovering Latent Covariance Structures for Multiple Time Series
https://arxiv.org/pdf/1703.09528.pdf
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 32 / 41

33. Subsection 3
Labels. . .
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 33 / 41

34. Labels. . .
Learning Dependency Structures for Weak Supervision Models
https://arxiv.org/pdf/1903.05844.pdf
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 34 / 41

35. Section 4
Day 4 - Interpretability, Natural Language Processing
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 35 / 41

36. Subsection 1
Interpretability
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 36 / 41

37. Interpretability
Towards a Deep and Unified Understanding of Deep Neural Models in NLP
http://proceedings.mlr.press/v97/guan19a/guan19a.pdf
Explaining Deep Neural Networks with a Polynomial Time Algorithm for
Shapley Values Approximation
http://proceedings.mlr.press/v97/ancona19a/ancona19a.pdf
https://icml.cc/media/Slides/icml/2019/grandball(13-09-00)
-13-09-25-4776-explaining_deep.pdf
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 37 / 41

38. Subsection 2
Natural Language Processing
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 38 / 41

39. Natural Language Processing
MeanSum: A Neural Model for Unsupervised Multi-Document
Abstractive Summarization
https://arxiv.org/pdf/1810.05739.pdf
https://icml.cc/media/Slides/icml/2019/104(13-11-00)
-13-12-10-4891-meansum_a_neur.pdf
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 39 / 41

40. Section 5
Day 5 - Workshop Time Series
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 40 / 41

41. GluonTS
GluonTS: Probabilistic Time Series Models in Python
https://arxiv.org/pdf/1906.05264.pdf
https://github.com/awslabs/gluon-ts
Gautier Marti (Shell Street Labs) Takeaways from ICML 2019 41 / 41