This is a summary of the latest research on model interpretability, including Recurrent neural networks (RNN) for Natural Language Processing (NLP) in terms of what's in an RNN.
In addition, it contains suggestion to improve machine learning based user interface, to engage users and encourage them to contribute data to adapt the models to them.
7. RecSys with Deep Learning
Autoencoders Meet Collaborative Filtering (Sedhain et al. AutoRec 2015)
Session Based Recommendations with RNN (Hidasi et al. ICLR 2016)
Wide & Deep Learning for Recommender Systems (Cheng et al. DLRS @ RecSys 2016)
17. How to explain a model prediction?
Which features contributed to the
model prediction?
And how?
Example: Decision Tree
18. “Why should I trust you?” - Explaining the Predictions of
Any Classifier
LIME: Local Interpretable Model-Agnostic Explanations
19. LIME
- Explains the model result
- Enhances User Trust
Supports Multi-class classifications, (example),
text documents, images, etc.
Packages exists for R, Python, etc.
20. LIME
● Linear approximated model with a subset of the features - O(2n)
● Lowest distance to the original inputs
● Measuring the magnitude of the predictions distance
21. SHAP - NIPS`17
SHapley Additive exPlanations
“A Unified Approach to Interpreting
Model Predictions” - Lundberg & Lee
“Connects game theory
with local explanations”
SHAP Value - Feature importance as
an impact (effect) on the output
22. - The marginal value of an agent in a coalition (impact)
- Average marginal contribution over all possible sequences
Cooperative Games:
http://www.lamsade.dauphine.fr/~airiau/Teaching/CoopGames/2011/coopgam
es-7[8up].pdf
Game Theory :: Shapley Values
23. SHAP - NIPS`17
- Calculates N linear models for a subsets of the features
- Calculates the impact on the result
- How much each feature contribute as part of a ‘coalition’?
26. Visualizing Image Classifications
As an interpretability method:
● What features are these networks really using?
● Do individual units have meaning?
● What roles are played by different layers?
● How are high-level concepts built from low-level ones?
27. Visualizing Image Classifications
Network Dissection: Quantifying Interpretability of Deep Visual Representations
https://arxiv.org/pdf/1704.05796.pdf
http://people.csail.mit.edu/bzhou/ppt/presentation_ICML_workshop.pdf
31. What is encoded / captured in a vector?
Fine-grained Analysis Of Sentence Embeddings Using Auxiliary Prediction Tasks
(Adi et al. ICLR 2017)
● Trained a model to predict: sentence length, word existence, word-order
● 300-D CBOW - most effective (even for word order!)
● LSTM Autoencoder (500 to 750 D)
32. What is encoded / captured in a vector?
Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive
neural networks process hierarchical structure
(Hypkes et al. NIPS 2016)
● Visualizing activated neurons
● Researching compositional structures (trees)
● “The scientist who wrote the natural language research paper”
33. Analyzing Hidden Representations in End-to-End Automatic Speech Recognition
Systems
(Belinkov, Glass, NIPS 2017)
● Speech Recognition
● Visualizing layers
● Which layer/neuron is responsible to which phone (sound)
What is encoded / captured in a vector?
34. What you can cram into a single $&!#* vector:
Probing sentence embeddings for linguistic properties
(Conneau et al, ACL 2018)
● Ray Mooney
● Set of tests:
○ Surface information: word content, sentence length…
○ Syntactic Information: sentence ‘correctness’, hierarchical structure (depth)
○ Semantic Information (tense, word usage)
● Testing the information that is captured in different vectors
● Again CBOW and BiLSTM stars
What is encoded / captured in a vector?
35. RNN - Debugging Translation
http://seq2seq-vis.io/ (Strobelt, 2018)
Examines the 5 stages (Encoder, Decoder, Attention, Predictions, Beam-search)
Nearest Neighbor
Visualization
- Examine model decisions
- Connect decisions to
previous examples
- Test alternative decisions
36. More open questions
What linguistic structures can be captured by
RNN?
How does a model reach a decision?
When would a model fail?
What can’t the model do?
http://u.cs.biu.ac.il/~yogo/blackbox2018.pdf
45. Human-Machine Interactions - Ofra Amir
PhD, Harvard University
Intelligent Interactive Systems
(advanced topics in information systems)
- Technion, Israel
AAMAS'18:
- HIGHLIGHTS: Summarizing Agent Behavior to
People
- Agent Strategy Summarization
50. Explainable AI
- Models are prune to make mistakes
- Interpretability to the rescue!
- Supplies a peek into the features
- Enhance user trust
- Enables a constructive feedback
51. Explainable AI using UI
- Be Creative: Explainable UI
- Using ‘stronger’ features for personalization
Future Research:
- Effective Agent Strategy Summary
- Expected Behavior under Different Conditions