The document discusses several challenges in applying deep learning to medical imaging tasks including the scarcity of labeled data, lack of interpretability, uncertainty in predictions, and difficulties with multi-center studies. It proposes and reviews various techniques to address these challenges such as data augmentation using generative models, visualization methods to interpret models, leveraging prediction uncertainty, and distributed training approaches to combine data from multiple centers. The goal is to develop artificial intelligence systems that can provide augmented intelligence to assist doctors in clinical decision making.

2. ▪Background
▪Data Augmentation
▪Visualization for Interpretability
▪Leveraging Prediction Uncertainty
▪Distributed Training for Multi-center Study
▪Other Topics
Contents

3. Background

4. Medical data has been ‘Big’ long before we use the term ‘Big Data’
The Situation
Source : http://newsletter.esahq.org/big-data-in-perioperative-medicine-research-why-we-need-it-and-the-challenges-in-employing-its-potential/
M
E
D
I
C
A
L

5. Shortage of doctors
The Situation
Source : Association of American Medical College (2017)

6. The accessibility bias – Spatiotemporal problem
The Situation
Source : Journal of American College of Radiology(2014), The Atlantic(2016)

7. Inconsistent nature of diagnostic decision making
The Situation
Source : Google(2017), Sci. Rep(2015)

8. We need a tireless and intelligent solution
The Solution
AccurateConsistent Scalable

9. Deep learning – data-driven approach for A.I.
The Solution

10. Deep learning – data-driven approach for A.I.
The Solution

11. Deep learning – data-driven approach for A.I.
The Solution

12. Three ingredients for deep learning for medical imaging
Deep Learning to the Rescue
SPIE, 1993
Med. Phys. 1995
Big Data Computational Power Algorithm

13. From natural image to medical image
Data-driven Medical Data Analysis
128K Fundus Image
with 54 Ophthalmologist label
Source : Google(2016)Source : Stanford(2017)Source : Google(2017)

14. From natural image to medical image
Data-driven Medical Data Analysis
Source : Google(2016)Source : Stanford(2017)
129K Skin Images with
dermatologists’ label
Source : Google(2017)

15. From natural image to medical image
Data-driven Medical Data Analysis
Source : Google(2016)Source : Stanford(2017)
270 Pathology slide
with pathologist label
Source : Google(2017)

16. Seminal Papers
Source : Eric Topol Twitter
Publication in Top-tier Journals on AI matching or Exceeding Doctor Performance

17. Data Augmentation

18. Unsupervised pre-training and Transfer learning
Challenges – Scarcity of Labeled Data
Source : J. Cheng et. al(2016) Source : N. Tajbakhksh et. al(2016)

19. –In many cases, data augmentation techniques used in natural images does not semantically make sense
in medical image
(flips, rotations, scale shifts, color shifts)
–Physically-plausible deformations or morphological transform can be used in limited cases.
–More augmentation choices for texture classification problems.
Data Augmentation for Effective Training Set Expansion
Challenges – Scarcity of Labeled Data
Source : H. R. Roth et. al., MICCAI, 2015

20. –In many cases, data augmentation techniques used in natural images does not semantically make sense
in medical image
(flips, rotations, scale shifts, color shifts)
–Physically-plausible deformations or morphological transform can be used in limited cases.
–More augmentation choices for texture classification problems.
Data Augmentation for Effective Training Set Expansion
Challenges – Scarcity of Labeled Data
Source : H. R. Roth et. al., MICCAI, 2015

21. Generation of synthetic dataset for dataset expansion
Challenges – Scarcity of Labeled Data
M.J.M. Chuquicusma, ISBI 2018
▪ “How To Fool Radiologists with Generative Adversarial Networks? A Visual Turing Test For Lung Cancer
Diagnosis”

22. Generation of synthetic dataset for dataset expansion
Challenges – Scarcity of Labeled Data
M.J.M. Chuquicusma, ISBI 2018
▪ Visual Turing Test

23. ▪ “Synthetic Data Augmentation using GAN for Improved Liver Lesion Classification”
Liver Lesion Generation for Data Augmentation
Challenges – Scarcity of Labeled Data
Maayan Frid-Ada et. al, ISBI 2018
Real Synthetic

24. ▪ “Generalization of Deep Neural Networks for Chest Pathology Classification in X-ray using Generative
Adversarial Networks”
Chest X-ray Generation for Data Augmentation
Challenges – Scarcity of Labeled Data
Hojjat Salehinejad et. al, ICASSP 2018

25. ▪ Generated Samples and Performance Improvement by Synthetic Augmentation
Chest X-ray Generation for Data Augmentation
Challenges – Scarcity of Labeled Data
Hojjat Salehinejad et. al, ICASSP 2018

26. ▪ “Semi-supervised Learning with Generative Adversarial Networks for Chest X-ray Classification with Ability
of Data domain Adaptation “
Semi-supervised GAN for Domain Adaptation
Challenges – Scarcity of Labeled Data
Ali Madani et. al, ISBI 2018
Real X-ray Images Generated X-ray Images

27. ▪ “Semi-supervised Learning with Generative Adversarial Networks for Chest X-ray Classification with Ability
of Data domain Adaptation “
Semi-supervised GAN for Domain Adaptation
Challenges – Scarcity of Labeled Data
Ali Madani et. al, ISBI 2018

28. Visualization for Interpretability

29. Visualization of salient regions in the image – Occlusion method
Challenges – Interpretability
Source : NYU(2013), MGH(2017)

30. Visualization of salient regions in the image – Prediction differential analysis
Challenges – Interpretability
Source : L. M. Zintgraf et. al(2017)

31. Visualization of salient regions in the image – ‘Evidence hotspot’
Challenges – Interpretability
Source : A. Jamaludin et. al(2016)

32. Visualization of salient regions in the image – Class activation map
Challenges – Interpretability
Source : Zhou et. al(2016), VUNO(2017)

33. ▪ “Improving Weakly-supervised Lesion Localization with Iterative Saliency Map Refinement”
Iterative Visualization of Region of Interest
Challenges – Interpretability
Florian Bordes et. al, MIDL, 2018

34. ▪ Experimental Results
Iterative Visualization of Region of Interest
Challenges – Interpretability
Florian Bordes et. al, MIDL, 2018

35. Region-guided Training for Localization and Classification of Findings
Challenges – Interpretability
Source : VUNO(2018)

36. Provide physician-friendly tools for annotation
Challenges – Interpretability
Source : VUNO(2018)

37. Region-guided Training for Localization and Classification of Findings
Challenges – Interpretability
Source : VUNO(2018)

38. Leveraging Prediction Uncertainty

39. ▪ “Leveraging Uncertainty Information from Deep Neural Networks for Disease Detection”
Leveraging Uncertainty for Medical Image Classification
Challenges – Uncertainty
Christian Leibig et. al., Sci Rep, 2017

40. ▪ Performance Assessment According to various Uncertainty Toleration
Leveraging Uncertainty for Medical Image Classification
Challenges – Uncertainty
Christian Leibig et. al., Sci Rep, 2017

41. ▪ Improvement in ROC via Uncertainty-informed Decision Referral
Leveraging Uncertainty for Medical Image Classification
Challenges – Uncertainty
Christian Leibig et. al., Sci Rep, 2017

42. ▪ “Leveraging Uncertainty Estimates for Predicting Segmentation Quality”
–Prediction of segmentation mask y using f
–Prediction of uncertainty z
–Prediction of segmentation quality v using g
Leveraging Uncertainty for Medical Image Segmentation
Challenges – Uncertainty
Terrance DeVries et. al., arXiv, 2018

43. ▪ Uncertainty Estimation Methods
–Maximum softmax probability
–Monte-carlo dropout
–Heteroscedastic classifier neural network
–Learned confidence estimate
Leveraging Uncertainty for Medical Image Segmentation
Challenges – Uncertainty
Terrance DeVries et. al., arXiv, 2018

44. ▪ Experimental Result
Leveraging Uncertainty for Medical Image Segmentation
Challenges – Uncertainty
Terrance DeVries et. al., arXiv, 2018

45. ▪ Mapping Analysis
Leveraging Uncertainty for Medical Image Segmentation
Challenges – Uncertainty
Terrance DeVries et. al., arXiv, 2018

46. Distributed Training for Multi-center Study

47. ▪ Sharing patient data often has limitations due to technical, legal, or ethical concerns.
▪ Distribution of model is cheaper and safer than distribution of patient data
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

48. ▪ Experimental Result
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

49. ▪ Experimental Result
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

50. ▪ Experimental Result
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

51. ▪ Experimental Result
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

52. ▪ Experimental Result
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

53. ▪ Experimental Result
Distributed Training of Deep Neural Network among Institutions
Challenges – Multicenter Study
Ken Chang et. al., JAMIA, 2018

54. ▪ “Multi-center Machine Learning in Imaging Psychiatry : A Meta-model Approach”
–Detection of first episode patient(FES) is difficult task especially with small dataset.
–Combining dataset from multiple center brings problems as:
• Technical issues : different scanners, imaging sequences
• Legal or ethical issues : sharing sensitive data
• Computational complexity : high-order time and memory complexity
Meta-model for Aggregating Models from Multiple Institutions
Challenges – Multicenter Study
Petr Dluhos et. al., NeuroImage, 2018

55. ▪ Experimental Result
Meta-model for Aggregating Models from Multiple Institutions
Challenges – Multicenter Study
Petr Dluhos et. al., NeuroImage, 2018

56. ▪ Experimental Result
Meta-model for Aggregating Models from Multiple Institutions
Challenges – Multicenter Study
Petr Dluhos et. al., NeuroImage, 2018

57. ▪ Labeler Modeling or Disagreement Prediction
▪ Security Concerns from Adversarial Attack
▪ Anomaly vs Abnormality
▪ and many more ….
Topics to be addressed
Other Topics
Samuel G. Finlayson et. al, arXiv, 2018 Maithra Raghu et. al, arXiv, 2018

58. Artificial intelligence will become doctors’ best
companion by providing augmented intelligence
for their clinical decision making
hello@vuno.co