Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
•
4 likes•5,369 views
RAAI summer school 2019, presented by Namkug Kim
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
Similar to Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3 (20)
More from Namkug Kim
More from Namkug Kim (16)
Recently uploaded
Recently uploaded (20)
Raai 2019 clinical unmet needs and its solutions of deep learning in medicine3
- 1. Clinical Unmet Needs and their Solutions of Deep Learning in Medicine Namkug Kim, PhD Medical Imaging & Intelligent Reality Lab. Convergence Medicine/Radiology, University of Ulsan College of Medicine Asan Medical Center South Korea
- 2. Conflict of Interests Establishments of Somansa Inc, CyberMed Inc, Clinical Imaging Solution Inc, and Anymedi Inc. Researches with LG Electronics, Coreline Soft Inc.,, Osstem Implant, CGBio, VUNO, Kakaobrain, AnyMedi Inc.
- 3. Slide share IFMIA 2019 plenary talk : https://www.slideshare.net/namkugkim/ifmia-2019-plenary- talk-deep-learning-in-medicine-engineers-perspectives 3
- 4. AMC : a leading hospital in Korea ▪ Total area of 524,000 m2 ▪ No.1 in number of patients and treatments ▪ No.1 in performing surgeries on top 6 cancer patients & 30 major diseases New BuildingEast BuildingWest Building Education & Research Center Convergence Innovation Building Dormitory Total Beds 2,715 licensed beds Outpatients Inpatients 13,380 (Daily average) 2,586 (Daily average) Medical Staffs 7,924 (MDs: 1,720 RNs: 3,627 Others: 2,577) Surgical Operations 91,000 Annual (250 / day) JapanChina South Korea North Korea Russia China
- 5. A world best hospital in organ transplantation & cancer surgery Transplantation in Korea Liver Heart Kidney Pancreas 34.9% 47.2% 14.6% 64.8% 98 85 88 Liver 98 97 95 Heart 95 86 86 Kidney 95 95 85 Pancreas AMC (Asan Medical Center) UNOS (United Network For Organ Sharing) KONOS (Korea Network For Organ Sharing) Survival rates (1 year)
- 6. Investing heavily in clinical and inter-disciplinary research Achieve a world-class medical reputation to aggressively invest in R&D and clinical trial Clinical Research Investigator (MD) 685 Lab Technician 839 Basic Research Investigator (PhD) 119 Administration 39 (Units : Number of personnel) Selected research-driven hospital by Korean government R&D funding US$ 1.2 billion by the government (10 hospital, 3 yrs) Over 1,600 Investigators for Research AI Established a new research center for artificial intelligence in medical imaging 3D Printing Leading 3D printing technology in medical field Genome Genomic studies on rare genetic diseases, cancer, chronic diseases, etc. Medical Robot Robotic training center for training & researching on robotic surgical skills ⋯ Development of clinical applications by open network and translating advanced technologies 1st ship building company in the world
- 7. Beyond Human-level Performance • Now, Machines Beat Human in Tasks Once Considered Impossible 5:0 vs Fan Hui (Oct. 2015) 4:1 vs Sedol Lee (Mar. 2016) Modified by KH Jung, PhD
- 9. Five tribes of AI researches From Pedro Domingos, Professor, University of Washington at MLconf ATL
- 10. Bio Plausible Neural Network Mimic human visual recognition system Neocognitron, proposed by Hubel & Wiesel in 1959 Visual primary cortex by cascading from S-Cell to C-Cell Each unit connected to a small subset of other units Based on what it sees, it decides what it wants to say Units must learn to cooperate to accomplish the task Nobel prize winner @ 1981 10From Gallant and van Esses, Simon Thorpe
- 11. CNN : Major Breakthroughs in Feedforward NN K. Fukushima Yann Lecun G. Hinton, S. Ruslan Neocognitron (1979) • By Kunihiko Fukushima • First proposed CNN Convolutional Neural Networks (1989) • Yann Lecun et.al • Back propagation for CNN • Theoretically learn any function Neocognitron LeNet-5 architecture Alex krizhevsky , Hinton LeNet-5 (1998) • Convolutional networks Improved by Yann Lecun et.al • Classify handwritten digits D. Rumelhart, G. Hinton, R. Wiliams 1960 1970 1980 1990 2000 2010 2012 Perceptron XOR Problem Golden Age 1957 1969 1986 F. RosenblattM. Minsky, S. Papert • Adjustable weights • Weights are not learned • XOR problem is not linearly separble • Solution to nonlinearity separable problems • Big computation, local optima and overfiting CNN Breakthrough (2012) • By Alex Krizhevsky et al. • Winner of ILSVRC2012 by large marginDark Age (AI winter) Back propagation (1981) • Train multiple layers Multi-layer Perceptron (1986) 1950 Neocognitron (1959) • Hubel & Wiesel • by cascading from S- Cell to C-Cell
- 12. Convolutional Neural Networks (CNN) A type of feed-forward neural network Inspired by biological process Weight sharing (convolution) + Subsampling (pooling) Reducing the number of parameters (Reduce over-fitting) Translation invariance Input 28 × 28 Feature maps 4@24 × 24 Feature maps 4@8 × 8 Feature maps 8@4 × 4 Feature maps 8@2 × 2 Feature maps 8 ⋅ 2 ⋅ 2 × 1 Output 10 × 1 Convolution layer Max-pooling layer Convolution layer Max-pooling layer Reshape Linear layer [LeCun, 1998]
- 15. Feature Extraction by CNN 15
- 16. Computational map 16 Dense Few Millions #ofVariables Completeness of Data (Sampling)Sparse More • Compute • Data • Storage • Bandwidth Computationally Intractable SpaceDeep Learning Neural Nets Statistical Analysis Algorithms, Closed Form Solutions Expert Systems Insufficient data for analysis [Un]supervised Learning Models Intuition Log scale Log scale
- 17. Machine Learning vs Deep Learning — Scale Matters — Millions to Billions of parameters — Data Matters — Regularize using more data — Productivity Matters — It’s simple, so we can make tools Data & Compute Accuracy Deep Learning Many previous methods Deep learning is most useful for large problems Modified by Nvidia DLI
- 18. BuildingAI : Data Data is the new source code 18NVidia
- 19. MEDICINE 19
- 20. AI + Healthcare Market Size Healthcare AI Market Size: 10B USD@ 2015 -> 67B USD(7조원)@2021, CAGR : 42%/y* *Prost and Sullivan, ** WHO, ***Variant Market Research, ****KHDI Global healthcare expenditure** Global IT healthcare market** AI healthcare market @ SouthKorea****Ads 20B USD <<< Healthcare 9,500B USD (50x)
- 21. AI Medical Device 21 • Verily@Google: Normal vs Abn, Anti-aging, Life Prolongation • IBM: Truven, 40B USD • Apple: GSK, EMR +iPhone Healthcare Platform • Facebook: Incurable dx, Human cell atlas, 5000M USD • Zebra Medical Vision • AI medical imaging Dx : 1st place of investment • Medical imaging reading cloud service/1 USD
- 22. AI Medical Device Cleared in FDA@2018 22 https://twitter.com/erictopol/status/1028642832171458563
- 23. 1st AI Doctor; IDx-DR the first device to provides a screeningdecision without the need for a clinician to also interpret the image or results IDx-DR >=mildretinopathy; refer to an eye care professional < mildretinopathy rescreen in 12 months 23
- 24. Requirements DICOM/ITK/HL7/FHIR library Physics of Imagingmodalities : X- ray, CT, MRI, US Domain knowledge of EMR, EHR, PHR, HIS Computer Science : Preprocessing, post-processing DL algorithms Public / Private Datasets Medical experts 24
- 25. Challenges Deep Learning Needs Why Data Scientists New computing model Latest Algorithms Rapidly evolving Fast Training Impossible -> Practical Deployment Platforms Must be available everywhere
- 26. Opportunities for AI inmedicine Ball, John, Erin Balogh, and Bryan T. Miller, eds. Improving diagnosis in health care. National Academies Press, 2015.
- 27. Opportunities for AI inmedicine Ball, John, Erin Balogh, and Bryan T. Miller, eds. Improving diagnosis in health care. National Academies Press, 2015.
- 28. E.g., AI Application in Medical Imaging Almost all aspects Image transformation Lesion segmentation Lesion classification Lesion detection Findingsimilar cases Assistance of interpretation
- 29. TASKS Image SegmentationObject Detection Image Classification + Localization Image Classification (inspired by a slide found in cs231n lecture from Stanford University) Nvidia DLI Education Materials
- 30. Clinical Unmet Needs andSolutions 1. Augmentation, curriculum learning, one / multi-shot learning to solve diseases’ imbalance, rare or a small number of dataset 2. Efficient anonymization, curation, and smart labeling for cheap labeling 3. Domain adaptation or Image normalization to overcome differences of multi-center trials 4. Interpretability, explainablity and visualization to mitigate black box property 5. Uncertainty of medical data, and artificial intelligence prediction 6. Reproducibility study of deep learning using repeatedly scanned images 7. Generative model 8. Novelty (Anomaly) detection under unsupervised learning for human decision in later 9. Big data PACS and Content based image retrieval 10. Deep radiomics and deep survival 11. Clinical decision support system 12. Develop physics-induced machine learning with well-known physics and medical laws 13. Active and Robust learning to overcome adversarial attack and input changes 14. Applications on 3D Printing, AR/VR, Robotics, Optic devices, Etc 31
- 31. 1. Perlin noise augmentation Infinity augmentation strategy Perlin noise by complexity theory 32Bae HJ, Kim N, Sci Report 2018
- 32. Perlin noise Randomnoise is too harsh to be natural The Perlin noise : by simply addingup noisy functions at a range of different scales. 33 Apply smooth filter to the interpolated noise 121 242 121
- 34. GAN augmentation to imbalanceddataset 35Hojjat Salehinejad, et al., ICASSP 2018
- 35. GAN synthetic images X-ray, Pathology classification task DCGAN, PGGAN 2-3% accuracy drop 36https://arxiv.org/ftp/arxiv/papers/1904/1904.08688.pdf Examining the Capability of GANs to Replace Real Biomedical Images in Classification Models Training
- 36. 1. Variable Patterns of Chest Abnormalities
- 37. Chest X-ray 38 약7000 1,053 944 (1,092) 1,189 (1,742) 550 (721) 853 (1,358) 280 (538) 998 (1,277) 1,361 (1,661) 1,009 (2,103) 589 (622) 944 (981) 604 (932) 404 (none) Normal 1089 121 (145) Pair_set x2 27 (36) Pair_set x2 12 (23) Pair_set x2 62 (74) Pair_set x2 21 (22) Pair_set x2 Reproducibility 144 Pair_set x2 Abnormal 1324 Other abnormal (500images) Nodule: 32(40) Consoidation:11(11) Interstitial Opacity: 1(1) Pleural Effusion: 114(184) Normal Nodule Consolidation Interstitial Opacity Pleural Effusion Pneumothora x TB active/advanced Cardiomegaly None None ASAN SNUBH Normal : 약7,000 Abnormal : 5,652 (6,890) Normal : 1,053 Abnormal : 4,993 (7,461) *추가 normal 데이터 (25만장)
- 38. Weakly Supervised Learning+ Class Activation Map 1. Curriculum learning : Chest PA Lee SM, Seo JB, Radiology, AMC
- 39. Curriculum learning 40 ❑ Two-steps curriculum learning • Step 1) training lesion-specified patch images • Step 2) fine-tuning with entire images NM ND Resnet-50 (pre-trained on imagenet dataset) CS IO PE PT NM ND CS IO PE PT transfer 1) 2) 512 512 1024 1024 NM : Normal ND : Nodule CS : Consolidation IO : Interstitial opacity PE : Pleural effusion PT : Pneumothorax
- 40. Preprocessing 41 ❑ Abnormal patch images • From abnormal region pneumothorax nodule
- 41. Comparison Results 42 ❑ Curriculum learning vs. Baseline (w/o curriculum) • Both models converged well. • Curriculum learning showed better result, more stable and shorter training time. Baseline Epoch 1257 Loss 0.163 Acc AMC: 96.1 SNUBH: 92.7 Curriculum learning Epoch 199 Loss 0.140 Acc AMC: 97.2 SNUBH: 94.2 Loss on tuning set Epoch Loss
- 42. 1. Curriculum learning : Endoscopy of Colonic Polyp Polyp detection in endoscopy (video) Close-up shot on polyp of interests Pathological Diagnosis (multi-modal) • https://arxiv.org/abs/1512.03385 “Deep Residual Learning for Image Recognition” • https://arxiv.org/abs/1512.04150 “Learning Deep Features for Discriminative Localization” AMC Digestive Internal Medicine Byeon JS Procedure of Detection and Classification of colon polyps CAM result of classification Colonoscopy Automation Histopathology Hyperplastic polyp : HP 65 166 Sessile serrated adenoma : SSA 101 Benign adenoma : BA 521 655Mucosal or superficial submucosal cancer : MSMC 134 Deep submucosal cancer : DSMC 101 101
- 43. Curriculum learning : Modeling
- 44. Further 46 Integration of CT virtual colonoscopy Depth generation from CT virtual colonoscopy endoscopyCT virtual colonoscopy
- 45. 2. Smart Labeling : Concept Manual label Smart label Smart label Manual label 200 hours Per 100 cases 500 hours Per 100 cases Manual label SW label Label Correction Total Dataset Manual Labeling Results Sub-dataset Manual Labeling Training model DL labeling Label correction Results Active learning Additional dataset
- 46. 2. Smart Labeling : Ex Manual label Smart label 50-60% faster Abdominal multi-organ segmentation (kidney) • Parenchyma, renal cell carcinoma, artery, vein, ureter • 4-5 hours per case by expert manual segmentation Abdominal multi-organ segmentation (kidney) • DL model segment images at first • Expert corrects segmentation result of DL models • 1-2 hours per case
- 47. 2. Smart Labeling; Comparison of AI Generated Dataset Kim YG, Kim N, et al, Sci Report 2019; Go HJ, Pathology, AMC, Sci Report (2018) Peri-tubular capillary (PTC) counting
- 48. 2. Smart Labeling : Surgical Imaging AI assisted labeling with semantic segmentation from medical images Pancreas Stomach, kidney, liver, etc Cervical spine (Med Phys Revision) MICCAI 2018 Segmentation Decathlon 2nd place Semantic segmentation of AI saving time • No human interaction • 10 msec/slice more accurate • More reproducible • More robust saving cost • Fast segmentation • 2~3 times faster after correction Tip! Maxillary sinus, mandible, mandibular canal (RSNA 2018) Glioblastoma (GBM) Lung lobe (JDI 2019) Breast (RSNA 2017) Airway (MIA 2019)
- 49. 2. Smart Labeling; Medical Segmentation Decathlon 51 10 organs; 52 labels MSD challenge, MICCAI 2018 Cascaded U-Net 2nd Place
- 51. Loss 53https://www.profillic.com/paper/arxiv:1812.07032 Volume overlay loss : DSC Boundary loss
- 52. Human Segmentation Human Segmentation AI_ 1st Test AI_ 1st Test AI_ 2st Test AI_ 2st Test AI_ 3rd Test AI_ 3rd Test rebuilding ground truth increasing data 0.88 2. Smart Labeling; Active Learning DICE 0.91 0.95 10 sec TH Kim, KH Lee, Kim N, Sci Report Submitted; Kyoung YS, Nephrology, AMC; Submitted to Sci Report
- 53. X-ray Line Tracing 55 The results of Cardiomegaly border line segmentation Yang DH, , Radiology AMC; Park JW, Kualldam Dental Hospital
- 54. Cascade landmark detection 4 13 10 56 4±3 pixels 13 ±10 pixels 10 ±12 pixels 9±12 pixels 18 ±25 pixels 12 ±20 pixels 17 ±23 pixels
- 55. 3. Domain Adaptation : Pancreatic Cancer ▪ Pancreas segmentation using domain adaptation Multi-center datasets: AMC and NIH ✓AMC: 220 patients ✓NIH: 82 patients Kim HJ, Radiology, AMC
- 56. 3. Domain Adaptation : Pancreatic Cancer ▪ Pancreas segmentation using domain adaptation Domain adaptation Kim HJ, Radiology, AMC
- 57. 3. Domain Adaptation : Pancreatic Cancer ▪ Pancreas segmentation using domain adaptation Domain adaptation Source Target Dice NIH NIH 0.7601 AMC 0.5833 AMC (baseline) AMC 0.8466 NIH 0.4649 AMC (with DA) AMC 0.8284 NIH 0.6770 Kim HJ, Radiology, AMC
- 58. 3. Normalization : Super-Resolution Undersampling Image Fully- Reconstructed Image Generator
- 59. 3. Image Normalization : CT Kernel Conversion 61D Eun, N Kim, Sci Report Revision, KJR 2018, Radiology 2019 Network architecture Conversion Baseline EDSR Ours (Single) Ours (Multi) B10f – B30f 15.67 / 0.9867 5.64 / 0.9970 4.72 / 0.9976 4.28 / 0.9978 B10f – B50f 56.02 / 0.8345 30.87 / 0.9329 29.02 / 0.9449 27.24 / 0.9458 B10f – B70f 114.49 / 0.6277 78.16 / 0.8103 75.57 / 0.8293 71.96 / 0.8270 B70f – B50f 63.33 / 0.8718 11.02 / 0.9933 9.31 / 0.9950 8.77 / 0.9949 B70f – B30f 102.99 / 0.6200 13.45 / 0.9885 10.46 / 0.9928 9.22 / 0.9931 B70f – B10f 114.49 / 0.6277 11.99 / 0.9907 10.05 / 0.9936 8.64 / 0.9939 To overcome CT vendor differences Iterative and progressive learning
- 60. Dataset : Compressive Sensing AMC Radiology Jung SC
- 61. Denoising network Yang, Q., Yan, P., Kalra, M. K., & Wang, G. (2017). CT image denoising with perceptive deep neural networks. arXiv preprint arXiv:1702.07019.
- 62. Denoising : Compressive Sensing 64 PredictOriginal Difference CS 6.84 -> CS5.56 Problems : Output Image Blurring AMC Radiology Jung SC
- 63. Img-to-Img ; Pix2Pix Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, et al, 2016 GAN Loss + L1Loss
- 64. Pix2Pix Network 2D ConvLayer Batch Normalization ReLU 2D ConvLayer Batch Normalization X 9 Generator Network : ResNet 9 blocks
- 65. Super Resolution : Compressive Sensing CS 6.84 -> CS5.56 PredictOriginal Difference Training Result with ~3000 training images AMC Radiology Jung SC
- 66. CycleGAN Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, et al, ICCV 2017, https://arxiv.org/pdf/1703.10593.pdf
- 67. CycleGAN Cycle-concistency loss Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, et al, ICCV 2017, https://arxiv.org/pdf/1703.10593.pdf Cycle loss Total loss
- 68. 70
- 69. —> They use same network both for Generator and Discriminator Same Generator and Discriminator
- 70. 4.Explainable AI – What Are We Trying To Do? 72 Today © Spin South West • Why did you do that? • Why not something else? • When do you succeed? • When do you fail? • When can I trust you? • How do I correct an error?Training Data Tomorrow Learned Function © Spin South West Output Userwith a Task • I understand why • I understand why not • I know when you’ll succeed • I know when you’ll fail • I know when to trust you • I know why you erredTraining Data Explainable Model Explanation Interface Userwith a Task DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) © UniversityOf Toronto Learning Process This is a cat (p = .93) © UniversityOf Toronto New Learning Process This is a cat: •It has fur, whiskers, and claws. •It has this feature:
- 71. 4. Interpretability vs Accuracy 73 PredictionAccuracy Explainability Learning Techniques (today) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs SRL CRFs Random Forests Graphical Models Explainability (notional) DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
- 72. DARPA XAI (eXplanable AI) 74 New Approach Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance PredictionAccuracy Explainability Learning Techniques (today) Explainability (notional) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs Model Induction Techniques to infer an explainable model from any model as a black box Deep Explanation Modified deep learning techniques to learn explainable features SRL Interpretable Models Techniques to learn more structured, interpretable, causal models CRFs Random Forests Graphical Models DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
- 73. 4. Visualization : Machine Operable, Human Readable Visual attention Class Activation Map (CAM) Category – feature mapping Sparsity and diversity 75
- 74. Visualization : Bias representation in CNN 76 Zhang et al., 2018b
- 75. Weakly Supervised Learning + Class Activation Map 4. Visualization : Chest PA @ AMC Lee SM, Seo JB, Radiology, AMC
- 76. Interesting Case 78 Normal Cardiomegaly Ground Truth Prediction Surgical Wires
- 77. Image Pattern Classification on DILD HRCT (a) Consolidation (b) Emphysema (c) Normal (d) Ground-glass opacity (e) Honeycombing (f) Reticular opacity Image Feature Extraction Machine Learning Training dataset Testing set Accuracy (%) SVM Classifiers GE GE 92.02 ± 0.56 Siemens Siemens 89.92 ± 0.36 GE+Siemens GE+Siemens 89.73 ± 0.43 N : 100 ROIs in each class of GE, Siemens CT Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 78. CNN networks 80 Input Image Conv. Layer #1 Conv. Layer #2 Conv. Layer#3 Conv. Layer#4 FC #1 FC #2 Local Response Normalization Dropout 100 6 20 20 19 19 64 64 64 64 9 9 4 4 4 4 4 4 4 4 3 3 3 3 3 3 Max Pooling Max Pooling Local Response Normalization • Two basic data replication ✓ random cropping and flipping: 20 x 20 pixels of ROI patches (top left, top right, center, bottom left, bottom right) & horizontally flipped ✓ mean centering and random noising: random noise of Gaussian distribution with mean zero and standard deviation Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 79. 4. Image Pattern Classification on DILD HRCT (a) Consolidation (b) Emphysema (c) Normal (d) Ground-glass opacity (e) Honeycombing (f) Reticular opacity DEEP Learning Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 80. 4. t-SNE Map ; SVM vs CNN 82 O = GE; X = SiemensSVM CNN Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 81. Knowledge hierarchy hidden in conv-layers 83Zhang et al., 2018a
- 82. Decision tree Disentangled representation 84Zhang et al., 2018a
- 83. 5. Uncertainty Uncertainty of training data In clinical situation, it is common Deep Bayesian Modeling Uncertainty of classification/prediction of Machine Learning 85
- 84. Kendal (2017) 5. Aleatoric & epistemic uncertainties 86 By chance System uncertainty
- 85. 6. Reproducibility Test-retest is one of most important issues of biomarker Multiple scanning within similar date Evaluate reliability of AI/Deep learning Chest PA (Nodule) Nodule Size on Chest PA with YOLO – 56% : Variability of Size Marker – Chest PA 50 pairs DILD CT 87
- 86. 6. Chest PA : 5 Disease Patterns ▪ Curriculum learning for multi-label classification Results Lee SM, Seo JB, Radiology, AMC
- 87. 6. Chest PA : Extra-validation ▪ Extra validation using multi-center datasets Results AMC Densenet 201 True Lesion Normal Pred. Lesion 690 44 Normal 30 1170 AMC Resnet 152 True Lesion Normal Pred. Lesion 668 61 Normal 52 1153 Sensitivity: 95.8%, Specificity: 96.4% Accuracy: 96.2% Sensitivity: 92.8%, Specificity: 95.0% Accuracy: 94.2% SNUBH Densenet 201 True Lesion Normal Pred. Lesion 99 14 Normal 1 86 SNUBH Resnet 152 True Lesion Normal Pred. Lesion 100 38 Normal 0 62 Sensitivity: 99%, Specificity: 86% Accuracy: 92.5% Sensitivity: 100%, Specificity: 62% Accuracy: 81.0% Lee SM, Seo JB, Radiology, AMC
- 88. 6. Chest PA Reproducibility : Nodule Detection ▪ Reproducibility analysis on CNN-based detection Comparison of various networks ✓YOLO v2 ✓VUNO-net Nodule (N = 121) 1st P N 2nd P 94 3 N 15 9 ✓Faster RCNN ✓Mask RCNN Nodule (N = 121) 1st P N 2nd P 90 8 N 11 12 Nodule (N = 121) 1st P N 2nd P 90 3 N 13 15 Nodule (N = 121) 1st P N 2nd P 97 4 N 14 6 Lee SM, Seo JB, Radiology, AMC Nodule (N = 121) 1st P N 2nd P 116 1 N 2 2 ✓Reader 2 Nodule (N = 121) 1st P N 2nd P 111 3 N 4 3 ✓Reader 1
- 89. 7. GAN 91 Kim, Namju’s Slideshare
- 90. PGGAN; Chest PA Xray 92 Progressive growing GAN (PGGAN) https://arxiv.org/abs/1710.10196 Tero Karras, et al Progressive Growing of GANs for Improved Quality, Stability, and Variation,
- 91. Normal only Normal+abnormal PGGAN; Chest PA Xray RSNA 2019
- 93. Human validation in GAN Chest PA Xray Chest CT RSNA 2019
- 95. TL-GAN 97
- 96. TL-GAN 98
- 97. Conditional generators GAN Architecture2 [3] Jun-Yan Zhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV, 2017 [4] Xi Chen et al., InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, NIPS, 2016 InfoGAN4 Style transfer Networks CycleGAN3 Latent Space of GAN [2] “An intuitive introduction to Generative Adversarial Networks (GANs)“, https://medium.freecodecamp.org/an-intuitive-introduction-to-generative-adversarial-networks-gans-7a2264a81394 Disentangling Latent Space
- 98. [5] TL-GAN: transparent latent-space GAN, https://github.com/SummitKwan/transparent_latent_gan PGGAN6 CNN Classifier7 [6] Tero Karras et al., Progressive Growing of GANs for Improved Quality, Stability, and Variation, ICLR, 2018 [7] Beomhee Park et al., A Curriculum Learning Strategy to Enhance the Accuracy of Classification of Various Lesions in Chest-PA X-ray Screening for Pulmonary Abnormalities, Scientific Reports, submitted Proposed Model Architecture5
- 99. CNN Classifier7 based on supervised learning [7] Beomhee et al., A Curriculum Learning Strategy to Enhance the Accuracy of Classification of Various Lesions in Chest-PA X-ray Screening for Pulmonary Abnormalities Dataset Chest PA X-ray images from 6069 healthy subjects and 3417 patients at AMC, 1035 healthy subjects and 4404 patients at SNUBH Task To detect 5 pulmonary abnormalities in Chest PA X-ray(CXR) :Nodules(ND), consolidation (CS), interstitial opacity (IO), pleural effusion (PE), and pneumothorax (PT) AMC SNUBH Sensitivity 85.4% 97.9% Specificity 99.8% 100.0% AUC* 0.947 0.983 Feature Extractor Classifier *AUC: Area Under the Curve Feature Extraction with Classifier
- 100. Vector Orthogonalization t-SNE projection of latent space representations on MNIST dataset8 HouseHolder QR Factorization9 Linear Regression on Latent Vectors [8] “THE MNIST DATABASE of handwritten digits”, http://yann.lecun.com/exdb/mnist/ [9] “QR decomposition using Householder transformations”, https://www.keithlantz.net/2012/05/qr-decomposition-using-householder-transformations/ Finding Feature Axes
- 101. Dataset Tree
- 102. Consolidation Interstitial opacity Pleural effusion Synthetic Image Visual Scoring Results Reader: An Expert Thoracic Radiologist Severity: Mild, Moderate, and Severe Visual Scoring Likelihood CS Normal → Moderate 0 → 0.97 IO Normal → Moderate 0 → 0.89 PE Normal → Severe 0 → 0.99 Disentangled Feature Axes
- 103. Pleural effusion (Left) Synthetic Image Pleural effusion (Right) Pleural effusion (Both) Latent Vector Arithmetic – Sub-Axes
- 104. 7. Generative Adversarial Network (GAN) Deep Convolutional Generative Adversarial Networks (DCGAN) Rotations are linear in latent space Bedroom generation Arithmetic on faces
- 106. 8. Novelty (Untrained catergory) In clinical situation Novelty is everywhere, especially supervised learning Rare diseases, but well known to medical doctors Hard to training How to determine novel (untrained) category Unsupervised learning Semi-unsupervised learning Normal vs abnormal Abnormality Detection 108
- 108. 8. Anomaly Detection 110Courtesy of Kang, Min-Guk
- 109. 8. Anomaly Detection 111Courtesy of Kang, Min-Guk
- 110. 8. Anomaly Detection 112Courtesy of Kang, Min-Guk
- 111. 114 Change latent space to generate normal chest X-ray
- 112. query image: abnormal similar image: normal (fake) difference Dx detection by AnoGAN without labeling RSNA 2019
- 113. Other Applications with Anomaly Detection 116 Liver Meta Brain Hemorrhage RSNA 2019
- 116. 9. Big data PACS platform Bigdata PACS Quantifying every data Transforming PACS into big data platform Advanced processing service (APS) Lung nodules – Detection : location – Segmentation : boundary drawing » Quantifying size, long/short axis, volume, etc 119 Zebra Medical Vision + Google, EnvoyAI+TeraRecon, Arterys, Fujifilm Nuance + Partners HC, Philips, Siemens, etc
- 117. Proposed Processing with NO Human Interaction 120 Deep Learning Support 10 sec 80 sec 3 min 1 min 10 sec 10 min RSNA 2018
- 118. Fully Automated COPD Analysis No Human Interaction LAA Analysis / LAA Size Analysis Emphysema Airway Vessel IN/EX Lung Seg. Lobe Seg. Airway Seg. Airway Graph Lumen / Wall Measure Accept / Reject Lumen / Wall Analysis Pulmo Vessel Seg. Artery / Vein Separation Vessel Analysis Air Trapping / IN/EX Parametric MapInspiration / Expiration Registration ~10 sec ~20 sec ~ 3min ~ 10 sec ~ 10 sec ~20 sec ~5 min ~ 20 min Use of Deep Learning 10 sec 80 sec 3 min 10 sec 1 min 20 sec 5 min 10 min 3 min RSNA 2018
- 119. Validation of Proposed Workflow 0.96 92% 94% RSNA 2018
- 120. Big-Data PACS for Quantitative COPD Reading ❖ Auto DICOM retrieve from legacy PACS. ❖ Auto processing supported by Deep-learning. ❖ Stored all the quantification values for every COPD patient ❖ Dashboard showing the overall statistics. ❖ Query by quantification. ❖ Similar case can be easily queried by quantification values when reading a patient’s chest CT images. RSNA 2018
- 121. 124 9. CBIR for Medical Images
- 122. Deep Lesion 32000 CT images, 4400 unique px Summers group @ NIH 125
- 123. Content Based Image Retrieval in Chest PA Xray RSNA 2019
- 124. Content Based Image Retrieval in Chest PA Xray RSNA 2019
- 125. Content Based Image Retrieval in DILD CT DILD Classification using SVM Normal Honeycombing Ground -glass opacity Consolidation Emphysema Reticular opacity 𝑁. 𝐿 . 𝑣𝑜𝑥𝑒𝑙𝑠 𝑙𝑢𝑛𝑔 𝑣𝑜𝑥𝑒𝑙𝑠 𝐻. 𝐶. 𝑣𝑜𝑥𝑒𝑙𝑠 𝑙𝑢𝑛𝑔 𝑣𝑜𝑥𝑒𝑙𝑠 𝑅. 𝑂. 𝑣𝑜𝑥𝑒𝑙𝑠 𝑙𝑢𝑛𝑔 𝑣𝑜𝑥𝑒𝑙𝑠 : 4 x 4 x 4 x 6 = 384 features Red : IPF Blue : COP Green : NSIP 76.7% and 81.7% in Top3, and Top5 Same patient with no change CTs 3.88 ± 0.96 5-scale visual scoring by two radiologists RSNA 2019
- 126. Content Based Image Retrieval in COPD CT COPD feature extraction for CBIR Top 1; similarity score 5/5 Top 2; similarity score 4/4 Top 3; similarity score 4/4 Top 4; similarity score 4/4 Top 5; similarity score 2/3 Query case CBIR retrieval for similar chest CT with COPD 60.0% and 68.0% in Top3, and Top5 Same patient with no change CTs 3.55±0.98 5-scale visual scoring by two radiologists RSNA 2019
- 129. Shape and Texture CT Features 133
- 130. CT Image Texture Features of NSCLC 134 Unsupervised Hierarchical Clustering
- 131. GBM from PCNSL (primary central nervous system lymphoma) 136 Park JE, Kim HS, et al, Sci Report 2019
- 132. GBM from PCNSL (primary central nervous system lymphoma) 137 Park JE, Kim HS, et al, Sci Report 2019
- 133. COPD Deep Radiomics : Survival Prediction Method KOLD cohort Random survival forest Extraction of deep features Design a CNN model for classifying 5- year survival Each slice was trained separately, and used for extracting representative deep features of it 138 Ishwaran, H., et al., “Random survival forests,” arXiv:0811.1645. Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D FCN(1,024) 5-year survival CT slice (a) A patient survived less than 5-year. (b) A patient survived more than 5-year. Data C-index Deep radiomics features (A1+A2+A3+A4+A5+A6) 0.8412 Demographic information 0.7688 PFT 0.7498 Handcrafted features 0.5994 Accuracy of survival prediction CAM of survival prediction CNN architecture for Deep radiomics
- 134. Extra-Validation Malaysian 111→102 COPD patients Death: 18 / censored: 0 / surviving: 84 11 CT slices based on preselected anatomic landmarks 3 images in the coronal plane – Vertebral body, center of tracheal carina and superior vena cava 2 images in the sagittal plane – Center of left and right lung 6 images in the axial plane – Upper 2 slices and lower 2 slices at intervals of 2 cm on the center of tracheal carina 139 Mortality data (based on 'copddeath) The median follow-up time as of 31 June 2017 was 1000 days (2 years and 9 months) (range, 60 to 1400 days). Eighteen out of 112 (16%) died during this time. Death causes were as follows: COPD (n=8), pneumonia (n=4), congestive cardiac failure/ coronary artery disease (n=2), lung carcinoma (n=1), stroke (n=1), motor traffic accident (n=1) and intestinal obstruction (n=1). The final two causes are considered as non-COPD associated causes. As such, 16 patients died of COPD associated death (14.2%). AMC Radiology, Seo JB
- 135. KOLD vs. Malaysian Age KOLD Malaysian Survival (days) 140
- 136. Result Interval validation (5-fold cross validation) External validation (Malaysian) A1 0.6714 (0.6371, 0.7057) 0.6656 A2 0.7462 (0.7178, 0.7746) 0.6452 A3 0.7229 (0.6542, 0.7916) 0.6062 A4 0.6818 (0.6468, 0.7168) 0.6019 A5 0.7409 (0.7030, 0.7789) 0.6217 A6 0.7076 (0.6879, 0.7273) 0.6830 A2 + + A5 0.7587 (0.7179, 0.7995) 0.6774 A2 + A3 + + A5 0.7819 (0.7289, 0.8348) 0.6650 A2 + A3 + + A5 + A6 0.7603 (0.7065, 0.8141) 0.6768 A2 + A3 + A4 + A5 + A6 0.7532 (0.6956, 0.8108) 0.6749 A1 + A2 + A3 + A4 + A5 + A6 0.7288 (0.6759, 0.7817) 0.6582 141
- 137. 11. CDSS on Liver Cancer Kim KM, GI, AMC Clinical data (20 variables) RFA / Operation Or not Operation Classifier 1 Prediction B TACE Or not TACL+RT Prediction C Classifier 2 Classifier 3 sorafenib Others Or not Or not RFA Prediction A LT Classifier 4 Prediction G Prediction D Supportive care Classifier 5 Classifier 6Prediction E Prediction F 147
- 138. CDSS on Liver Cancer Treatment recommender system for liver cancer Patient’s baseline data: 20 variables 1,000 patients Hierarchical decision modeling Survival prediction Cox hazard regression, random forest survival 148 Survival Curve on Tx of Liver Cancer Kim KM, GI, AMC
- 140. 12. CNN for Steady Flow Prediction 150 CNN Prediction LBM 𝑦=f’(x) https://www.biorxiv.org/content/biorxiv/early/2017/12/30/240317.full.pdf
- 141. 12. CNN for Steady Flow Prediction 151 𝑦=f’(x)
- 142. 14. Surgical Robot with AI Will robot steal surgeons’ job? NO. Will robot CHANGE surgeons’ job? It may... Will robot and SUPER COMPUTER steal surgeons’ job? …
- 143. 14. Simple Mastoidectomy Surgery Video Simple Mastoidectomy Surgery Video Understanding Surgical Step Understanding 9 Steps 153 Example AMC ENT Jung JW Classification algorithm of surgical stage by learning video data 중이염 동영상 내 수술도구(위), Henle recognized (below)
- 144. Colonoscopy CAD 154Byeon JS, AMC
- 145. Head&Neck CT of Trauma Px in ER Hemorrhage Px 813 Patient → 28667 Slices (512 X 512 X 3) Abnormal Slice : 11786 Train : 8186, Valid : 3000 Hemorrhage segmentation using 2D U- Net (DenseNet121) 155 RSNA 2019
- 146. Whole-body Trauma Px in ER Trauma Px 438 CT scans NL : 596 CT scans Abnormal Slice : 11786 Label 1) hemothorax 2) pneumothorax 3) hemomediastinum 4) pneumomediastinum 5) hemoperitoneum 6) pneumoperitoneum 7) hemoretroperitoneum 8) pneumoretroperitoneum 156 RSNA 2019
- 147. 14. Case Orchestration (Triage) 157 Reading ordering by AI for efficient reading Agfa, IBM, Philips, etc
- 149. 14. Hospital Workflow Improvement Intelligent Hand Hygiene Support @ Lucile Packard Children's Hospital at Stanford Intelligent ICU Clinical Pathway Support @ Stanford Hospital and Clinics @ Intermountain Healthcare Intelligent Senior Wellbeing Support @ Palo Alto Medical Foundation @ On Lok Senior Health Services @ Intermountain Healthcare ** Stanford AI-assisted Care Courtesy of Ha HS, VAIIM consulting, modified
- 150. 14. Beyond computer vision 160 Computer vision to discern clinical behaviors Bedside Computer Vision, Yeung S, Fei-Fei Li et al, NEJM 2018 April H Rosette *
- 151. Hand Hygiene Monitoring Clean Clips Images Clips Images Training 428 8,325 444 11,988 Validation 46 871 43 1,047 Testing 10 340 8 516 Total 484 9,536 495 13,551 Test accuracy: 0.82
- 152. Hand Gesture 162
- 153. Capsule Net 163A Lee, N Kim, CMPB 2018
- 154. EAU 2015, AUA 2015, RSNA 2015, WIP for publication 14. Modeling for 3DP, AR, Robot
- 155. 3D Printings 165
- 156. The limitations and solutions of 3DP in medicine 166 Limitations Solutions Slow printing speed New 3DP for 100~1000 times faster Variations of clinical imaging protocols ; especially thick slice thickness Domain adaptation; Super resolution with DL Limitation of materials Develop new materials and mechanism Slow segmentation for specific anatomy Training DL for semantic segmentation New manpower Train new expertise with help of AI Physicians’ additional efforts Communication tools with AI Already doing well Find new killer apps with 3DP RSNA 2018
- 157. Semantic Segmentation 167 Medical Images Super Resolution Module Auto Design Module GAN 진단용 저해상도 영상 3D모델링용 고해상도 영상 train/test set Generator Discriminator Y/N Similar? Auto Segmentation Module 3D모델링용 고해상도 영상 Segmented mask FCN Network Labeling Pre-processor train/test set End-to-End learning Segmented mask Surgical planner Ref. tracker Border tracker Surgical planning Y/N meet Design spec..? DeepLearning RSNA 2018
- 158. AI Solution for 3DP 168 Data acquisition DL Modeling 3DP Confirm by an expert Confirm by M.D. Application Yes NoNo Yes Segmentation with human-AI interaction Communication with AI Low-resolution image for diagnosis High-resolution image for 3DP with super-resolution Semantic segmentation method Super-resolution method ▪ Super-resolution convert low-resolution images into high resolution images. ▪ The result images of super-resolution could be better 3D models. Gold Standard 1st training Segmentation enhancement with human-AI interaction 2nd training 3rd training AI based 3DP workflow Manual segmentation result Result of 2D U-net (about 30 sec / case) Result of 3D U-net (about 1 sec / case) RSNA 2018
- 160. VR/AR/MR
- 161. 3DP Housing for AI in Smart Phone 171
- 162. Artificial Intelligence (+Big Data) Will Redesign Healthcare 1. Artificial Intelligence (x) ->Augmented Intelligence (o) 2. Just Starting 3. Precision medicine / Mining medical records / Designing treatment plans 4. Getting the most out of in-person and online consultations / Health assistance and medication management / Open AI helping people make healthier choices and decisions 5. Assisting repetitive jobs 6. Drug discovery / Clinical trial Case Matching 7. Analyzing, redesigning a healthcare system http://medicalfuturist.com/ Aug, 2016
- 163. Job Openings @ MI2RL, AMC Post-doc research fellow, PhD Students, Researchers 173 AMC, UoU Seoul South Korea Contact to namkugkim@gmail.com
- 164. Collaborators Radiology Joon Beom Seo, SangMin LeeA,B, Dong Hyun, Yang, Hyung Jin Won, Ho Sung Kim, Seung Chai Jung, Ji Eun Park, So Jung Lee,Jeong Hyun Lee, Gilsun Hong Neurology Dong-Wha Kang, Chongsik Lee, Jaehong Lee, Sangbeom Jun, Misun Kwon, Beomjun Kim Cardiology Jaekwan Song, Jongmin Song, Young-Hak Kim Emergency Medicine Dong-Woo Seo Pathology Hyunjeong Go, Gyuheon Choi, Gyungyub Gong, Dong Eun Song Surgery Beom Seok Ko, JongHun Jeong, Songchuk Kim, Tae-Yon Sung Internal Medicine Jeongsik Byeon, Kang Mo Kim Anesthesiology Sung-Hoon Kim, Eun Ho Lee