Ccids 2019 cutting edges of ai technology in medicine
•
4 likes•933 views
"cutting edges of ai technology in medicine" presentation at CCIDS 2019 workshop @YUMC
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Ccids 2019 cutting edges of ai technology in medicine
Similar to Ccids 2019 cutting edges of ai technology in medicine (20)
More from Namkug Kim
More from Namkug Kim (16)
Recently uploaded
Recently uploaded (20)
Ccids 2019 cutting edges of ai technology in medicine
- 1. Cutting edges of AI technology in medicine Namkug Kim, PhD Medical Imaging & Intelligent Reality Lab. Convergence Medicine/Radiology, University of Ulsan College of Medicine Asan Medical Center South Korea
- 2. Conflict of Interests Establishments of Somansa Inc, CyberMed Inc, Clinical Imaging Solution Inc, and Anymedi Inc. Researches with LG Electronics, Coreline Soft Inc.,, Osstem Implant, CGBio, VUNO, Kakaobrain, AnyMedi Inc.
- 3. Slide share RAAI 2019 plenary talk : https://www.slideshare.net/namkugkim/raai-2019-clinical- unmet-needs-and-its-solutions-of-deep-learning-in- medicine3 3
- 4. Computational map 4 Dense Few Millions #ofVariables Completeness of Data (Sampling)Sparse More • Compute • Data • Storage • Bandwidth Computationally Intractable SpaceDeep Learning Neural Nets Statistical Analysis Algorithms, Closed Form Solutions Expert Systems Insufficient data for analysis [Un]supervised Learning Models Intuition Log scale Log scale
- 5. Machine Learning vs Deep Learning — Scale Matters — Millions to Billions of parameters — Data Matters — Regularize using more data — Productivity Matters — It’s simple, so we can make tools Data & Compute Accuracy Deep Learning Many previous methods Deep learning is most useful for large problems Modified by Nvidia DLI
- 6. BuildingAI : Data Data is the new source code 6NVidia
- 7. Requirements DICOM/ITK/HL7/FHIR library Physics of Imagingmodalities : X- ray, CT, MRI, US Domain knowledge of EMR, EHR, PHR, HIS Computer Science : Preprocessing, post-processing DL algorithms Public / Private Datasets Medical experts 7
- 8. Opportunities for AI inmedicine Ball, John, Erin Balogh, and Bryan T. Miller, eds. Improving diagnosis in health care. National Academies Press, 2015.
- 9. Opportunities for AI inmedicine Ball, John, Erin Balogh, and Bryan T. Miller, eds. Improving diagnosis in health care. National Academies Press, 2015.
- 10. TASKS Image SegmentationObject Detection Image Classification + Localization Image Classification (inspired by a slide found in cs231n lecture from Stanford University) Nvidia DLI Education Materials
- 11. Clinical Unmet Needs andSolutions 1. Augmentation, curriculum learning, one / multi-shot learning to solve diseases’ imbalance, rare or a small number of dataset 2. Efficient anonymization, curation, and smart labeling for cheap labeling 3. Domain adaptation or Image normalization to overcome differences of multi-center trials 4. Interpretability, explainablity and visualization to mitigate black box property 5. Uncertainty of medical data, and artificial intelligence prediction 6. Reproducibility/Followup study of deep learning using repeatedly scanned images 7. Generative models 8. Novelty (Anomaly) detection under unsupervised learning for human decision in later 9. Big data PACS and Content based image retrieval 10. Deep radiomics and deep survival 11. Clinical decision support system 12. Develop physics-induced machine learning with well-known physics and medical laws 13. Active and Robust learning to overcome adversarial attack and input changes 14. Applications on 3D Printing, AR/VR, Robotics, Optic devices, Etc 11
- 12. 1. Perlin noise augmentation Infinity augmentation strategy Perlin noise by complexity theory 12Bae HJ, Kim N, Sci Report 2018
- 13. Perlin noise Randomnoise is too harsh to be natural The Perlin noise : by simply addingup noisy functions at a range of different scales. 13 Apply smooth filter to the interpolated noise 121 242 121
- 14. 1. Solution to imbalanceddata 14
- 15. GAN augmentation 15Hojjat Salehinejad, et al., ICASSP 2018
- 16. GAN synthetic images X-ray, Pathology classification task DCGAN, PGGAN 2-3% accuracy drop 16https://arxiv.org/ftp/arxiv/papers/1904/1904.08688.pdf Examining the Capability of GANs to Replace Real Biomedical Images in Classification Models Training
- 17. Weakly Supervised Learning+ Class Activation Map 1. Curriculum learning : Chest PA Lee SM, Seo JB, Radiology, AMC, Submitted to Scientific Report
- 18. Curriculum learning 18 ❑ Two-steps curriculum learning • Step 1) training lesion-specified patch images • Step 2) fine-tuning with entire images NM ND Resnet-50 (pre-trained on imagenet dataset) CS IO PE PT NM ND CS IO PE PT transfer 1) 2) 512 512 1024 1024 NM : Normal ND : Nodule CS : Consolidation IO : Interstitial opacity PE : Pleural effusion PT : Pneumothorax Lee SM, Seo JB, Radiology, AMC, Submitted to Scientific Report
- 19. Preprocessing 19 ❑ Abnormal patch images • From abnormal region pneumothorax nodule Lee SM, Seo JB, Radiology, AMC, Submitted to Scientific Report
- 20. Comparison Results 20 ❑ Curriculum learning vs. Baseline (w/o curriculum) • Both models converged well. • Curriculum learning showed better result, more stable and shorter training time. Baseline Epoch 1257 Loss 0.163 Acc AMC: 96.1 SNUBH: 92.7 Curriculum learning Epoch 199 Loss 0.140 Acc AMC: 97.2 SNUBH: 94.2 Loss on tuning set Epoch Loss Lee SM, Seo JB, Radiology, AMC, Submitted to Scientific Report
- 21. 1. Curriculum learning : Endoscopy of Colonic Polyp Polyp detection in endoscopy (video) Close-up shot on polyp of interests Pathological Diagnosis (multi-modal) • https://arxiv.org/abs/1512.03385 “Deep Residual Learning for Image Recognition” • https://arxiv.org/abs/1512.04150 “Learning Deep Features for Discriminative Localization” AMC Digestive Internal Medicine Byeon JS, Submitted to Sci Reports Procedure of Detection and Classification of colon polyps CAM result of classification Colonoscopy Automation Histopathology Hyperplastic polyp : HP 65 166 Sessile serrated adenoma : SSA 101 Benign adenoma : BA 521 655Mucosal or superficial submucosal cancer : MSMC 134 Deep submucosal cancer : DSMC 101 101
- 22. Curriculum learning : Modeling AMC Digestive Internal Medicine Byeon JS, Submitted to Sci Reports
- 23. Label Noise healthy : 10137 + 1035 Patients : 3244 + 4404 nodule(ND), consolidation(CS), interstitial opacity(IO), pleural effusion(PE), and pneumothorax(PT) confirmed by radiologist with CT Y label noise : 0%, 1%, 2%, 4%, 8%, 16%, 32% resnet-50 24 Loglog plot plot RSNA 2019
- 24. 2. Smart Labeling : Concept Manual label Smart label Smart label Manual label 200 hours Per 100 cases 500 hours Per 100 cases Manual label SW label Label Correction Total Dataset Manual Labeling Results Sub-dataset Manual Labeling Training model DL labeling Label correction Results Active learning Additional dataset
- 25. 2. Smart Labeling : Ex Manual label Smart label 50-60% faster Abdominal multi-organ segmentation (kidney) • Parenchyma, renal cell carcinoma, artery, vein, ureter • 4-5 hours per case by expert manual segmentation Abdominal multi-organ segmentation (kidney) • DL model segment images at first • Expert corrects segmentation result of DL models • 1-2 hours per case
- 27. Human Segmentation Human Segmentation AI_ 1st Test AI_ 1st Test AI_ 2st Test AI_ 2st Test AI_ 3rd Test AI_ 3rd Test rebuilding ground truth increasing data 0.88 2. Smart Labeling; Active Learning DICE 0.91 0.95 10 sec TH Kim, KH Lee, Kim N, Sci Report Under-revision
- 28. 2. Smart Labeling : Surgical Imaging AI assisted labeling with semantic segmentation from medical images Pancreas Stomach, kidney, liver, etc Cervical spine (CMPB submitted) MICCAI 2018 Segmentation Decathlon 2nd place Semantic segmentation of AI saving time • No human interaction • 10 msec/slice more accurate • More reproducible • More robust saving cost • Fast segmentation • 2~3 times faster after correction Tip! Maxillary sinus, mandible, mandibular canal (RSNA 2018) Glioblastoma (Med Phys submitted) Lung lobe (JDI 2019) Breast (RSNA 2017) Airway (MIA 2019)
- 29. 2. Smart Labeling; Medical Segmentation Decathlon 31 10 organs; 52 labels MSD challenge, MICCAI 2018 Cascaded U-Net 2nd Place
- 30. 2. Smart Labeling; Comparison of AI Generated Dataset Kim YG, Kim N, et al, Sci Report 2019; Go HJ, Pathology, AMC, Sci Report (2018) Peri-tubular capillary (PTC) counting
- 31. X-ray Tracing Yang DH, , Radiology AMC; Park JW, Kualldam Dental Hospital, Ha Y, NeuroSurgery, YUMC 초기 계측선 생성의 예: Using basic U-Net model Name Mi n Mean Std ROI size Sella 0 0.2 0.2 256X256 Porion 0 0.8 0.8 256X256 Basion 0 0.6 0.8 256X256 Hinge 0.1 0.9 1.2 256X256 Pterygoid 0 0.9 1.0 512X512 Nasion 0 0.2 0.4 256X256 Orbitale 0.1 0.6 0.7 256X256 A 0.1 0.6 0.6 512X512 PM 0 0.5 0.5 256X256 Pogonion 0 0.7 0.6 256X256 B 0 0.7 0.8 256X256 Pns 0.2 0.8 0.9 256X256 Ans 0.1 0.4 0.7 256X256 R1 0.3 1.4 1.1 512X512
- 32. 3. Normalization : Super-Resolution Undersampling Image Fully- Reconstructed Image Generator
- 33. 3. Image Normalization : CT Kernel Conversion 38D Eun, N Kim, Sci Report Revision, KJR 2018, Radiology 2019 Network architecture Conversion Baseline EDSR Ours (Single) Ours (Multi) B10f – B30f 15.67 / 0.9867 5.64 / 0.9970 4.72 / 0.9976 4.28 / 0.9978 B10f – B50f 56.02 / 0.8345 30.87 / 0.9329 29.02 / 0.9449 27.24 / 0.9458 B10f – B70f 114.49 / 0.6277 78.16 / 0.8103 75.57 / 0.8293 71.96 / 0.8270 B70f – B50f 63.33 / 0.8718 11.02 / 0.9933 9.31 / 0.9950 8.77 / 0.9949 B70f – B30f 102.99 / 0.6200 13.45 / 0.9885 10.46 / 0.9928 9.22 / 0.9931 B70f – B10f 114.49 / 0.6277 11.99 / 0.9907 10.05 / 0.9936 8.64 / 0.9939 To overcome CT vendor differences Iterative and progressive learning
- 34. Dataset : Compressive Sensing AMC Radiology Jung SC
- 35. Denoising network Yang, Q., Yan, P., Kalra, M. K., & Wang, G. (2017). CT image denoising with perceptive deep neural networks. arXiv preprint arXiv:1702.07019. PredictOriginal Difference CS 6.84 -> CS5.56 Problems : Output Image Blurring
- 36. Img-to-Img ; Pix2Pix Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, et al, 2016 GAN Loss + L1Loss 2D ConvLayer Batch Normalization ReLU 2D ConvLayer Batch Normalization X 9 Generator Network : ResNet 9 blocks
- 37. Super Resolution : Compressive Sensing CS 6.84 -> CS5.56 PredictOriginal Difference Training Result with ~3000 training images AMC Radiology Jung SC
- 38. CycleGAN Cycle-concistency loss Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, et al, ICCV 2017, https://arxiv.org/pdf/1703.10593.pdf Cycle loss Total loss
- 39. 45
- 40. —> They use same network both for Generator and Discriminator Same Generator and Discriminator
- 41. 4.Explainable AI – What Are We Trying To Do? 47 Today © Spin South West • Why did you do that? • Why not something else? • When do you succeed? • When do you fail? • When can I trust you? • How do I correct an error?Training Data Tomorrow Learned Function © Spin South West Output User with a Task • I understand why • I understand why not • I know when you’ll succeed • I know when you’ll fail • I know when to trust you • I know why you erredTraining Data Explainable Model Explanation Interface Userwith a Task DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) © UniversityOf Toronto Learning Process This is a cat (p = .93) © UniversityOf Toronto New Learning Process This is a cat: •It has fur, whiskers, and claws. •It has this feature:
- 42. 4. Interpretability vs Accuracy 48 PredictionAccuracy Explainability Learning Techniques (today) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs SRL CRFs Random Forests Graphical Models Explainability (notional) DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
- 43. DARPA XAI (eXplanable AI) 49 New Approach Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance PredictionAccuracy Explainability Learning Techniques (today) Explainability (notional) Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning SVMs AOGs Bayesian Belief Nets Markov Models HBNs MLNs Model Induction Techniques to infer an explainable model from any model as a black box Deep Explanation Modified deep learning techniques to learn explainable features SRL Interpretable Models Techniques to learn more structured, interpretable, causal models CRFs Random Forests Graphical Models DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
- 44. 50Nature Medicine | VOL 24 | SEPTEMBER 2018 | 1342–1350
- 45. Knowledge hierarchy hidden in conv-layers 51Zhang et al., 2018a
- 47. Visualization : Bias representation in CNN 53 Zhang et al., 2018b
- 48. 4. Visualization : Machine Operable, Human Readable Visual attention Class Activation Map (CAM) Category – feature mapping Sparsity and diversity 54
- 49. 4. Variable Patterns of Chest Abnormalities
- 50. Chest X-ray 56 약7000 1,053 944 (1,092) 1,189 (1,742) 550 (721) 853 (1,358) 280 (538) 998 (1,277) 1,361 (1,661) 1,009 (2,103) 589 (622) 944 (981) 604 (932) 404 (none) Normal 1089 121 (145) Pair_set x2 27 (36) Pair_set x2 12 (23) Pair_set x2 62 (74) Pair_set x2 21 (22) Pair_set x2 Reproducibility 144 Pair_set x2 Abnormal 1324 Other abnormal (500images) Nodule: 32(40) Consoidation:11(11) Interstitial Opacity: 1(1) Pleural Effusion: 114(184) Normal Nodule Consolidation Interstitial Opacity Pleural Effusion Pneumothora x TB active/advanced Cardiomegaly None None ASAN SNUBH Normal : 약7,000 Abnormal : 5,652 (6,890) Normal : 1,053 Abnormal : 4,993 (7,461) *추가 normal 데이터 (25만장)
- 51. Interesting Case 58 Normal Cardiomegaly Ground Truth Prediction Surgical Wires
- 52. Image Pattern Classification on DILD HRCT (a) Consolidation (b) Emphysema (c) Normal (d) Ground-glass opacity (e) Honeycombing (f) Reticular opacity Image Feature Extraction Machine Learning Training dataset Testing set Accuracy (%) SVM Classifiers GE GE 92.02 ± 0.56 Siemens Siemens 89.92 ± 0.36 GE+Siemens GE+Siemens 89.73 ± 0.43 N : 100 ROIs in each class of GE, Siemens CT Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 53. CNN networks 60 Input Image Conv. Layer #1 Conv. Layer #2 Conv. Layer#3 Conv. Layer#4 FC #1 FC #2 Local Response Normalization Dropout 100 6 20 20 19 19 64 64 64 64 9 9 4 4 4 4 4 4 4 4 3 3 3 3 3 3 Max Pooling Max Pooling Local Response Normalization • Two basic data replication ✓ random cropping and flipping: 20 x 20 pixels of ROI patches (top left, top right, center, bottom left, bottom right) & horizontally flipped ✓ mean centering and random noising: random noise of Gaussian distribution with mean zero and standard deviation Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 54. 4. Image Pattern Classification on DILD HRCT (a) Consolidation (b) Emphysema (c) Normal (d) Ground-glass opacity (e) Honeycombing (f) Reticular opacity DEEP Learning Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 55. 4. t-SNE Map ; SVM vs CNN 62 O = GE; X = SiemensSVM CNN Kim GB, Kim N, et al, Journal of Digital Imaging, 2017
- 56. 5. Uncertainty Uncertainty of training data In clinical situation, it is common Deep Bayesian Modeling Uncertainty of classification/prediction of Machine Learning 63
- 57. Kendal (2017) 5. Aleatoric & epistemic uncertainties 64 By chance System uncertainty
- 59. 6. Reproducibility Test-retest is one of most important issues of biomarker Multiple scanning within similar date Evaluate reliability of AI/Deep learning Chest PA (Nodule) Nodule Size on Chest PA with YOLO – 56% : Variability of Size Marker – Chest PA 50 pairs DILD CT 66
- 60. 6. Chest PA : 5 Disease Patterns ▪ Curriculum learning for multi-label classification Results Lee SM, Seo JB, Radiology, AMC
- 61. 6. Chest PA : Extra-validation ▪ Extra validation using multi-center datasets Results AMC Densenet 201 True Lesion Normal Pred. Lesion 690 44 Normal 30 1170 AMC Resnet 152 True Lesion Normal Pred. Lesion 668 61 Normal 52 1153 Sensitivity: 95.8%, Specificity: 96.4% Accuracy: 96.2% Sensitivity: 92.8%, Specificity: 95.0% Accuracy: 94.2% SNUBH Densenet 201 True Lesion Normal Pred. Lesion 99 14 Normal 1 86 SNUBH Resnet 152 True Lesion Normal Pred. Lesion 100 38 Normal 0 62 Sensitivity: 99%, Specificity: 86% Accuracy: 92.5% Sensitivity: 100%, Specificity: 62% Accuracy: 81.0% Lee SM, Seo JB, Radiology, AMC
- 62. 6. Chest PA Reproducibility : Nodule Detection Comparison of various networks Lee SM, Seo JB, Radiology, AMC, Submitted to ER
- 63. 6. Chest PA Reproducibility : Abnormality Detection Comparison of various abnormality Lee SM, Seo JB, Radiology, AMC, Submitted to Sci Report
- 65. Autoregressive Models : Fully visible beliefnet J. Menick, et al., Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
- 66. Normalizing Flow Models : Change of variables : D. Kingma, et. al., Glow: Generative Flow with Invertible 1x1 Convolutions
- 67. Interpolations via Glow D. Kingma, et. al., Glow: Generative Flow with Invertible 1x1 Convolutions
- 68. 7. Generative Models 75 Kim, Namju’s Slideshare
- 69. GAN in Medicine Review Low Dose CT Denoising Segmentation Detection Medical Image Synthesis Reconstruction Classification Registration Others https://github.com/xinario/awesome-gan-for-medical-imaging
- 70. PGGAN; Chest PA Xray 77 Progressive growing GAN (PGGAN) https://arxiv.org/abs/1710.10196 Tero Karras, et al Progressive Growing of GANs for Improved Quality, Stability, and Variation,
- 71. Normal only Normal+abnormal PGGAN; Chest PA Xray RSNA 2019
- 73. Image turing test Chest PA Xray Chest CT RSNA 2019
- 74. Fake X-ray CAD 81 RSNA 2019
- 75. Conditional generators GAN Architecture2 [3] Jun-Yan Zhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV, 2017 [4] Xi Chen et al., InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, NIPS, 2016 InfoGAN4 Style transfer Networks CycleGAN3 Latent Space of GAN [2] “An intuitive introduction to Generative Adversarial Networks (GANs)“, https://medium.freecodecamp.org/an-intuitive-introduction-to-generative-adversarial-networks-gans-7a2264a81394 Disentangling Latent Space RSNA 2019
- 76. [5] TL-GAN: transparent latent-space GAN, https://github.com/SummitKwan/transparent_latent_gan PGGAN6 CNN Classifier7 [6] Tero Karras et al., Progressive Growing of GANs for Improved Quality, Stability, and Variation, ICLR, 2018 [7] Beomhee Park et al., A Curriculum Learning Strategy to Enhance the Accuracy of Classification of Various Lesions in Chest-PA X-ray Screening for Pulmonary Abnormalities, Scientific Reports, submitted Proposed Model Architecture5 RSNA 2019
- 77. CNN Classifier7 based on supervised learning [7] Beomhee et al., A Curriculum Learning Strategy to Enhance the Accuracy of Classification of Various Lesions in Chest-PA X-ray Screening for Pulmonary Abnormalities Dataset Chest PA X-ray images from 6069 healthy subjects and 3417 patients at AMC, 1035 healthy subjects and 4404 patients at SNUBH Task To detect 5 pulmonary abnormalities in Chest PA X-ray(CXR) :Nodules(ND), consolidation (CS), interstitial opacity (IO), pleural effusion (PE), and pneumothorax (PT) AMC SNUBH Sensitivity 85.4% 97.9% Specificity 99.8% 100.0% AUC* 0.947 0.983 Feature Extractor Classifier *AUC: Area Under the Curve Feature Extraction with Classifier RSNA 2019
- 78. Vector Orthogonalization t-SNE projection of latent space representations on MNIST dataset8 HouseHolder QR Factorization9 Linear Regression on Latent Vectors [8] “THE MNIST DATABASE of handwritten digits”, http://yann.lecun.com/exdb/mnist/ [9] “QR decomposition using Householder transformations”, https://www.keithlantz.net/2012/05/qr-decomposition-using-householder-transformations/ Finding Feature Axes RSNA 2019
- 80. Consolidation Interstitial opacity Pleural effusion Synthetic Image Visual Scoring Results Reader: An Expert Thoracic Radiologist Severity: Mild, Moderate, and Severe Visual Scoring Likelihood CS Normal → Moderate 0 → 0.97 IO Normal → Moderate 0 → 0.89 PE Normal → Severe 0 → 0.99 Disentangled Feature Axes RSNA 2019
- 81. Pleural effusion (Left) Synthetic Image Pleural effusion (Right) Pleural effusion (Both) Latent Vector Arithmetic – Sub-Axes RSNA 2019
- 83. 8. Novelty (Untrained catergory) In clinical situation Novelty is everywhere, especially supervised learning Rare diseases, but well known to medical doctors Hard to training How to determine novel (untrained) category Unsupervised learning Semi-unsupervised learning Normal vs abnormal Abnormality Detection 92
- 85. 8. Anomaly Detection 94Courtesy of Kang, Min-Guk
- 86. 98 Change latent space to generate normal chest X-ray
- 87. Anomaly detection with VAE in PET 99
- 88. query image: abnormal similar image: normal (fake) difference Dx detection by AnoGAN without labeling RSNA 2019
- 89. Other Applications with Anomaly Detection 101 Liver Meta Brain Hemorrhage RSNA 2019
- 92. 9. Big data PACS platform Bigdata PACS Quantifying every data Transforming PACS into big data platform Advanced processing service (APS) Lung nodules – Detection : location – Segmentation : boundary drawing » Quantifying size, long/short axis, volume, etc 106 Zebra Medical Vision + Google, EnvoyAI+TeraRecon, Arterys, Fujifilm Nuance + Partners HC, Philips, Siemens, etc RSNA 2018, 2019
- 93. Proposed Processing with NO Human Interaction 107 Deep Learning Support 10 sec 80 sec 3 min 1 min 10 sec 10 min RSNA 2018
- 94. Fully Automated COPD Analysis No Human Interaction LAA Analysis / LAA Size Analysis Emphysema Airway Vessel IN/EX Lung Seg. Lobe Seg. Airway Seg. Airway Graph Lumen / Wall Measure Accept / Reject Lumen / Wall Analysis Pulmo Vessel Seg. Artery / Vein Separation Vessel Analysis Air Trapping / IN/EX Parametric MapInspiration / Expiration Registration ~10 sec ~20 sec ~ 3min ~ 10 sec ~ 10 sec ~20 sec ~5 min ~ 20 min Use of Deep Learning 10 sec 80 sec 3 min 10 sec 1 min 20 sec 5 min 10 min 3 min RSNA 2018, 2019
- 95. Validation of Proposed Workflow 0.96 92% 94% RSNA 2018, 2019
- 96. Big-Data PACS for Quantitative COPD Reading ❖ Auto DICOM retrieve from legacy PACS. ❖ Auto processing supported by Deep-learning. ❖ Stored all the quantification values for every COPD patient ❖ Dashboard showing the overall statistics. ❖ Query by quantification. ❖ Similar case can be easily queried by quantification values when reading a patient’s chest CT images. RSNA 2018, 2019
- 97. 111 9. CBIR for Medical Images RSNA 2019
- 98. Deep Lesion 32000 CT images, 4400 unique px Summers group @ NIH 112
- 99. CBIR+GAN+AnoGAN 113 1. Database: PostgreSQL 2.Web application with Python: Flask 3.ORM (Object Relation Mapping): Flask-SqlAlchemy 4.REST API: Interacting with PostgreSQL using Psycopg2 5.Searching Algorithm: FLANN, LocalitySensitiveHashing library RSNA 2019
- 100. Content Based Image Retrieval in Chest PA Xray RSNA 2019
- 101. Content Based Image Retrieval in Chest PA Xray RSNA 2019
- 102. Content Based Image Retrieval in DILD CT DILD Classification using SVM Normal Honeycombing Ground -glass opacity Consolidation Emphysema Reticular opacity 𝑁. 𝐿 . 𝑣𝑜𝑥𝑒𝑙𝑠 𝑙𝑢𝑛𝑔 𝑣𝑜𝑥𝑒𝑙𝑠 𝐻. 𝐶. 𝑣𝑜𝑥𝑒𝑙𝑠 𝑙𝑢𝑛𝑔 𝑣𝑜𝑥𝑒𝑙𝑠 𝑅. 𝑂. 𝑣𝑜𝑥𝑒𝑙𝑠 𝑙𝑢𝑛𝑔 𝑣𝑜𝑥𝑒𝑙𝑠 : 4 x 4 x 4 x 6 = 384 features Red : IPF Blue : COP Green : NSIP 76.7% and 81.7% in Top3, and Top5 Same patient with no change CTs 3.88 ± 0.96 5-scale visual scoring by two radiologists RSNA 2019
- 103. Content Based Image Retrieval in COPD CT COPD feature extraction for CBIR Top 1; similarity score 5/5 Top 2; similarity score 4/4 Top 3; similarity score 4/4 Top 4; similarity score 4/4 Top 5; similarity score 2/3 Query case CBIR retrieval for similar chest CT with COPD 60.0% and 68.0% in Top3, and Top5 Same patient with no change CTs 3.55±0.98 5-scale visual scoring by two radiologists RSNA 2019
- 105. COPD Deep Radiomics : Survival Prediction Method KOLD cohort Random survival forest Extraction of deep features Design a CNN model for classifying 5- year survival Each slice was trained separately, and used for extracting representative deep features of it 122 Ishwaran, H., et al., “Random survival forests,” arXiv:0811.1645. Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D Conv2D Batchnorm ReLu Maxpooling2D FCN(1,024) 5-year survival CT slice (a) A patient survived less than 5-year. (b) A patient survived more than 5-year. Data C-index Deep radiomics features (A1+A2+A3+A4+A5+A6) 0.8412 Demographic information 0.7688 PFT 0.7498 Handcrafted features 0.5994 Accuracy of survival prediction CAM of survival prediction CNN architecture for Deep radiomics AMC Radiology, Seo JB
- 106. KOLD vs. Malaysian Age KOLD Malaysian Survival (days) 124AMC Radiology, Seo JB
- 107. Result Interval validation (5-fold cross validation) External validation (Malaysian) A1 0.6714 (0.6371, 0.7057) 0.6656 A2 0.7462 (0.7178, 0.7746) 0.6452 A3 0.7229 (0.6542, 0.7916) 0.6062 A4 0.6818 (0.6468, 0.7168) 0.6019 A5 0.7409 (0.7030, 0.7789) 0.6217 A6 0.7076 (0.6879, 0.7273) 0.6830 A2 + + A5 0.7587 (0.7179, 0.7995) 0.6774 A2 + A3 + + A5 0.7819 (0.7289, 0.8348) 0.6650 A2 + A3 + + A5 + A6 0.7603 (0.7065, 0.8141) 0.6768 A2 + A3 + A4 + A5 + A6 0.7532 (0.6956, 0.8108) 0.6749 A1 + A2 + A3 + A4 + A5 + A6 0.7288 (0.6759, 0.7817) 0.6582 125AMC Radiology, Seo JB
- 108. Deep Radiomics for Lung Cancer Deep learning based feature extraction Dataset Resample to 0.625mm x 0.625mm x 1.0 mm ROI cropping to 192 x 120 x 120 (12cm3) Adjust the size to fit 10cm3 in case of nodules larger than 10cm3 Margin: 2cm Non-linear classifier/predictor Deep learning based feature extraction Deep Features SurvivalChest CT 데이터 만드는 중 데이터 만드는 중 CT Segmented nodule Binary mask CT value Margin Nodule with margin Nodule AMC Radiology, Lee SM
- 109. Deep Radiomics for Lung Cancer Non-linear classifier/predictor Deep learning based feature extraction Deep Features SurvivalChest CT • Survival analysis using random survival forest – Training: 985(death: 144) / test: 110(death: 15) = total: 1,095(death: 159) AMC Radiology, Lee SM
- 110. 11. CDSS on Liver Cancer Kim KM, GI, AMC Clinical data (20 variables) RFA / Operation Or not Operation Classifier 1 Prediction B TACE Or not TACL+RT Prediction C Classifier 2 Classifier 3 sorafenib Others Or not Or not RFA Prediction A LT Classifier 4 Prediction G Prediction D Supportive care Classifier 5 Classifier 6Prediction E Prediction F 131
- 111. CDSS on Liver Cancer Treatment recommender system for liver cancer Patient’s baseline data: 20 variables 1,000 patients Hierarchical decision modeling Survival prediction Cox hazard regression, random forest survival 132 Survival Curve on Tx of Liver Cancer Kim KM, GI, AMC
- 112. Mutli-center Trials 4 CDSS on Liver Cancer 7 Multi-centers Amazon web service 133Kim KM, GI, AMC
- 114. 12. CNN for Steady Flow Prediction 135 CNN Prediction LBM 𝑦=f’(x) https://www.biorxiv.org/content/biorxiv/early/2017/12/30/240317.full.pdf
- 115. 12. CNN for Steady Flow Prediction 136 𝑦=f’(x)
- 116. 14. Surgical Robot with AI Will robot steal surgeons’ job? NO. Will robot CHANGE surgeons’ job? It may... Will robot and SUPER COMPUTER steal surgeons’ job? …
- 117. 14. Simple Mastoidectomy Surgery Video Simple Mastoidectomy Surgery Video Understanding Surgical Step Understanding 9 Steps 138 Example AMC ENT Jung JW Classification algorithm of surgical stage by learning video data 중이염 동영상 내 수술도구(위), Henle recognized (below)
- 118. Colonoscopy CAD 139Byeon JS, AMC
- 119. 14. Case Orchestration (Triage) 140 Reading ordering by AI for efficient reading Agfa, IBM, Philips, etc
- 121. 14. Hospital Workflow Improvement Intelligent Hand Hygiene Support @ Lucile Packard Children's Hospital at Stanford Intelligent ICU Clinical Pathway Support @ Stanford Hospital and Clinics @ Intermountain Healthcare Intelligent Senior Wellbeing Support @ Palo Alto Medical Foundation @ On Lok Senior Health Services @ Intermountain Healthcare ** Stanford AI-assisted Care Courtesy of Ha HS, VAIIM consulting, modified
- 122. 14. Beyond computer vision 143 Computer vision to discern clinical behaviors Bedside Computer Vision, Yeung S, Fei-Fei Li et al, NEJM 2018 April H Rosette *
- 123. Hand Hygiene Monitoring Clean Clips Images Clips Images Training 428 8,325 444 11,988 Validation 46 871 43 1,047 Testing 10 340 8 516 Total 484 9,536 495 13,551 Test accuracy: 0.82
- 124. Hand Gesture 145
- 125. Capsule Net 146A Lee, N Kim, CMPB 2018
- 126. EAU 2015, AUA 2015, RSNA 2015, WIP for publication 14. Application : 3DP, AR, Robot
- 127. 3D Printings 148
- 128. The limitations and solutions of 3DP in medicine 149 Limitations Solutions Slow printing speed New 3DP for 100~1000 times faster Variations of clinical imaging protocols ; especially thick slice thickness Domain adaptation; Super resolution with DL Limitation of materials Develop new materials and mechanism Slow segmentation for specific anatomy Training DL for semantic segmentation New manpower Train new expertise with help of AI Physicians’ additional efforts Communication tools with AI Already doing well Find new killer apps with 3DP RSNA 2018
- 129. Semantic Segmentation 150 Medical Images Super Resolution Module Auto Design Module GAN 진단용 저해상도 영상 3D모델링용 고해상도 영상 train/test set Generator Discriminator Y/N Similar? Auto Segmentation Module 3D모델링용 고해상도 영상 Segmented mask FCN Network Labeling Pre-processor train/test set End-to-End learning Segmented mask Surgical planner Ref. tracker Border tracker Surgical planning Y/N meet Design spec..? DeepLearning RSNA 2018
- 130. AI Solution for 3DP 151 Data acquisition DL Modeling 3DP Confirm by an expert Confirm by M.D. Application Yes NoNo Yes Segmentation with human-AI interaction Communication with AI Low-resolution image for diagnosis High-resolution image for 3DP with super-resolution Semantic segmentation method Super-resolution method ▪ Super-resolution convert low-resolution images into high resolution images. ▪ The result images of super-resolution could be better 3D models. Gold Standard 1st training Segmentation enhancement with human-AI interaction 2nd training 3rd training AI based 3DP workflow Manual segmentation result Result of 2D U-net (about 30 sec / case) Result of 3D U-net (about 1 sec / case) RSNA 2018
- 131. Slice thickness 152 4mm thickness 1mm original 1mm predicted
- 133. VR/AR/MR
- 135. Artificial Intelligence (+Big Data) Will Redesign Healthcare 1. Artificial Intelligence (x) ->Augmented Intelligence (o) 2. Just Starting 3. Precision medicine / Mining medical records / Designing treatment plans 4. Getting the most out of in-person and online consultations / Health assistance and medication management / Open AI helping people make healthier choices and decisions 5. Assisting repetitive jobs 6. Drug discovery / Clinical trial Case Matching 7. Analyzing, redesigning a healthcare system http://medicalfuturist.com/ Aug, 2016
- 136. Job Opening @ MI2RL_AMC, Seoul, SouthKorea Post-doc research fellow, PhD Students, Researchers 157 AMC, UoU Seoul South Korea Contact (namkugkim@gmail.com)
- 137. Collaborators Radiology Joon Beom Seo, SangMin LeeA,B, Dong Hyun, Yang, Hyung Jin Won, Ho Sung Kim, Seung Chai Jung, Ji Eun Park, So Jung Lee,Jeong Hyun Lee, Gilsun Hong Neurology Dong-Wha Kang, Chongsik Lee, Jaehong Lee, Sangbeom Jun, Misun Kwon, Beomjun Kim Cardiology Jaekwan Song, Jongmin Song, Young-Hak Kim Emergency Medicine Dong-Woo Seo Pathology Hyunjeong Go, Gyuheon Choi, Gyungyub Gong, Dong Eun Song Surgery Beom Seok Ko, JongHun Jeong, Songchuk Kim, Tae-Yon Sung Internal Medicine Jeongsik Byeon, Kang Mo Kim Anesthesiology Sung-Hoon Kim, Eun Ho Lee