IFMIA 2019 Plenary Talk : Deep Learning in Medicine; Engineers' Perspectives

Deep Learning in Medicine:
Engineering Perspectives
Namkug Kim, PhD
Medical Imaging & Intelligent Reality Lab.
Convergence Medicine/Radiology,
University of Ulsan College of Medicine
Asan Medical Center
South Korea

Researches with
LG Electronics
Coreline Soft Inc.
Osstem Implant
CGBio
VUNO
Kakaobrain
Conflict of Interests
Stockholder
Coreline Soft, Inc.
AnyMedi
Co-founder
Somansa Inc.
Cybermed Inc.
Clinical Imaging Solution, Inc
AnyMedi, Inc.
Selected Grants as PI
NRF, South Korea
7T용 4D 자기공명유속영상을 이용한 심뇌혈관 질환의 in-vivo 유동 정량화 SW개발, 2016
4D flow MRI을 이용한 심혈관 질환의 in-vivo 유동 연구, 2015-7
자기공명분광영상 및 MRI의 통합 분석 소프트웨어 개발
KEIT, South Korea
Digital Dentistry, 2018-2022
의료영상 인공지능 PACS 과제, 2016-20
3DP 척추 맞춤형 임플란트, 2016-20
3D 프린터 기반 무치악 및 두개악안면결손 환자용 수복 보철물 제작, 재건 시스템 개발, 2015-9
근골격계 복구 수술 로봇 개발, 2012-7
영상중재시술 로봇시스템 개발, 2012-7
Spine및 Neurosurgery 수술보조용 항법 시스템 개발, 2001
의료용 3차원 모델 제작 S/W 기술 개발, 정통부, 2000
의료영상재구성에 의한 가상시술 소프트웨어 개발, 중소기업기술혁신개발, 중기청, 2001
KHIDI, South Korea
연구중심병원 육성과제, 2019-2028
인공지능 학습센터 과제, 2018-2023
영상 뇌졸중 예후 예측 및 치료방침 결정 시스템 개발, 2012-8
관동맥 관류 CT 의 자동 진단 프로그램을 활용한 허혈성 질환의 진단과 치료, 2013-6
RP를 이용한 척추나사못 삽입술 계획 프로그램 개발, 2000
Companies Fundings
Siemens Germany, Hyundai Heavy Industry, Osstem Implant, S&G Biotech, Coreline soft, Midas IT,
AnyMedi, Hitachi Medical, Japan, Kakaobrain

Hyper-Connectivity
3
Computer : Calculator
(Electronic Numerical Integrator And
Computer; ENIAC) Digital Number,
Integral, and Calculator,
1947.07.29~1955.10
Computer + Network (Internet) :
Information processor,
ARPANET, 1969
DNS, 1984
WWW, 1989
Internet, 1995 (SK)
Computer + Connectivity (IoT) :
Big Data, Automation

Internet of Thing (IoT)
1% devices were connected each other

Big data : IoT Thermometer
IoT Thermometer
Kinsa : Startup @ USA
Real-time body temperature bigdata@USA
Patients-derived health data
Regional basis
Influenza stats
Kinsa : Realtime vs CDC : 3 week delay
B2B model:
Demands and production
Drugs of flu shot, anti-bacteria, etc
Tooth brush, orange juice, soup, etc
5
Influenza trends: comparison
with CDC 2.5 y

Big data : Google Trends
6Nature 2008

Big data : Facebook
7
Correlation between Facebook usage vs drug addiction
Accuracy : Tobacco(86%), Alcohol(81%), Drug(84%)

Healthcare Bigdata
7.5 Exa Byte/day (30% of every data)
Biological characteristics, health histories, wellness status, going places,
expenditure history, sleep state, meal and excretion / experiment record,
medical imaging, genetic information, liquid biopsy, electrocardiogram /
insurance claim, clinical trial, prescription etc
8IBM Healthcare

Opportunity
9
8 trillion exam:
Healthcare Industry
2 trillion : wastes in
healthcare industry
Better experience
Imaging :
Unnecessary tests
Lower cost
Oncology:
Variability of Care
Better outcomes
Life sciences:
Failed clinical trials
Government:
Fraud, Waste and Abuse
Value Based Care:
Cost of chronic disease
360 billion : total IT and
healthcare market
opportunity
*IBM Watson

Beyond Human-level Performance
• Now, AI can beat humans in tasks which once considered impossible
5:0
vs Fan Hui
(Oct. 2015)
4:1
vs Sedol Lee
(Mar. 2016)
RF vs SL
Modified from Kyuhwan Jung’s slide

TPU Server
used against Lee Sedol
TPU Board used
against Ke Jie

Libratus(Jan 30, 2017) DeepStack(Science, Mar 02, 2017)

AI Medical Device Cleared in FDA
14
https://twitter.com/erictopol/status/1028642832171458563

AI Medical Device Cleared in South Korea
Clinical trials on AI Medical Device (kFDA)
Vuno – Bone aging
Lunit - Chest PA X-ray Nodule CAD
JLK Inspection - Stroke MRI CAD
Midas IT – Dementia MRI Index
15
DILDlung disease Chest PA, MammoCAD,Pathology,etc Dementia MRI Stroke MRICAD (3rd grade)

AI + Healthcare Market Size
Healthcare AI Market Size:
10B USD(1조원)@ 2015 -> 67B USD(7조원)@2021, CAGR : 42%/y*
*Prost and Sullivan, ** WHO, ***Variant Market Research, ****한국보건산업진흥원
Global healthcare expenditure**
Global IT healthcare market**
AI healthcare market @ SouthKorea****Ads 20B USD <<< Healthcare 9,500B USD (50x)

AI Medical Device
17
• Verily@Google: Normal vs Abn, Anti-aging, Life
Prolongation
• IBM: Truven, 40B USD
• Apple: GSK, EMR +iPhone Healthcare Platform
• Facebook: Incurable dx, Human cell atlas, 5000M USD
• Zebra Medical Vision
• AI medical imaging Dx : 1st place of investment
• Medical imaging reading cloud service/1 USD

Reimbursement on AI Medical Device
World first PACS Reimbursement -> AI Medical Device?
4차산업혁명위원회, 연구용역 중
NECA : Fast track
문재인 케어?
18

Digital Hospital
Command center@Johns Hopkins
Data + AI
Health records, emergency medical
services, research results, number of
available beds
– Decision on preparing surgery team or transfer
19GE healthcare
• Cancer patient admission
capacity :About 60% increase
• ER waiting Pts: 25% decrease
• Pts for surgery : 60% decrease

EMR + IoT
AI based alerting system in monitoring devices
Detect minute changes in vital signs -> Prevention
Code blue code blues reduced by 56% **
Risk index of each pt evaluated -> Nurses care dangerous
patients first
20
After recording the patient's
vital signs on the chart, enter
the individual numbers in the
HER manually
Data is automatically
uploaded from patient's wrist
band, nurse enters from ward
to portable equipment
Current status ->
AI help human health care with smart, efficient, and accurate manner
**Philips Healthcare

ICT + Healthcare Innovation
ICT : Healthcare Innovation
21
• Medical Insurance $18,764/man@2017
• Company (70%) : $13000/employee
• Drugstore CVS: Acquire Aetna (Health Insurance, 690억 USD)
• Health Insurance Cigna : Acquire Express Scripts (Drug Expense, 500억 USD)
Amazon, Berkshire Hathaway, JPMorgan

Machine Learning
scikit-learn algorithm cheat-sheet.

Comparison btw Brain and NN
25

Bio Plausible Neural Network
Mimic human visual
recognition system
Neocognitron, proposed by
Hubel & Wiesel in 1959
Visual primary cortex by
cascading from S-Cell to C-Cell
Each unit connected to a small
subset of other units
Based on what it sees, it decides
what it wants to say
Units must learn to cooperate to
accomplish the task
26From Gallant and van Esses, Simon Thorpe

CNN : Major Breakthroughs in Feedforward NN
K. Fukushima Yann Lecun G. Hinton, S. Ruslan
Neocognitron (1979)
• By Kunihiko Fukushima
• First proposed CNN
Convolutional Neural Networks (1989)
• Yann Lecun et.al
• Back propagation for CNN
• Theoretically learn any function
Neocognitron
LeNet-5 architecture
Alex krizhevsky , Hinton
LeNet-5 (1998)
• Convolutional networks
Improved by Yann Lecun et.al
• Classify handwritten digits
D. Rumelhart, G. Hinton, R.
Wiliams
1960 1970 1980 1990 2000 2010 2012
Perceptron
XOR
Problem
Golden Age
1957 1969 1986
F. RosenblattM. Minsky, S. Papert
• Adjustable weights
• Weights are not learned
• XOR problem is not linearly
separble
• Solution to nonlinearity separable problems
• Big computation, local optima and
overfiting
CNN Breakthrough (2012)
• By Alex Krizhevsky et al.
• Winner of ILSVRC2012 by
large marginDark Age (AI winter)
Back propagation (1981)
• Train multiple layers
Multi-layer Perceptron
(1986)
1950
Neocognitron (1959)
• Hubel & Wiesel
• by cascading from S-
Cell to C-Cell

Feature Engineering vs Feature Learning
Modified From Yann LeCun
Knowledge-driven Feature Engineering
Conventional Radiomics
Data-driven Feature Learning
Deep Radiomics
•Feature Learning instead of Conventional Feature Engineering Removes Barriers for
Multi-modal Studies and Data-driven Approaches in Medical Data Analysis

Data driven Medicine
End-to-End
Prostate cancer radiomics and the promise of radiogenomics, TCR 2016
http://tcr.amegroups.com/article/view/8705/html End-to-end

Machine Learning vs Deep Learning
— Scale Matters
— Millions to Billions of parameters
— Data Matters
— Regularize using more data
— Productivity Matters
— It’s simple, so we can make tools
Data & Compute
Accuracy Deep Learning
Many previous
methods
Deep learning is most
useful for large problems
Modified by Nvidia DLI

Computational map
31
Dense
Few
Millions
#ofVariables
(logscale)
Completeness of Data (Sampling)Sparse
More
• Compute
• Data
• Storage
• Bandwidth
Computationally
Intractable SpaceDeep Learning
Neural Nets
Statistical Analysis
Algorithms, Closed Form Solutions
Expert
Systems
Space of insufficient
data for analysis
[Un]supervised Learning
Models
Intuition
Modified by Philips Healthcare Inc.

Convolutional Neural Networks (CNN)
Inspired by biological process
Weight sharing (convolution) +
Subsampling (pooling)
Reducing the number of parameters (Reduce over-
fitting)
Translational invariance
Input
28 × 28
Feature maps
4@24 × 24
Feature maps
4@8 × 8
Feature maps
8@4 × 4
Feature maps
8@2 × 2
Feature maps
8 ⋅ 2 ⋅ 2 × 1
Output
10 × 1
Convolution
layer
Max-pooling
layer
Convolution
layer
Max-pooling
layer
Reshape Linear layer
[LeCun, 1998]

Paradigm shift
39
Analog Digital
vs
Program Deep Learning
vs
Data GS
Results
Data GS
Program<<
Ref
Little cost for duplication
Little cost for developing program

Deep learning & Medicine
Keyword Search “Deep learning” in PubMed
Updated on September 14th, 2017

Better Decision in Medicine:
Clinical Decision Support System /
Risk Prediction
Precision medicine
Massive search of medical information
Mining medical records
Advanced analytics
Designing individualized treatment plans
Individualized/group risk prediction

Group/Individual Risk Prediction
Lumiata
Predicting health with transparent, precise analytics to
automate risk and revenue operations
Developed first-of-its-kind ‘medical graph’ in order to
build graph representations of how illnesses and
patients are connected
Ingested more than 260 million data points from
textbooks, journal articles, public data sets and other
places
Analyzes the complex, multidimensional relationships
between them, allowing clinical insights across the
entire healthcare network
The Lumiata Medical Graph

Better Patient Management
Health assistance and medication management
Getting the most out of in-person and online consultations
Open AI helping people make healthier choices and decisions

Medication Monitoring Solution
▪ AiCure
▪ A provider of a facial recognition and motion sensing technology to
medical ingestion
-Substantial funding from pharmaceutical industry, academic collaborators,
and the National Institutes of Health
▪ Combine machine learning with smartphone technology to remind
people to take their medicine
▪ The data it provides to its systems transmits in real time back to a
clinician through a HIPPA – compliant network
-Clinicians conforming through the system that the patients are taking their
medicine as instructed
Sends the patient a
reminder, and then
requests that they use
the camera built into
their phone to video
themselves taking the
medicine
Visually confirms that the
person in the video is the
patient, and then to identify
the pill in the mouth of the
patient to prove that they
have taken their medicine
1) Since 2009, New York-based, $12M Funding

Interactive Telemedicine : Platform for Follow-up
Patients
• Sense.ly
• “Virtual nurse" application that provides proven,
personalized patient monitoring and follow-up
care1)
– In collaboration with MindMeld, AI technology
used to understand the meaning of the spoken
questions and provide answers or relevant
medical information
• Combines avatar-based technology, sensor
capabilities, and telemedicine features that
generate actionable, real-time data and
intelligent analytics
– The app has already reduced patient time by
20% if places where it has been used
– 91% patient enrollment rate, 87% assessment
completion rate, 72% avoided unnecessary calls
Voice Recognition
Understanding, QNA
Disease Information Alarm
system
Telemedicine
1) The first platform for building AI-powered voice-driven applications
2) https://techcrunch.com/2017/02/14/virtual-nurse-app-sense-ly-raises-8-million-from-investors-including-the-mayo-
clinic/?ncid=rss&utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Techcrunch+%28TechCrunch%29&sr_share=facebook

Sugar.IQ
Hypoglycemic event prediction
Continuous monitoring of glucose
Lift style monitoring
Pattern analysis (AI)
Insulin pump

Efficiency
Speech Recognition
Medical imaging
Image processing, detection, diagnosis, classification
AI assistant / Chatbot
Scheduling, consulation
Surgical Robot

Virtual Interviewer to Treat PTSD
Linguistic + behavioral nuances
Institute for Creative Technologies (ICT) at the University of Southern California

Drug Discovery
http://fortune.com/2016/04/22/berg-pacreatic-cancer-artificial-intelligence/
http://tech.co/berg-medicine-artificial-intelligence-2016-07
http://www.wired.co.uk/article/niven-r-narain-ai-drugs-wired2015
Data analytics software + in-the-lab drug
development to find new treatments
Analysis of massive amounts of biological data to
uncover unexpected connections between
healthy and sick patients
-The resulting insights allow for a more informed
hypothesis, which in turn enables more efficient drug
development
-Provides real time analytic solutions that predict the
impact of treatment plans at the individual level to
optimize population health strategies
✓ Starts by drawing sequencing data from human tissue samples, as well as
information about protein formation, metabolites, and other elements of
functional data.
✓ The process produces trillions of data points from a single sample. The
data is then combined with patient clinical information and analyzed by
our proprietary artificial intelligence machine learning analytics program.
The BERG Interrogative Biology® Platform

AI Application in Medical Imaging
Almost all aspects
Image transformation
Lesion segmentation
Lesion classification
Lesion detection
Finding similar cases
Assistance of interpretation

TASKS
Image SegmentationObject Detection
Image Classification +
Localization
Image Classification
(inspired by a slide found in cs231n lecture from Stanford University)
Nvidia DLI Education Materials

Clinical Unmet Needs on Deep Learning
1. Efficient anonymization, curation, and smart labeling for cheap labeling
2. Domain adaptation or image normalization to overcome differences of multi-center trials
3. Interpretability and visualization to mitigate black box property
4. Novelty (Abnomly) detection under supervised learning for human decision in later
5. Uncertainty of medical data, and uncertainty of artificial intelligence decision
6. Reproducibility study of deep learning using repeatedly scanned images
7. Content based image retrieval
8. GAN Applications
9. Deep radiomics and deep survival
10. Augmentation, curriculum learning, one / multi-shot learning to solve diseases’ imbalance, rare or a
small number of dataset
11. Big data PACS
12. Develop physics-induced machine learning with well-known physics and medical laws
13. Robust to adversarial attack
14. Etc Applications
55

1. Smart Labeling : Surgical Imaging
AI assisted labeling with semantic
segmentation from medical images
Pancreas
Stomach, kidney, liver, etc
Cervical spine
(Med Phys Revision)
MICCAI 2018 Segmentation Decathlon 2nd place
Semantic segmentation of AI
saving time
• No human
interaction
• 10 msec/slice
more accurate
• More
reproducible
• More robust
saving cost
• Fast
segmentation
• 2~3 times faster
after correction
Tip!
Maxillary sinus, mandible,
mandibular canal (RSNA 2018) Glioblastoma (GBM)
Lung lobe (JDI submitted)
Breast (RSNA 2017)
Airway (MedIA 2018)

1. Smart Labeling; Medical Segmentation Decathlon
59
10 organs; 52 labels
MSD challenge, MICCAI 2018
Cascaded U-Net
2nd Place

1. Smart Labeling; 2.5D CNN Airway Segmentation in 3DCT
80 COPD Patients’ Inspiration CT
69 CT volumes are included in training
11 CT volumes are NOT included in training
GS : Manual segmentation
60AMC, Radiology, Seo JB, Lee SM
Airway Labelling SW
2.5D CNN

1. Airway Segmentation; Label SW
61
Yoon JH, Kim N, et al, Medical Image Analysis 2019
1~2Hr -> 2 min

1. Smart Labeling; Lung Lobe Seg
62
3D U-net 모식도 Comparisons of segmentation accuracy
(a) Gold standard, (b) Hessian based segmentation result, (c) Deep learning
based segmentation results
AMC, Radiology, Seo JB, Lee SM ; JDI Revision

1. Multi-organ segmentation for abdominal CT scans

1. Smart Labeling; Active Learning
Volumetric CT with contrast enhancement
(20 cases : 6 left, 14 right)
AMC Nephrology, Kyoung YS
Kidney 3D Semantic Segmentation6 classes :
artery, cancer, cyst, parenchyma, ureter, vein

Human Segmentation
Human Segmentation
AI_ 1st Test
AI_ 1st Test
AI_ 2st Test
AI_ 2st Test
AI_ 3rd Test
AI_ 3rd Test
rebuilding ground truth increasing data
0.88
1. Smart Labeling; Active Learning
DICE 0.91 0.95 10 sec
AMC Nephrology, Kyoung YS

1. Smart Labeling; Comparison of AI Generated Dataset
Kim YG, Kim N, et al, Sci Report Revision
Peri-tubular capillary (PTC) counting

2. Domain Adaptation : Pancreatic Cancer
▪ Pancreas segmentation using domain adaptation
Multi-center datasets: AMC and NIH
✓AMC: 220 patients ✓NIH: 82 patients
AMC Radiology Kim HJ

Domain adaptation

Domain adaptation Source Target Dice
NIH NIH 0.7601
AMC 0.5833
AMC (baseline) AMC 0.8466
NIH 0.4649
AMC (with DA) AMC 0.8284
NIH 0.6770

2. Normalization : Super-Resolution
Undersampling
Image
Fully-
Reconstructed
Image
Generator

2. Image Normalization : CT Kernel Conversion
71AMC Radiology Seo JB, Lee SM, Submitted
Network architecture
Conversion Baseline EDSR Ours (Single) Ours (Multi)
B10f – B30f 15.67 / 0.9867 5.64 / 0.9970 4.72 / 0.9976 4.28 / 0.9978
B10f – B50f 56.02 / 0.8345 30.87 / 0.9329 29.02 / 0.9449 27.24 / 0.9458
B10f – B70f 114.49 / 0.6277 78.16 / 0.8103 75.57 / 0.8293 71.96 / 0.8270
B70f – B50f 63.33 / 0.8718 11.02 / 0.9933 9.31 / 0.9950 8.77 / 0.9949
B70f – B30f 102.99 / 0.6200 13.45 / 0.9885 10.46 / 0.9928 9.22 / 0.9931
B70f – B10f 114.49 / 0.6277 11.99 / 0.9907 10.05 / 0.9936 8.64 / 0.9939
To overcome CT vendor differences
Iterative and progressive learning

Dataset : Compressive Sensing
AMC Radiology Jung SC

Denoising network
Yang, Q., Yan, P., Kalra, M. K., & Wang, G. (2017). CT image denoising with perceptive deep
neural networks. arXiv preprint arXiv:1702.07019.

Denoising : Compressive Sensing
74
PredictOriginal Difference
CS 6.84 -> CS5.56
Problems : Output Image Blurring

Img-to-Img ; Pix2Pix
Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, et al, 2016
GAN Loss + L1Loss

Pix2Pix Network
2D ConvLayer
Batch
Normalization
ReLU
2D ConvLayer
Batch
Normalization
X 9
Generator Network : ResNet 9 blocks

Super Resolution : Compressive Sensing
CS 6.84 -> CS5.56
PredictOriginal Difference
Training Result with ~3000 training images

CycleGAN
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu*, et al, ICCV 2017, https://arxiv.org/pdf/1703.10593.pdf

CycleGAN
Cycle-concistency loss
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu*, et al, ICCV 2017, https://arxiv.org/pdf/1703.10593.pdf
Cycle loss Total loss

—> They use same network both for Generator and Discriminator
Same Generator and Discriminator

3. Explainable AI – What Are We Trying To Do?
84
Today © Spin South West
• Why did you do that?
• Why not something else?
• When do you succeed?
• When do you fail?
• When can I trust you?
• How do I correct an error?Training
Data
Tomorrow
Learned
Function
© Spin South West
Output User with
a Task
• I understand why
• I understand why not
• I know when you’ll succeed
• I know when you’ll fail
• I know when to trust you
• I know why you erredTraining
Data
Explainable
Model
Explanation
Interface
Userwith
a Task
DARPA; Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
© UniversityOf Toronto
Learning
Process
This is a cat
(p = .93)
© UniversityOf Toronto
New Learning
Process
This is a cat:
•It has fur, whiskers,
and claws.
•It has this feature:

3. Interpretability vs Accuracy
85
PredictionAccuracy
Explainability
Learning Techniques (today)
Neural Nets
Statistical
Models
Ensemble
Methods
Decision
Trees
Deep
Learning
SVMs
AOGs
Bayesian
Belief Nets
Markov
Models
HBNs
MLNs
SRL
CRFs
Random
Forests
Graphical
Models
Explainability
(notional)

3. DARPA XAI (eXplanable AI)
86
New
Approach
Create a suite of
machine learning
techniques that
produce more
explainable models,
while maintaining a
high level of learning
performance
PredictionAccuracy
Explainability
Learning Techniques (today) Explainability
(notional)
Neural Nets
Statistical
Models
Ensemble
Methods
Decision
Trees
Deep
Learning
SVMs
AOGs
Bayesian
Belief Nets
Markov
Models
HBNs
MLNs
Model Induction
Techniques to infer an explainable model
from any model as a black box
Deep Explanation
Modified deep learning techniques to
learn explainable features
SRL
Interpretable Models
Techniques to learn more structured,
interpretable, causal models
CRFs
Random
Forests
Graphical
Models

Explainable AI – Why Do You Think It Will Be Successful?
87
Training
Data
Explainable
Model
Explanation
Interface
Deep Explanation
LearningSemantic
Associations
H. Sawhney (SRI Sarnoff)
Learning to Generate
Explanations
T. Darrell, P. Abeel (UCB)
Interpretable Models
StochasticAnd-Or-
Graphs(AOG) Song-
Chun Zhu (UCLA )
Bayesian Program
Learning
J. Tenenbaum (MIT)
Model Induction
Local Interpretable
Model-agnostic
Explanations (LIME)
C. Guestrin (UW)
Bayesian Rule Lists
C. Rudin (MIT)
HCI
Prototype
Explanation Interface
T. Kulesza (OSU/MSR)
UX Design, Language
Dialog, Visualization
ENGINEERING PRACTICE
Psychology
Principles of Explanatory
Machine Learning
M. Burnett (OSU)
PsychologicalTheories
of Explanation
T. Lombrozo (UCB)
© University Of Toronto
New Learning
Process
This is a cat:
• It has fur, whiskers,
and claws.
• It has this feature:

Interpretable model
88
Patrick Hall et al. (O’REILLY, 2017) Macro Tulio Ribeiro et al. (O’REILLY, 2016)
Surrogate models Local-Interpretable-Model-agnostic Explanations
(Perturbation)
Data Accurate
Model

해석력 강화 모델
89
Yin Lou et al, ACM 2012
GAM, Generalized Additive Models Everything should be made as simple as possible, but
not simpler. — Albert Einstein.
• 변수간 상호작용 효과를 배제
• 설명력 제고
• 개별 변수별로 복잡한 구조의
알고리즘을 적용한 후 이를
더하기 형태로 종합

Analysis on Deep Learning Methods for Predicting Patient Survival
Basic CNN Modified CNN Multi-layer CNN Residual NetworkFeed-forward NN

3. Visualization : Machine Operable, Human Readable
Visual attention
Class Activation Map (CAM)
Category – feature mapping
Sparsity and diversity
91

Visualization of salient region in bone x-ray
92

Evidence hotpot for lesion visualization
“SpineNet: Automatically Pinpointing Classification Evidence in
Spinal MRIs”
93

3. Visualization : Bias representation in CNN
94
Zhang et al., 2018b

3. Knowledge hierarchy hidden in conv-layers
95
Zhang et al., 2018a

96
Zhang et al., 2018a

97
Zhang et al., 2018a

3. Decision tree
Disentangled
representation
99

3. Interpretable convolution layer
100

3. Visualization of interpretable filters in top conv-
layer
101

3. Parse tree
102
Latent parse tree and configuration as quantitatively extractive rationale in detection

3. Filter’s location instability
103

3. Variable Patterns of Chest Abnormalities

Weakly Supervised Learning + Class Activation Map
3. Visualization : Chest PA @ AMC
Lee SM, Seo JB, Radiology, AMC

Interesting Case
107
Normal Cardiomegaly
Ground Truth Prediction
Surgical Wires

Image Pattern Classification on DILD HRCT
(a) Consolidation (b) Emphysema (c) Normal
(d) Ground-glass opacity (e) Honeycombing (f) Reticular opacity
Image
Feature
Extraction
Machine
Learning
Training dataset Testing set
Accuracy (%)
SVM Classifiers
GE GE 92.02 ± 0.56
Siemens Siemens 89.92 ± 0.36
GE+Siemens GE+Siemens 89.73 ± 0.43
N : 100 ROIs in each class of GE, Siemens CT
Kim GB, Kim N, et al, Journal of Digital Imaging, 2017

CNN networks
109
Input Image Conv. Layer #1 Conv. Layer #2 Conv. Layer#3 Conv. Layer#4 FC #1 FC #2
Local Response
Normalization
Dropout
100
6
20
20
19
19
64
64
64 64
9
9
4
4
4
4
4
4
4
4
3
3 3
3
3
3
Max
Pooling
Max
Pooling
Local Response
Normalization
• Two basic data replication
✓ random cropping and flipping: 20 x 20 pixels of ROI patches (top left, top right, center, bottom left, bottom right) & horizontally flipped
✓ mean centering and random noising: random noise of Gaussian distribution with mean zero and standard deviation

3. Image Pattern Classification on DILD HRCT
(a) Consolidation (b) Emphysema (c) Normal
(d) Ground-glass opacity (e) Honeycombing (f) Reticular opacity
DEEP
Learning

3. t-SNE Map ; SVM vs CNN
111
O = GE; X = SiemensSVM CNN

3. t-SNE map of VAE ; DILD CT Patches
DILD 600 patches (20x20=400 dim.)--> autoencoder --> 16
dim. (96% reduction) -->
112
0: normal, 1: GGO, 2: Consolidation, 3:
ReticularOpacity, 4: Emphysema, 5:
Honeycombing

3. t-SNE Map of VAE ; Chest PA X-rays
113
1 = normal / 0 = abnormal
T-SNE map
Lung Boundary
VAE
Whole Texture

3. Interpretability; DILD UIP/NSIP Differentiation
Disease Distribution
Feature Extraction
Machine
Learning UIP or NSIP?
(91%)
Jun S, Park B, Seo JB, Lee S, Kim N*. J Digit Imaging. 2017

4. Novelty (Untrained catergory)
In clinical situation
Novelty is everywhere, especially supervised learning
Rare diseases, but well known to medical doctors
Hard to training
How to determine novel (untrained) category
Unsupervised learning
Semi-unsupervised learning
Normal vs abnormal
Abnormality Detection
115

4. Detecting Out-of-Distribution Samples
116

4. Anomaly Detection
117Courtesy of Kang, Min-Guk

5. Uncertainty
Uncertainty of training data
In clinical situation, it is common
Deep Bayesian Modeling
Uncertainty of classification/prediction of Machine Learning
121

Kendal (2017)
5. Aleatoric & epistemic uncertainties
122
By chance System uncertainty

6. Reproducibility
Test-retest is one of most important issues of biomarker
Multiple scanning within similar date
Evaluate reliability of AI/Deep learning
Chest PA (Nodule)
Nodule Size on Chest PA with YOLO
– 56% : Variability of Size Marker
– Chest PA 50 pairs
DILD CT
123

6. Chest PA : 5 Disease Patterns
▪ Curriculum learning for multi-label classification
Results

6. Chest PA : Extra-validation
▪ Extra validation using multi-center datasets
Results
AMC
Densenet 201
True
Lesion Normal
Pred.
Lesion 690 44
Normal 30 1170
AMC
Resnet 152
True
Lesion Normal
Pred.
Lesion 668 61
Normal 52 1153
Sensitivity: 95.8%, Specificity: 96.4% Accuracy: 96.2% Sensitivity: 92.8%, Specificity: 95.0% Accuracy: 94.2%
SNUBH
Densenet 201
True
Lesion Normal
Pred.
Lesion 99 14
Normal 1 86
SNUBH
Resnet 152
True
Lesion Normal
Pred.
Lesion 100 38
Normal 0 62
Sensitivity: 99%, Specificity: 86% Accuracy: 92.5% Sensitivity: 100%, Specificity: 62% Accuracy: 81.0%

6. Chest PA Reproducibility : Nodule Detection
▪ Reproducibility analysis on CNN-based detection
Comparison of various networks
✓YOLO v2
✓VUNO-net
Nodule
(N = 121)
1st
P N
2nd
P 94 3
N 15 9
✓Faster RCNN
✓Mask RCNN
Nodule
(N = 121)
1st
P N
2nd
P 90 8
N 11 12
Nodule
(N = 121)
1st
P N
2nd
P 90 3
N 13 15
Nodule
(N = 121)
1st
P N
2nd
P 97 4
N 14 6
Nodule
(N = 121)
1st
P N
2nd
P 116 1
N 2 2
✓Reader 2
Nodule
(N = 121)
1st
P N
2nd
P 111 3
N 4 3
✓Reader 1

127
7. CBIR for Medical Images
AMC 영상의학과 서준범, 이상민 교수

7. Deep Lesion
32000 CT images, 4400 unique px
Summers group @ NIH
128

129
7. CBIR for Medical Images
AMC Radilogy, Seo JB, Lee SM
The Previous Research for Quantification
Definition of Similar Lung Images
Extraction of Distribution
Features
Extraction of Distribution Features
* Y.J.Chang, et
al,.Medical Physics
40 (5), 2013

8. Generative Adversarial Network (GAN)
Deep Convolutional Generative Adversarial Networks (DCGAN)
Rotations are linear in latent space
Bedroom generation
Arithmetic on faces

8. GAN
131
Kim, Namju’s Slideshare

8. Text-to-Image generative model (cGAN)
Zhang et al

8. PGGAN; Chest PA Xray
133
Progressive growing GAN (PGGAN)
https://arxiv.org/abs/1710.10196
Tero Karras, et al
Progressive Growing of GANs for Improved Quality, Stability,
and Variation,

138
Change latent space to generate normal chest X-ray

139
9. Deep Radiomics
QIRR@RSNA2017

140
9. Deep Radiomics
QIRR@RSNA2017QIRR@RSNA2017

9. Deep Radiomics for Lung Cancer
Deep feature extraction using segmentation network
Variable input: 163, 323, 643, 1283 according to the nodule size
Data augmentation: rotation, translation, scaling, shearing
Accuracy of trained model: Dice similarity coefficient = 88.14 %
Deep features: last convolutional layer of encoder stack (# of features: 256)
142
Non-linear
classifier/predictor
Deep learning based
feature extraction
Deep Features SurvivalChest CT
ROI
(lung nodule)
Chest CT
3D U-Net based
nodule segmentation
Segmented
lung nodule
영상의학과 이상민B 교수님 협력연구

GBM from PCNSL (primary central nervous system lymphoma)
144
Park JE, Kim HS, et al, Sci Report Under Revision

145

146

147

9. COPD Deep Radiomics : Survival Prediction
Method
KOLD cohort
Random survival forest
Extraction of deep features
Design a CNN model for classifying 5-
year survival
Each slice was trained separately, and
used for extracting representative deep
features of it
148
Ishwaran, H., et al., “Random survival forests,” arXiv:0811.1645.
Conv2D
Batchnorm
ReLu
Maxpooling2D
Conv2D
Batchnorm
ReLu
Maxpooling2D
Conv2D
Batchnorm
ReLu
Maxpooling2D
Conv2D
Batchnorm
ReLu
Maxpooling2D
Conv2D
Batchnorm
ReLu
Maxpooling2D
FCN(1,024)
5-year
survival
CT slice
AMC 영상의학과 서준범, 이상민 교수 등
(a) A patient survived less than 5-year.
(b) A patient survived more than 5-year.
Data C-index
Deep radiomics features (A1+A2+A3+A4+A5+A6) 0.8412
Demographic information 0.7688
PFT 0.7498
Handcrafted features 0.5994
Accuracy of survival prediction
CAM of survival prediction
CNN architecture for Deep radiomics

9. CDSS on Liver Cancer
Treatment recommender system for liver cancer
Patient’s baseline data: 20 variables
1,000 patients
Hierarchical decision modeling
Survival prediction
Cox hazard regression, random forest survival
153AMC 소화기내과 김강모 교수 등
Liver Cancer Tx Decision
Survival Curve on Tx of Liver Cancer

10. Perlin noise augmentation
Infinity augmentation strategy
Perlin noise by complexity theory
156Bae HJ, Kim N, Sci Report 2018

10. Solution to imbalanced data
157

10. GAN augmentation
158Hojjat Salehinejad, et al., ICASSP 2018

Weakly Supervised Learning + Class Activation Map
10. Curriculum learning : Chest PA

Dataset
161
 Adults chest-PA X-ray images
• Confirmed using CT images by expert radiologists
• Annotated with 5 diseases
• 7 : 1 : 2 = training : tuning : test set
AMC SNUBH
Normal 6,069 1,035
Abnormal 3,417 4,402
Nodule 944 1,189
Consolidation 550 853
Interstitial opacity 280 1,009
Pleural effusion 1,364 998
Pneumothorax 331 944

Problem Definition
162
 Multi-class problem vs. Multi-label problem
• Multi-class problem) Each sample belongs to one class.
• Multi-label problem) Each sample can belong to more than one class.
https://gombru.github.io/2018/05/23/cross_entropy_loss/

Chest PA X-ray (2)
Multi-class classification
✓ Each instance exclusively belong to single class
Multi-label classification
✓ Each instance can belong to multiple classes

Multi-label
164
 Multi-label problem (multi-binary classification)
• Multiple sigmoid functions
• Average of multiple binary cross-entropy losses
Loss 𝑦, ො𝑦 = ෍
𝑗=0
𝐾
𝑤𝑗 −
1
𝑁
෍
𝑖=0
𝑁
𝑦𝑖,𝑗 ∙ log ෞ𝑦𝑖,𝑗 + (1 − 𝑦𝑖,𝑗) ∙ log(1 − ෞ𝑦𝑖,𝑗)
ෞ𝑦𝑖,𝑗 = 𝑆𝑖𝑔𝑚𝑜𝑖𝑑 ෞ𝑎𝑖,𝑗 =
exp ෞ𝑎𝑖,𝑗
1 + exp ෞ𝑎𝑖,𝑗
ෞ𝑦𝑖,𝑗; the independent probability corresponding to class j of sample i
ෞ𝑎𝑖,𝑗; the activation value.
𝐾; the total number of classes
𝑁; the total number of samples
𝑤; the weight terms to deal with imbalanced problem

Curriculum learning
165
 Two-steps curriculum learning
• Step 1) training lesion-specified patch images
• Step 2) fine-tuning with entire images
NM
ND
Resnet-50
(pre-trained on imagenet dataset)
CS
IO
PE
PT
NM
ND
CS
IO
PE
PT
transfer
1)
2)
512
512
1024
1024
NM : Normal
ND : Nodule
CS : Consolidation
IO : Interstitial opacity
PE : Pleural effusion
PT : Pneumothorax

Preprocessing
167
 Abnormal patch images
• From abnormal region
pneumothorax
nodule

Comparison Results
168
 Curriculum learning vs. Baseline (w/o curriculum)
• Both models converged well.
• Curriculum learning showed better result, more stable and shorter training time.
Baseline
Epoch 1257
Loss 0.163
Acc
AMC: 96.1
SNUBH: 92.7
Curriculum
learning
Epoch 199
Loss 0.140
Acc
AMC: 97.2
SNUBH: 94.2
Loss on tuning set
Epoch
Loss

3. Interpretation; Evaluation
170
 t-SNE visualization with Embedded features
• All disease pattern was clearly separated into different clusters.
• Nodule and consolidation seem like difficult to be distinguished.

3. Interpretation; Evaluation
171
 Are the two-center datasets well mixed ?
• No, different imaging characteristics could have big problems.
• It would be required methods such as domain adaptation

3. Visualization; Chest PA X-ray
Results

10. Curriculum learning : Endoscopy of Colonic Polyp
Polyp detection
in endoscopy
(video)
Close-up shot on
polyp of interests
Pathological
Diagnosis
(multi-modal)
• https://arxiv.org/abs/1512.03385 “Deep Residual Learning for Image Recognition”
• https://arxiv.org/abs/1512.04150 “Learning Deep Features for Discriminative Localization”AMC Digestive Internal Medicine Byeon JS
Procedure of Detection and Classification of colon polyps
CAM result of classification
Colonoscopy Automation
Histopathology
Hyperplastic
polyp
: HP 65
166
Sessile serrated
adenoma
: SSA 101
Benign
adenoma
: BA 521
655Mucosal or
superficial
submucosal
cancer
: MSMC 134
Deep
submucosal
cancer
: DSMC 101 101

Artificial intelligence in gastrointestinal endoscopy
175World J Gastrointest Endosc. 2018 Oct 16;10(10):239-249

10. Curriculum learning : Modeling

3. Visualization: CAM(Class Activation Map)

11. Big data PACS platform
Bigdata PACS
Quantifying every data
Transforming PACS into big data platform
Advanced processing service (APS)
Lung nodules
– Detection : location
– Segmentation : boundary drawing
» Quantifying size, long/short axis, volume, etc
183
Zebra Medical Vision + Google, EnvoyAI+TeraRecon, Arterys, Fujifilm
Nuance + Partners HC, Philips, Siemens, etc

11. Proposed Processing with NO Human Interaction
185
Deep Learning
Support
10 sec
80 sec
3 min
1 min
10 sec
10 min
RSNA 2018

11. Fully Automated COPD Analysis
No
Human
Interaction
LAA Analysis / LAA Size Analysis
Emphysema
Airway
Vessel
IN/EX
Lung Seg. Lobe Seg.
Airway Seg. Airway Graph
Lumen / Wall Measure
Accept / Reject Lumen / Wall Analysis
Pulmo Vessel Seg. Artery / Vein Separation Vessel Analysis
Air Trapping / IN/EX Parametric MapInspiration / Expiration
Registration
~10 sec ~20 sec
~ 3min ~ 10 sec ~ 10 sec
~20 sec ~5
min
~ 20
min
Use of Deep Learning
10 sec 80 sec
3 min 10 sec 1 min
20 sec 5 min
10 min
3 min
RSNA 2018

11. Validation of Proposed Workflow
0.96
92%
94%
RSNA 2018

11. Big-Data PACS for Quantitative COPD Reading
❖ Auto DICOM retrieve from legacy PACS.
❖ Auto processing supported by Deep-learning.
❖ Stored all the quantification values for every COPD
patient
❖ Dashboard showing the overall statistics.
❖ Query by quantification.
❖ Similar case can be easily queried by quantification
values when reading a patient’s chest CT images.
RSNA 2018

https://physicsml.github.io/pages/papers.html
12. Physics induced ML

https://indico.cern.ch/event/671371/attachments/1535379/2405010/MLsi mulation_GeantV.pdf
12. CERN; A Deep learning tool for fast simulation

12. CNN for Steady Flow Prediction
191
CNN Prediction
LBM
𝑦=f’(x)
https://www.biorxiv.org/content/biorxiv/early/2017/12/30/240317.full.pdf

12. A Deep learning approach to auto-tuning CFD
http://moss.csc.ncsu.edu/~mueller/autotune.html

12. Physics informed ML
Facebook group

EAU 2015, AUA 2015, RSNA 2015, WIP for publication, 비뇨기과 김청수, 경윤수 교수 협업
14. Application : 3DP, AR, Robot

14. Simple Mastoidectomy Surgery Video
Simple Mastoidectomy Surgery Video
Understanding
Surgical Step Understanding
9 Steps
195
Example
AMC ENT Jung JW
Classification algorithm of
surgical stage by learning
video data
중이염 동영상 내
수술도구(위),
Henle recognized
(below)

14. Surgical Robot with AI
Will robot steal surgeons’ job?
NO.
Will robot CHANGE surgeons’ job?
It may...
Will robot and SUPER COMPUTER steal surgeons’ job?
…

14. da Vinci + Deep Learning
199

Sharing, Clouding, Workflow Management, Hybrid Rendering, Mobile Interface
Patient Specific
Precision Model
Image acquisition Segmentation
3D Modeling
(STL)
3DP
3DP workflow for clinical application
Surgery Planning
CT/MRI/3DUS Organ/Disease
Partitioning
3D mesh model Fabrication/polishing
Works
@AMC
Simulation
Guide
Implant
Education/Resear
ch
Patient specific
planning
[In-house SW] [3DP and Material][In-house SW ][ In house SW:AView]
Bioprinting
Workflow of 3DPM
RSNA 2018

The limitations and solutions of 3DP in medicine
201
Limitations Solutions
Slow printing speed New 3DP for 100~1000 times faster
Variations of clinical imaging protocols ;
especially thick slice thickness
Domain adaptation;
Super resolution with DL
Limitation of materials Develop new materials and mechanism
Slow segmentation for specific anatomy Training DL for semantic
segmentation
New manpower Train new expertise with help of AI
Physicians’ additional efforts Communication tools with AI
Already doing well Find new killer apps with 3DP
RSNA 2018

Semantic Segmentation
202
Medical Images
Super Resolution Module
Auto Design Module
GAN
진단용
저해상도 영상
3D모델링용
고해상도 영상
train/test set
Generator Discriminator Y/N
Similar?
Auto Segmentation Module
3D모델링용
고해상도 영상
Segmented
mask
FCN Network
Labeling
Pre-processor
train/test set
End-to-End learning
Segmented
mask
Surgical planner
Ref. tracker Border tracker Surgical
planning
Y/N
meet
Design
spec..?
DeepLearning
RSNA 2018

AI Solution for 3DP
203
Data
acquisition
DL Modeling 3DP
Confirm by
an expert
Confirm by
M.D.
Application
Yes
NoNo
Yes
Segmentation with human-AI interaction
Communication
with AI
Low-resolution image
for diagnosis
High-resolution image for
3DP with super-resolution
Semantic segmentation method
Super-resolution method
▪ Super-resolution convert low-resolution
images into high resolution images.
▪ The result images of super-resolution could
be better 3D models.
Gold Standard 1st training
Segmentation enhancement with human-AI interaction
2nd training 3rd training
AI based 3DP workflow
Manual
segmentation result
Result of 2D U-net
(about 30 sec / case)
Result of 3D U-net
(about 1 sec / case)
RSNA 2018

14. Case Orchestration (Triage)
205
Reading ordering by AI for
efficient reading
Agfa, IBM, Philips, etc

14. Image Captioning
Image Caption Generation
Video Caption Generation

14. Generated annotation : Mesh terms in radiologic
report
- Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-
Fushman, Jianhua Yao, Ronald M Summers, "Learning to
Read Chest X-Rays: Recurrent Neural Cascade Model for
Automated Image Annotation," CVPR 2016.

14. Audio data @ Hospital
Nuance Revenue
2016년
$1,502.3 M
AMC audio dictation
Radiology : 0.5M /y
Nuclear Medicine : 0.2M /y
Pathology : 0.3M /y
US, Endoscopy, Nursing, Out Patients, OR : +5M/y
Pathology, Radiology etc.
Recognition during reading process
The medical typist confirms the recorded
contents and hand-typed the data.
Considerable manpower and time, and
the patient is delayed in confirming the
results.

14. Speech Recognition
뷰노,

14. Beyond computer vision
210
Computer vision to discern clinical behaviors
Bedside Computer Vision, Yeung S, Fei-Fei Li et al, NEJM 2018 April
H Rosette **수술장의 단위(꽃잎이 방사상으로 난 모양), 가운데를 중심으로 수술실 배열

14. Hospital Workflow Improvement
Intelligent Hand
Hygiene Support
@ Lucile Packard Children's
Hospital at Stanford
Intelligent ICU Clinical
Pathway Support
@ Stanford Hospital and Clinics
@ Intermountain Healthcare
Intelligent Senior Wellbeing
Support
@ Palo Alto Medical Foundation
@ On Lok Senior Health Services
@ Intermountain Healthcare
** Stanford AI-assisted Care
Courtesy of Ha HS, VAIIM consulting, modified

Artificial Intelligence (+Big Data) Will Redesign
Healthcare
1. Artificial Intelligence (x) ->Augmented Intelligence (o)
2. Just Starting
3. Precision medicine / Mining medical records / Designing
treatment plans
4. Getting the most out of in-person and online consultations /
Health assistance and medication management / Open AI
helping people make healthier choices and decisions
5. Assisting repetitive jobs
6. Drug discovery / Clinical trial Case Matching
7. Analyzing, redesigning a healthcare system
http://medicalfuturist.com/ Aug, 2016

[Parallel Worlds, Michio Kaku]
Temperature Precipitation
Wind …
Weather
E1 E2
E3 Ex
Longitude
C1
Landscape
C2
Altitude
C3
…
C4
!!
Disease
Predictability
??
Prediction (Complexity)

Collaborators
Radiology
Joon Beom Seo, SangMin LeeA,B, Dong Hyun, Yang, Hyung Jin Won, Ho Sung Kim, Seung
Chai Jung, Ji Eun Park, So Jung Lee,Jeong Hyun Lee, Gilsun Hong
Neurology
Dong-Wha Kang, Chongsik Lee, Jaehong Lee, Sangbeom Jun, Misun Kwon, Beomjun Kim
Cardiology
Jaekwan Song, Jongmin Song, Young-Hak Kim
Emergency Medicine
Dong-Woo Seo
Pathology
Hyunjeong Go, Gyuheon Choi, Gyungyub Gong, Dong Eun Song
Surgery
Beom Seok Ko, JongHun Jeong, Songchuk Kim, Tae-Yon Sung
Internal Medicine
Jeongsik Byeon, Kang Mo Kim
Anesthesiology
Sung-Hoon Kim, Eun Ho Lee

IFMIA 2019 Plenary Talk : Deep Learning in Medicine; Engineers' Perspectives

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IFMIA 2019 Plenary Talk : Deep Learning in Medicine; Engineers' Perspectives

Similar to IFMIA 2019 Plenary Talk : Deep Learning in Medicine; Engineers' Perspectives (20)

More from Namkug Kim

More from Namkug Kim (16)

Recently uploaded

Recently uploaded (20)

IFMIA 2019 Plenary Talk : Deep Learning in Medicine; Engineers' Perspectives