3. ▪ Generative
–Can generate samples
▪ Adversarial
–Trained by competing each other
▪ Networks
–Use neural networks
Definition
Generative Adversarial Networks
4. ▪ Discriminative Models
–Given X, predict Y
–We learn P(Y | X) directly
–We cannot can generate samples
–Examples
• Logistic regression, SVM, CRF, Decision trees
▪ Generative models
–Given X, predict P(X|Y)
–We learn P(X, Y)
–We can generate samples from P(X)
–Examples
• Markov chains, Naïve Bayes, GMM
Discriminative Models vs Generative Models
Generative Models?
Richard Feynmann
5. ▪ Adversarial Training between Discriminator vs Generator
–Discriminator is a classifier that determines whether given images is real data from the
world or fake data generated by generator.
–Proposed by Ian Goodfellow(NIPS 2014)
Discriminator vs Generator
Adversarial Networks?
https://www.oreilly.com/learning/generative-adversarial-networks-for-beginners?imm_mid=0f6436&cmp=em-data-na-na-newsltr_ai_20170918
VS
6. ▪ Alternate the training of discriminator and generator until convergence(may not happen)
Overview
Training GAN
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
7. ▪ Fix the generator, then train discriminator to distinguish samples of real images from
samples generated by the generator
Discriminator Training
Training GAN
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
8. ▪ Fix the discriminator, then train generator from the feedback of discriminator using
samples generated by the generator
Generator Training
Training GAN
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
9. ▪ Training GAN is a minmax problem where
–The discriminator D tries to maximize its classification accuracy
–The generator G tries to minimize the discriminator’s classification accuracy
–The optimal solution for D
–The optimal solution for G
Mathematical Formulation
Training GAN
Iam Goodfellow et. al, NIPS 2014
Maximized by D Minimized by G
(in practice, we maximize )
11. ▪ With the optimal discriminator, training GAN is equivalent to minimizing Jensen-Shannon
divergence as
What happens during the training of GAN?
Why GAN Works?
http://videolectures.net/site/normal_dl/tag=1129740/deeplearning2017_courville_generative_models_01.pdf
Christian Leidig et. al, CVPR 2017
the only solution is
12. Visualization of samples generated from trained Generator
Generating Samples from GAN
Iam Goodfellow et. al, NIPS 2014
18. ▪ Replace discriminator loss from binary cross entropy to least square loss
Least Square GAN(LS-GAN)
Improving Loss Function of GAN
Xudong Mao et. al, arXiv 2016
They don’t move.
0
1
19. ▪ Is JS divergence good enough to train GAN?
Wasserstein GAN(WGAN)
Improving Loss Function of GAN
Martin Arjovsky et. al, arXiv 2016
Continuous everywhere and differentiable almost everywhere
Vanishing gradient and bad convergence
20. ▪ WGAN objective function
▪ How can we efficiently enforce the Lipschitz
constraint on the critic D ?
–Weight clipping
Wasserstein GAN(WGAN)
Improving Loss Function of GAN
𝐷 𝑥1 − 𝐷 𝑥2 ≤ 𝐾 𝑥1 − 𝑥2 , K=1
Martin Arjovsky et. al, arXiv 2016
21. ▪ Learning curve and sample quality
Wasserstein GAN(WGAN)
Improving Loss Function of GAN
Vanilla GAN
WGAN
Martin Arjovsky et. al, arXiv 2016
22. ▪ Issues with Weight Clipping
–Fail to capture higher moments of the data distribution
–Either exploding or vanishing gradient
Wasserstein GAN with Gradient Penalty(WGAN-GP)
Improving Loss Function of GAN
Weight
Clipping
Gradient
Clipping
Ishaan Gulrajani et. al, NIPS 2017
23. ▪ WGAN with Gradient Penalty
–Penalize the norm of the gradient instead of clipping the weights of critics
Wasserstein GAN with Gradient Penalty(WGAN-GP)
Improving Loss Function of GAN
Ishaan Gulrajani et. al, NIPS 2017
24. ▪ Comparison of various loss functions and their sample quality
Wasserstein GAN with Gradient Penalty(WGAN-GP)
Improving Loss Function of GAN
Ishaan Gulrajani et. al, NIPS 2017
26. ▪ Conditioning the model on additional information for better multi-modal learning
Conditional GAN(CGAN)
Conditional Generation
Mirza et. al, arXiv 2014
https://github.com/hwalsuklee/tensorflow-generative-model-collections
27. ▪ Disentangle individual dimensions in latent vector for capturing key attributes
InfoGAN
Conditional Generation
Xi Chen et al arXiv 2016
https://github.com/hwalsuklee/tensorflow-generative-model-collections
29. ▪ Use both real/fake and label classifiers for discriminator training
Auxiliary Classifier GAN(AC-GAN)
Conditional Generation
Augustus Odena et al arXiv 2016
https://github.com/hwalsuklee/tensorflow-generative-model-collections
31. ▪ “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”
–Perceptual Loss
GAN for Image Super Resolution
SRGAN
Chrisitan Leidig et al, CVPR 2017
32. ▪ Super Resolution Results
GAN for Image Super Resolution
SRGAN
Chrisitan Leidig et al, CVPR 2017
33. ▪ Super Resolution Results
GAN for Image Super Resolution
SRGAN
Chrisitan Leidig et al, CVPR 2017
34. ▪ “Image-to-Image Translation with Conditional Adversarial Networks”
Conditional GAN for Image Domain Transfer
Image-to-Image Translation
Phillip Isola et al, CVPR 2017
35. ▪ Image Translation Results
Conditional GAN for Image Domain Transfer
Image-to-Image Translation
Phillip Isola et al, CVPR 2017
36. ▪ Image Translation Results
Conditional GAN for Image Domain Transfer
Image-to-Image Translation
Phillip Isola et al, CVPR 2017
37. ▪ “Generative Adversarial Text to Image Synthesis”
–Generator is conditioned on text embedding
–Discriminator uses both visual and textual features by concatenation
Conditional GAN for Text to Image Translation
Text2Image
Scott Reed et al, ICML 2016
38. ▪ Text-to-Image Translation Results
Conditional GAN for Text to Image Translation
Text2Image
Scott Reed et al, ICML 2016
39. ▪ “Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks”
–Generate high-resolution images from text by stacking conditional GANs
Stacked Conditional GAN for Text to Image Translation
StackGAN
Han Zhang et al, ICCV 2017
40. ▪ Text-to-Image Translation Results
Stacked Conditional GAN for Text to Image Translation
StackGAN
Han Zhang et al, ICCV 2017
41. ▪ Text-to-Image Translation Results
Stacked Conditional GAN for Text to Image Translation
StackGAN
Han Zhang et al, ICCV 2017
42. ▪ “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”
Unpaired Image Translation
CycleGAN
Jun-Yan Zhu et al, ICCV 2017
43. ▪ Paired/Unpaired and Generated Image/Reconstruction
Unpaired Image Translation
CycleGAN
Jun-Yan Zhu et al, ICCV 2017
44. ▪ Image Translation Example
Unpaired Image Translation
CycleGAN
Jun-Yan Zhu et al, ICCV 2017
45. ▪ “StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation”
–Instead of training generators for each source-domain pair, we can train one generator handling
multiple domains to utilize all training dataset for all domain pairs and even for different dataset
Multi-domain Image-to-Image Translation with a Single Network
StarGAN
for real
for fake
Yunjey Choi et. al, CVPR 2018
46. ▪ Multi-domain Translation Result
Multi-domain Image-to-Image Translation with a Single Network
StarGAN
Yunjey Choi et. al, CVPR 2018
47. ▪ Use of ‘mask vector’ to jointly train multiple dataset
Multi-domain Image-to-Image Translation with a Single Network
StarGAN
Yunjey Choi et. al, CVPR 2018
48. ▪ Use of ‘mask vector’ to jointly train multiple dataset
Multi-domain Image-to-Image Translation with a Single Network
StarGAN
Yunjey Choi et. al, CVPR 2018
49. ▪ Multi-domain Translation Results
Multi-domain Image-to-Image Translation with a Single Network
StarGAN
Yunjey Choi et. al, CVPR 2018
50. ▪ “Learning from Simulated and Unsupervised Images through Adversarial Training”
Refining Simulated Image for Data Augmentation
SimGAN
Ashish Shrivastava et. al, CVPR 2017
Self-regularization loss
51. ▪ “Learning from Simulated and Unsupervised Images through Adversarial Training”
Refining Simulated Image for Data Augmentation
SimGAN
Ashish Shrivastava et. al, CVPR 2017
Visual Turing Test d=7 degree
53. –In many cases, data augmentation techniques used in natural images does not
semantically make sense in medical image
(flips, rotations, scale shifts, color shifts)
–Physically-plausible deformations or morphological transform can be used in limited
cases.
–More augmentation choices for texture classification problems.
Data Augmentation for Effective Training Set Expansion
Medical Data Generation
Source : H. R. Roth et. al., MICCAI, 2015
54. Generation of synthetic dataset for dataset expansion
Medical Data Generation
M.J.M. Chuquicusma, ISBI 2018
▪ “How To Fool Radiologists with Generative Adversarial Networks? A Visual Turing Test For
Lung Cancer Diagnosis”
55. Generation of synthetic dataset for dataset expansion
Medical Data Generation
M.J.M. Chuquicusma, ISBI 2018
▪ Visual Turing Test
56. ▪ “Synthetic Medical Images from Dual Generative Adversarial Networks”
Generation of Fundus Image using Dual GANs
Medical Data Generation
John T. Guibas et. al, NIPS Workshop 2017
57. ▪ “Synthetic Medical Images from Dual Generative Adversarial Networks”
Generation of Fundus Image using Dual GANs
Medical Data Generation
John T. Guibas et. al, NIPS Workshop 2017
58. ▪ “Synthetic Data Augmentation using GAN for Improved Liver Lesion Classification”
Liver Lesion Generation for Data Augmentation
Medical Data Generation
Maayan Frid-Ada et. al, ISBI 2018
Real Synthetic
59. ▪ “Generalization of Deep Neural Networks for Chest Pathology Classification in X-ray using
Generative Adversarial Networks”
Chest X-ray Generation for Data Augmentation
Medical Data Generation
Hojjat Salehinejad et. al, ICASSP 2018
60. ▪ Generated Samples and Performance Improvement by Synthetic Augmentation
Chest X-ray Generation for Data Augmentation
Medical Data Generation
Hojjat Salehinejad et. al, ICASSP 2018
61. ▪ “Visual Feature Attribution using Wasserstein GANs “
VA-GAN : Understanding Visual Feature Attribution for Alzheimer Disease
Visual Feature Attribution
Christian F. Baumgatrner et. al, arXiv 2018
Overall Objective
GAN Loss Term
RegularizationTerm
62. ▪ “Visual Feature Attribution using Wasserstein GANs “
VA-GAN : Understanding Visual Feature Attribution for Alzheimer Disease
Visual Feature Attribution
Christian F. Baumgatrner et. al, arXiv 2018
63. Enhance the quality of low-dose CT to normal-dose CT
Synthesis or Enhancement of Medical Image
Dong Nie et. al, MICCAI 2017
▪ “Medical Image Synthesis using Context-aware Generative Adversarial Networks”
64. Unpaired image translation from MR to CT using CycleGAN
Synthesis or Enhancement of Medical Image
Jelmer M. Wolterink et. al, MICCAI 2017 Workshop
Setting Mean Absolute Error
Paired Voxel-wise Loss 89.4 ± 6.8 HU
Unpaired Cycle Consistency Loss 73.7 ± 2.3 HU
▪ “Deep MR to CT Synthesis using Unpaired Data”
65. Enhance the quality of low-dose CT to normal-dose CT
Synthesis or Enhancement of Medical Image
J. M. Wolterink et. al, IEEE Trans. Medical Imaging
▪ “Generative Adversarial Networks for Noise Reduction in Low-Dose CT”
66. Deep learning for undersampled MRI reconstruction
Synthesis or Enhancement of Medical Image
Tran Min Quan et. al, arXiv 2018
▪ “Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss”
–Problem of compressed sensing MRI reconstruction
–CS-MRI using GAN objective
67. Deep learning for undersampled MRI reconstruction
Synthesis or Enhancement of Medical Image
Tran Min Quan et. al, arXiv 2018
▪ “Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss”
68. Deep learning for undersampled MRI reconstruction
Synthesis or Enhancement of Medical Image
Tran Min Quan et. al, arXiv 2018
▪ Experimental Results for Understampled MRI Reconstruction using GAN
69. Generation of segmentation mask undistinguishable from physician’s mask
Physician Friendly Loss for Segmentation
Source : P. Costa(2017), VUNO(2017)
▪ “Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks”
70. ▪ “Adversarial Networks for the Detection of Aggressive Prostate Cancer”
Generation of segmentation mask undistinguishable from physician’s mask
Physician Friendly Loss for Segmentation
Source : S. Kohl et. al(2017)
71. ▪ “SegAN : Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation”
SegAN : Adversarial Network with Multi-scale Loss
Physician Friendly Loss for Segmentation
Yuan Xue et. al, arXiv 2017
72. ▪ “Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-
Consistency Generative Adversarial Network”
Segmentation of Multimodal Images using Image-to-Image Translation
Multimodal Image Segmentation
Zizhao Zhang et. al, CVPR 2018
73. ▪ “Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-
Consistency Generative Adversarial Network”
Segmentation of Multimodal Images using Image-to-Image Translation
Multimodal Image Segmentation
Zizhao Zhang et. al, CVPR 2018
75. ▪ Image Generation
– Generation of rare cases
– Understanding latent structure of lesions
– Improving the performance of diagnostic models
▪ Image Synthesis and Translation
– Noise reduction, modality translation
– Accelerating image acquisition time
– Improving diagnostic performance
▪ Lesion Detection and Segmentation
– More physician friendly training
– Better performance for lesions or organs with complex structure
▪ Future of GANs in Medical Imaging
– More GANs to come in medical imaging with clinical and commercial values
More GANs in Medical Imaging
Future of GANs