Jack Burdick, Oge Marques, Borko Furht, Janet Weinthal (FAU, Boca Raton, FL, USA), Adria Romero Lopez, and Xavier Giro-i-Nieto (UPC, Barcelona). Presented at SIIM 2017 Annual Meeting: http://siim.org/page/17s_analytics3

1. #SIIM17
The Impact of Segmentation on the
Accuracy and Sensitivity of a
Melanoma Classifier
Based on Skin Lesion Images
Oge Marques, PhD
Professor
College of Engineering and Computer Science
Florida Atlantic University

2. #SIIM17
Our Team
Adrià Romero López Xavier Giró-i.Nieto
Image Processing Group
Signal Theory and
Communications Department
MIDDLE Research Group
Oge Marques Borko Furht Jack Burdick Janet Weinthal
NSF Award No. 1464537, I/UCRC Phase II under NSF 13-542

3. #SIIM17
Outline
• Motivation
• Context
• Scope and goals
• Challenges
• State of the art
• Our work
• Hypothesis
• Methodology
• Experimental results
• Ongoing and future work
• Concluding remarks

4. #SIIM17
Motivation
[Source: https://seer.cancer.gov/statfacts/html/melan.html]
Melanoma of the skin

5. #SIIM17
Context
• This is not a typical SIIM presentation
• No specialized imaging equipment
• No PACS
• No DICOM
• No metadata
• No workflows
• Instead...
• Regular photographs
• Unstructured (and purely visual) data
• Minimal ground truth

6. #SIIM17
Scope
• Skin Disease: An
Illustrated Taxonomy
• Our focus: skin
lesion analysis for
(early) melanoma
detection
[Source: Esteva et al., Nature (2017)]

7. #SIIM17
Scope and Goals
• Scope:
• Help physicians to detect melanoma (a 2-class
classifier)
• Goals:
• Design an intelligent medical imaging skin lesion
diagnosis system using deep learning techniques
• Achieve (or improve upon) state-of-the-art results for
skin lesion classification

8. #SIIM17
A Challenging Problem
8

9. #SIIM17
Success Rate - Sensitivity
Physicians, as low as:
• 43 % - naked eye
• 79 % - dermoscopy
(Vestergaard et al, 2008)
Deep learning based methods, as high as:
• 94.83 %
(Jafari et al, 2016; Premaladha and Ravichandran 2016)

10. #SIIM17
Classical Techniques
[Source: Center For Excellence In Dermatology - Kennewick, WA]
CASH (Henning et al., 2007)
Color
Architecture
Symmetry
Homogeneity

11. #SIIM17
Typical workflow (pre-DL)
Source: [Glaister2013]
11

12. #SIIM17
Challenges
• Segmentation
• Artifacts, such as: freckles, hair, shading, and noise.
• Irregular, fuzzy lesion border
• Low contrast between lesion and surrounding skin
• Multi-fragment lesions
• Classification
• Manual feature extraction is not effective
• False positives and false negatives have different impact

13. #SIIM17
State of the art
• Esteva et al. Nature 2017

14. #SIIM17
Our Approach
• Pretrained VGG16
(Simonyan & Zisserman, 2014)
• Transfer Learning
[Source:Blier 2016]

15. #SIIM17
Transfer Learning
Source: [http://www.slideshare.net/hunkim/transfer-defectlearningnew-completed]

16. #SIIM17
Transfer Learning
1. Train on
Imagenet
3. Medium dataset:
finetuning
more data = retrain more of
the network (or all of it)
2. Small dataset:
feature extractor
Freeze these
Train this
Freeze these
Train this
Slide credit: Bay Area Deep Learning School Presentation by A. Karpathy
Medical Imaging case

17. #SIIM17
Prediction Examples:
Getting It Right
True Positives True Negatives

18. #SIIM17
Prediction Examples:
Getting It Wrong
False Positives False Negatives

19. #SIIM17
Our Hypothesis
• Image segmentation improves the performance of skin lesion
classifiers using convolutional neural networks.
[Source:International Skin Imaging Collaboration Archive]
Not segmented
Perfectly segmented
Partially segmented

20. #SIIM17
Methods
• ISBI 2016 Challenge dataset
• Skin Lesion Analysis towards melanoma detection
• 1279 RGB images
• Labeled as either benign or malignant
Class
Benign Malignant Total Images
Training subset 727 173 900
Testing subset 304 75 379

21. #SIIM17
Methods
• Dataset balancing through downsampling.
• Dataset split: 70-30% training/testing
• Input images:
• Unsegmented images: straight from the dataset.
• Perfectly segmented images: bitwise AND operation of the unaltered
images and its corresponding binary mask provided by the ISIC
dataset.
• Partially segmented images: original binary masks morphologically
dilated with a disk-shaped structuring element (50 pixel radius).
• Additional preprocessing methods (resizing and normalization) were
also performed to match the input size expected by the VGG16
architecture.

22. #SIIM17
Results

23. #SIIM17
Further Investigation
• What if we vary the degree of border expansion?
Sensitivity Accuracy AUC
Perfect Segmentation 45.3% 58.7% 62.2%
+25 53.3% 61.3% 64.2%
+50 56.0% 60.7% 62.6%
+75 57.3% 59.3% 60.8%
+100 34.7% 55.3% 57.9%
Unsegmented 24.0% 51.3% 53.2%

24. #SIIM17
Ongoing and Future Work
•Additional / larger / more challenging datasets
•Other CNN architectures
•Better image preprocessing
•Partnerships and collaborations
•Mobile app

25. #SIIM17
Concluding remarks
• Challenges
• Difficulty in acquiring datasets and reproducing / benchmarking results
• The “black box” aspect of DL-based solutions
• Hard to tell positives from negatives
• Learning curve: TensorFlow, Keras, HPC, DL concepts and best practices, etc.
• Opportunities
• Many variations of the basic classification problem
• Mobile app market
• Tech-minded dermatology practices

26. #SIIM17