Search images by similarity with Deep Learning, Transfer Learning, Pre-trained models, Tensorflow, Keras and Scikit-Learn. Talk given for SF Python Meetup project night, March 2019. Check out our bootcamp: https://bootcamp.zerotodeeplearning.com

1. Catalit LLC
IMAGE SIMILARITY SEARCH
Data Weekends
Catalit
Data Science
Zero to
Deep Learning
Francesco Mosconi
Yelp / Mar 2019

2. Catalit LLC
ABOUT US
www.catalit.com
Data Weekends
Catalit
Data Science
Zero to
Deep Learning
www.dataweekends.com bootcamp.zerotodeeplearning.com

3. Catalit LLC
TODAY
• Images and CNN crash course
• Transfer learning
• Image similarity

4. Catalit LLC
IMAGES AND CNN

5. Catalit LLC
Caption generation
Object recognition
http://www.image-net.org/challenges/LSVRC/

6. Catalit LLC
APPLICATIONS
Speech recognition and synthesis
http://venturebeat.com/2015

7. Catalit LLC
APPLICATIONS
Neural MachineTranslation
https://devblogs.nvidia.com/parallelforall/ https://smallbiztrends.com/2017/

8. Catalit LLC

9. Catalit LLC
https://deepmind.com www.nervanasys.com

10. Catalit LLC
https://www.pcper.com/category/tags/deep-neural-network

11. Catalit LLC
ML ON IMAGES

12. Catalit LLC
FULLY CONNECTED
…
xN
x1
b
Input Output
b
b
…
b
b
Layer 1 Layer L
b
b
b
…
b
Layer 2
y
…

13. Catalit LLC
LOCAL PATTERNS

14. Catalit LLC
STACKING LAYERS
Conv ConvPooling Pooling Fully connected
ClassificationFeature Extraction

15. Catalit LLC

16. Catalit LLC
1.00 0.33
MAX POOLING
0.77 -0.11 0.11 0.33 0.55 -0.11 0.33
-0.11 1.00 -0.11 0.33 -0.11 0.11 -0.11
0.11 -0.11 1.00 -0.33 0.11 -0.11 0.55
0.33 0.33 -0.33 0.55 -0.33 0.33 0.33
0.55 -0.11 0.11 -0.33 1.00 -0.11 0.11
-0.11 0.11 -0.11 0.33 -0.11 1.00 -0.11
0.33 -0.11 0.55 0.33 0.11 -0.11 0.77
maximum

17. Catalit LLC
Convolution
ReLU
Pooling
Convolution
ReLU
Convolution
ReLU
Pooling
Fully
connected
Fully
connected
STACK
Feature Extraction Classification
…
0
1
9
Better features than pixels

18. Catalit LLC
FEATURE EXTRACTION
-1 -1 -1 -1 -1 -1 -1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 -1 -1 1 -1 -1 -1 -1
-1 -1 -1 1 -1 1 -1 -1 -1
-1 -1 1 -1 -1 -1 1 -1 -1
-1 1 -1 -1 -1 -1 -1 1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1
Convolution
ReLU
Pooling
Convolution
ReLU
Convolution
ReLU
Pooling
1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
Layers can be repeated several (or many) times.

19. Catalit LLC
FLATTEN1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55

20. Catalit LLC
FULLY CONNECTED LAYER
X
O
1.00 0.55
0.55 1.00
0.55 1.00
1.00 0.55
1.00 0.55
0.55 0.55
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55

21. Catalit LLC
PRE-TRAINED MODELS

22. Catalit LLC
IMAGENET
• 1000 classes
• 1.2 M train images
• 50k validation images
• 100k test images

23. Catalit LLC
VGG16 &VGG19
https://www.cs.toronto.edu/~frossard/post/vgg16/

24. Catalit LLC
RESNET
• Much deeper model
• Much smaller (100 Mb)
• Trains faster

25. Catalit LLC
INCEPTIONV3

26. Catalit LLC
XCEPTION

27. Catalit LLC
COMPARISON

28. Catalit LLC
BOTTLENECK FEATURES
What comes out here!

29. Catalit LLC
IMAGE SIMILARITY SEARCH

30. Catalit LLC
XCEPTION
Input
Image
Base
Model
Bottleneck
Features
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55
Formula
racing
Beach
volleyball
Cross-country
skiing
0.93
0.06
0.01
Label
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55
Dense +
Softmax
Feature Extraction Classification

31. Catalit LLC
XCEPTION
Input
Image
Base
Model
Bottleneck
Features
1.00
0.55
0.55
1.00
1.00
0.55
0.55
0.55
0.55
1.00
1.00
0.55
Feature Extraction
Use distance measure to
find images with
similar bottlenecks

32. Catalit LLC
DISTANCE & SIMILARITY

33. Catalit LLC
PAIRWISE DISTANCES
• Measure distance between bottleneck features
• Rank images by closest image

34. Catalit LLC
github.com/zerotodeeplearning
• deep_learning_image_similarity
• ztdl-5-day-bootcamp

35. Catalit LLC
ZTDL BOOTCAMP
bootcamp.zerotodeeplearning.com
Code: PYTHONSF10 for 10% OFF