This document summarizes an upcoming training session on advanced Keras techniques. The session will recap deep learning and Keras basics, then cover callbacks and multiple inputs, multi-GPU setups, using pre-trained models, and transfer learning. Attendees can expect an interactive and fun learning experience with accelerated learning and opportunities to take notes. The session will include tutorials demonstrating Keras callbacks, multi-GPU usage, pre-trained models from ImageNet, and performing transfer learning.

1. Catalit LLC
ADVANCED KERAS
Francesco Mosconi
Data Weekends Catalit LLC

2. Catalit LLC
THIS SESSION
• Recap Deep Learning
• Keras Recap
• Callbacks and Multiple Inputs
• Multi GPU
• Pretrained Models
• Transfer Learning

3. Catalit LLC
WHATTO EXPECT
• Accelerated learning
• Interactive
• Fun / Discovery
• Take notes

4. Catalit LLC
KERAS

5. Catalit LLC
KERAS => DEEP LEARNING
1.
Collection
2.
Processing
3. Model
Building
4.
Evaluation
5.
Deployment

6. Catalit LLC
Caption generationObject recognition
http://www.image-net.org/challenges/LSVRC/

7. Catalit LLC
MACHINETRANSLATION
https://devblogs.nvidia.com/parallelforall/ https://smallbiztrends.com/2017/

8. Catalit LLC
DEEP LEARNING
Deep Learning
Traditional
Machine Learning
# training data
Performance
Feature
extract
OutputFeatures
Shallow
ML algo
Input Deep Learning Output
Input

9. Catalit LLC
KERAS
• User friendly
• Modular
• Extensible
• Models are Python

10. Catalit LLC
BACKENDS
• Theano
• Tensorflow
• Microsoft CNTK
• more coming …
SAME KERAS CODE!

11. Catalit LLC
NEURAL NETWORKS

12. Catalit LLC
PERCEPTRON

13. Catalit LLC
DEEP NETWORK

14. Catalit LLC
REALLY DEEP NETWORKS

15. Catalit LLC
MASTER FORMULA
model = data + structure + loss + optimizer

16. Catalit LLC
LAYERS
• Fully connected
• Convolutional
• Recurrent
• …

17. Catalit LLC
MODULES
• layers
• optimizers
• activations
• losses
• datasets
• metrics
• initializers
• …

18. Catalit LLC
TUTORIAL 1

19. Catalit LLC
KERAS CALLBACKS

20. Catalit LLC
CALLBACKS
• Logging (BaseLogger, ProgbarLogger. RemoteMonitor)
• Stopping (EarlyStopping,TerminateOnNaN)
• Checkpoints
• Tensorboard
• LearningRateScheduler
• …
https://keras.io/callbacks/

21. Catalit LLC
EARLY STOPPING
• Save model
• Restore model
Wikimedia Commons

22. Catalit LLC
CHECKPOINTS
• Save model
• Restore model

23. Catalit LLC
TENSORBOARD

24. Catalit LLC

25. Catalit LLC

26. Catalit LLC

27. Catalit LLC
TUTORIAL 2

28. Catalit LLC
MULTI GPU

29. Catalit LLC
PARALLELISM
• Data Parallel
• Device Parallel

30. Catalit LLC
DATA PARALLEL
https://img.clipartfest.com
MASTER
GPU
GPU
GPU
GPU

31. Catalit LLC
DATA PARALLEL
https://keras.io/getting-started/faq/#data-parallelism

32. Catalit LLC
DEVICE PARALLEL

33. Catalit LLC
DATA PARALLEL
https://keras.io/getting-started/faq/#data-parallelism

34. Catalit LLC
TUTORIAL 3

35. Catalit LLC
PRE-TRAINED MODELS

36. Catalit LLC
IMAGENET
• 1000 classes
• 1.2 M train images
• 50k validation images
• 100k test images

37. Catalit LLC
VGG16 &VGG19
https://www.cs.toronto.edu/~frossard/post/vgg16/

38. Catalit LLC
VGG16 &VGG19
https://arxiv.org/abs/1409.1556

39. Catalit LLC
VGG
• Slow to train
• Large (500+ MB) => not easy to deploy

40. Catalit LLC
RESNET
• Much deeper model
• Much smaller (100 Mb)
• Trains faster

41. Catalit LLC
INCEPTIONV3

42. Catalit LLC
INCEPTION MODULE

43. Catalit LLC
XCEPTION

44. Catalit LLC
COMPARISON

45. Catalit LLC
TUTORIAL 4

46. Catalit LLC
TRANSFER LEARNING

47. Catalit LLC
PRE-TRAINED MODEL

48. Catalit LLC
FREEZE
Freeze

49. Catalit LLC
BOTTLENECK FEATURES
What comes out here!

50. Catalit LLC
FULLY CONNECTED
…
xN
x1
b
Bottleneck Output
b
b
…
b
b
Layer 1 Layer L
b
b
b
…
b
Layer 2
y
…

51. Catalit LLC
FINETUNING
Freeze

52. Catalit LLC
TUTORIAL 5

53. Catalit LLC
SWITCH BACKEND

54. Catalit LLC
~/.keras/keras.json
{
"image_data_format": "channels_last",
"epsilon": 1e-07,
"floatx": "float32",
"backend": "tensorflow"
}

55. Catalit LLC
~/.keras/keras.json
{
"image_data_format": "channels_last",
"epsilon": 1e-07,
"floatx": "float32",
"backend": "cntk"
}

56. Catalit LLC
THANKYOU!
Francesco Mosconi
www.dataweekends.com www.catalit.com
Data Weekends Catalit LLC