Nowadays, Deep neural networks are the most popular approach which we see its usage in different applications and tasks. As day growth its usage in different tasks, checking the vulnerability of these networks is being a very important fundamental issue. Therefore, analyzing of each machine learning model (such as neural network) for its vulnerability, is a useful task to assess the usage of that in critical situations. In this session, We try to cover the key definition step's of vulnerability of deep neural networks and its defense strategies against simplest vulnerability at first. Then when the minds are boiled, we try to implement and test them in a practical manner. Also, covering a teamwork remote session for more collaboration is available at the end of the session. (Winter Seminar Series - WSS - Mohammad Khalooei - Sharif University of Technology)

1. Robustness of Deep Neural Networks
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir 1 of 80

2. Robustness of Deep Neural Networks
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir 2 of 80

3. • Decision-boundary crossing
• Vulnerability of deep neural networks in different data
• Different attacks and penetration ways of ML & DL
approaches
• Attack Toolboxes
• Different defense strategies for DNN
• Tips to stay safe in DNN
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Outlines
3 of 84

4. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Data
https://www.slideshare.net/BrianKim244/dcgan-77452250
𝑑2
𝑑1
c
4 of 84

5. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Data
https://www.slideshare.net/BrianKim244/dcgan-77452250
c
𝑑1 × 𝑑2 × 𝑐
0.975 , 0.2, 0.0023, 0.5, ….. , 0.5156
𝑑2
𝑑1
5 of 84

6. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Data
6 of 84

7. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Data
https://www.slideshare.net/BrianKim244/dcgan-77452250
7 of 84

8. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Life of the points …
https://www.slideshare.net/khalooei/life-of-points-machine-learning-with-orange-flavor
https://ceit.aut.ac.ir/~khalooei/presentations/
https://www.slideshare.net/khalooei/life-of-points-machine-learning-with-orange-flavor
8 of 84

9. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Discriminative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
9 of 84

10. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Discriminative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
10 of 84

11. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Discriminative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
11 of 84

12. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Discriminative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
12 of 84

13. Generative Model
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Generative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
TrainingData
Training
Generated Data
Unseen new data
13 of 84

14. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Generative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
Distribution of the actual images
14 of 84

15. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Generative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
15 of 84

16. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Generative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
16 of 84

17. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Generative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
17 of 84

18. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review:: Generative Model
https://www.slideshare.net/BrianKim244/dcgan-77452250
Distribution of the actual images
18 of 84

19. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Decision boundary crossing
https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
19 of 84

20. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Decision boundary crossing
http://www.cse.chalmers.se/~richajo/dit866/lectures/l3/Plotting%20decision%20boundaries.html
Decision Tree Perceptron
20 of 84

21. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Vulnerability of Deep Neural Networks
https://www.euclidean.com/deep-learning-and-value-investing
21 of 84

22. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
AI and the top innovation
https://www.usenix.org/sites/default/files/conference/protected-files/enigma17_slides_papernot.pdf
22 of 84

23. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
AI and the top innovation
23 of 84

24. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
AI and the top innovation
24 of 84

25. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Machine learning …
Adversarial Machine Learning,
Author(s): Anthony D. Joseph, Blaine Nelson, Benjamin I. P. Rubinstein, J. D. Tygar
Publisher: Cambridge University Press, Year: 2019
25 of 84

26. Fooling Google's image-
recognition AI 1000x faster
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
On the line of recent researches …
`
December 20 '17
26 of 84

27. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
On the line of recent researches …
https://syncedreview.com/2019/04/24/now-you-see-me-now-you-dont-fooling-a-person-detector/
27 of 84

28. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
On the line of recent researches …
https://towardsdatascience.com/evasion-attacks-on-machine-learning-or-adversarial-examples-12f2283e06a1
28 of 84

29. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
An interesting usage :: Google verification!
29 of 84

30. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
On the line of recent researches …
Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
Nicholas Carlini, David Wagner
Deep Learning and Security Workshop, 2018. Best Paper
30 of 84

31. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
On the line of recent researches …
Generating Natural Language Adversarial Examples
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, Kai-Wei Chang
Published in EMNLP 2018
DOI:10.18653/v1/d18-1316
31 of 84

32. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Delving into the problem …
32 of 84

33. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Motivation
33 of 84

34. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Motivation
Gradient-descent
34 of 84

35. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Motivation
Gradient-descent
35 of 84

36. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Motivation
Gradient-descent
36 of 84

37. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
https://www.slideshare.net/DavidKim486/universal-adversarial-perturbation
37 of 84

38. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
What is Adversarial Example?
𝑋 − ෠𝑋 𝑝 < 𝜀
𝑋
෠𝑋
𝑋 ෠𝑋
39 of 84

39. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Examples of Adversarial Example
https://www.kdnuggets.com/2018/10/adversarial-examples-explained.html
?!
?!
40 of 84

40. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
From WWW
41 of 84

41. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir 42 of 84

42. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Perturbation's effect on class distributions
43 of 84

43. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
https://www.slideshare.net/DavidKim486/universal-adversarial-perturbation
44 of 84

44. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Examples from Overfitting
• Adversarial Examples rooted in :
• Overfitting
• Excessive Linearity
45 of 84

45. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Examples from Excessive Linearity
• Adversarial Examples rooted in :
• Overfitting
• Excessive Linearity
46 of 84

46. • The Attack Surface
• The Adversarial Capabilities
• The Adversarial Goals
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Threat Model
47 of 84

47. • The Attack Surface
• Evasion Attack :: during the testing phase (* the most common type of attack!)
• Poisoning Attack :: during the training time
• Exploratory Attack :: during the testing phase (Given black box access to the model
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Threat Model
try to gain as much knowledge as possible)
http://fna.ir/a5g
48 of 84

48. • The Adversarial Capabilities
• Training Phase Capabilities
• Data Injection :: does not have any access to the training data as well as to the learning algorithm but has ability to
• Data Modification :: does not have access to the learning algorithm but has full access to the training data
• Logic Corruption :: meddle with the learning algorithm
• Testing Phase Capabilities
• White-Box Attacks :: an adversary has total knowledge about the model (f), algorithm (train), training data distribution (𝜇),
• Black-Box Attacks :: no knowledge about the model and uses information about the settings or past inputs
• Non-Adaptive Black-Box Attack :: only gets access to the target model’s training data distribution (μ)
• Adaptive Black-Box Attack :: doesn’t have any information regarding the training process but can access the target model as an oracle
• Strict Black-Box Attack :: may not contain the data distribution(μ) but has the ability to collect the input-output pairs(x,y) from the
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Threat Model
augment a new data to the training set
target classifier. However, he can not change the inputs to observe the changes in output like an adaptive attack procedure
parameters (𝜃) of the fully trained model architecture
http://fna.ir/a5g
50 of 84

49. • Adversarial Goals:
• Confidence Reduction
• The adversary tries to reduce the confidence of prediction for the target model
• Misclassification
• The adversary tries to alter the output classification of an input example to any class different
from the original class.
• Targeted Misclassification
• The adversary tries to produce inputs that force the output of the classification model to be a
specific target class
• Source/Target Misclassification
• The adversary attempts to force the output of classification for a specific input to be a
particular target class
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Threat Model
http://fna.ir/a5g
51 of 84

50. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Threat Model
http://fna.ir/a5g
52 of 84

51. Adversarial example crafting procedures :
1. Direction sensitivity estimation
• The adversary evaluate the sensitivity of a class change to each input feature
• By identifying the direction in the data manifold around the example X
2. Perturbation selection
• The adversary exploits the knowledge of sensitive information to select a perturbation
• Selecting the perturbation 𝛿𝑋
3. Replace 𝑿 with 𝑿 + 𝜹𝑿
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial Example Generation
Most DNN models make this
formulation non-linear and
non-convex, making it hard
to find a closed-solution in
most of the cases
http://fna.ir/a5g
53 of 84

52. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Crafting adversarial examples:
https://thomas-tanay.github.io/post--L2-regularization/http://fna.ir/a5g
55 of 84

53. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Defenses against Adversarial Examples
Defenses
http://www.rmmagazine.com/2016/06/01/the-3-lines-of-defense-for-good-risk-management/
56 of 84

54. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Review ….
?!
?!
57 of 84

55. ❑Defense is hard!
A theoretical model of the adversarial example crafting process is very difficult to construct.
▪ Non-linearity
▪ non-convex
▪ Complex optimization process
▪ …
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Advances in defense strategies
❑Most of the current defense strategies are
▪not adaptive to all types of adversarial attack
❑Implementation of such defense strategies
▪may incur performance overhead
58 of 84

56. • Removing perturbation with an autoencoder
• Adding noise at test time
• Ensembles
• Confidence-reducing perturbation at test time
• Dropout
• Adding noise at train time
• Various non-linear units
• …
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Failed defenses
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture16.pdf
59 of 84

57. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial ☺ Timeline
60 of 84

58. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Defense
Attack
Concept
61 of 84

59. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
62 of 84

60. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
63 of 84

61. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Adversarial Training
64 of 84

62. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
65 of 84

63. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
7
𝑓 𝑥 + 𝜂 ≠ 𝑓 𝑥 𝑓𝑜𝑟 "𝑚𝑜𝑠𝑡" 𝑥~𝑋
𝜂 𝑝≤ 𝜉
ℙ
𝑥~𝑋
𝑓 𝑥 + 𝜂 ≠ 𝑓 𝑥 ≥ 1 − 𝛿
∆𝜂 ← 𝑎𝑟𝑔 min
𝑟
𝑟 2 𝑠. 𝑡. 𝑓 𝑥𝑖 + 𝜂 + 𝑟 ≠ 𝑓 𝑥𝑖
𝒫 𝑝,ξ(η) = arg min
𝜂′
𝜂 − 𝜂′
2 𝑠. 𝑡. 𝜂′
𝑝 ≤ 𝜉൯𝜂 ← 𝒫 𝑝,ξ(η + ∆𝜂𝑖
66 of 84

64. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
7
𝒫 𝑝,ξ(η) = arg min
𝜂′
𝜂 − 𝜂′
2 𝑠. 𝑡. 𝜂′
𝑝 ≤ 𝜉
67 of 84

65. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
68 of 84

66. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Carlini
69 of 84

67. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Samangouei
70 of 84

68. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Tramèr
71 of 84

69. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Samangouei
72 of 84

70. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Stutz
73 of 84

71. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Trends on Adversarial learning
http://fna.ir/a5d
643
0
783
1
979
8
1294
33
CVPR 2019
74 of 84

72. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Adversarial timeline …
Sabokrou, Khalooei, Adeli
75 of 84

73. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
List of toolboxes
ToolBox Base Lib. Usability Updating
Clever-Hans TensorFlow, Pytorch Semi Well (Oct 2019)
Fool-Box TensorFlow, Keras,
Pytorch, MXnet
Easy Well (Oct 2019)
IBM ART python Semi Well (Oct 2019)
AdverTorch Python Semi Well (Dec 2019)
…
77 of 84

74. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
ToolBoxes :: CleverHans
https://github.com/tensorflow/cleverhans
http://www.cleverhans.io/
78 of 84

75. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
ToolBoxes :: CleverHans
https://cleverhans.readthedocs.io/en/latest/
79 of 84

76. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
ToolBoxes :: FoolBox
80 of 84

77. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
ToolBoxes :: FoolBox
81 of 84

78. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
Tips to stay safe
82 of 84

79. • Decision-boundary crossing
• Vulnerability of deep neural networks in different data
• Different attacks and penetration ways of ML & DL approaches
• Different defense strategies for DNN
• Roadmap of the Adversarial Machine learning
• Attack Toolboxes
Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir
A summary ☺
83 of 84

80. Robustness of Deep Neural Networks Mohammad Khalooei | khalooei@aut.ac.ir 84 of 84