ROMAN PALKIN Backed up with real examples, this talk reviews the capabilities of widely-used frameworks TensorFlow and PyTorch for creating and spreading malicious software as well as implementing covert data communication channels. The purpose of this presentation is to draw attention of the community to the danger posed by careless use of Machine Learning models from unreliable sources.

1. MALIGN MACHINE
LEARNING MODELS
Roman Palkin,
Tomsk State University, AISec

2. github.com/sdnewhop/AISec
2
The project’s goal: Cybersecurity of Machine Learning and Artificial
Intelligence Implementations
Contributors:
● Sergey Gordeychik
● Denis Kolegov
● Antoniy Nikolaev
● Roman Palkin
● Maria Nedyak
AISec Team
That’s me!

3. Problem overview
3
Data science in the 60s Data science in the 80s Data science in the 10s
Data science in 2019

4. Frameworks
4
(Google)
(Facebook)

5. More parameters -> Longer train
5

6. Pre-trained model workflow
6
1. Model interface
(some wrapper, cli,
etc.)
2. Download the
weights in some form
3. Run the model
.py / .sh / etc .pb / .h5 / .pth
.json / .yml /.csv

7. Distribution
7
•~ 2k repos on github
•~ 100 repos on gitlab
•~ 500 models on
https://modelzoo.co/

8. Part I
8

9. Documentation
9
Whole model Weights only
PyTorch model (.pth)

10. Reality
10
Whole model Weights only

11. Step 1. Find an existing model
11

12. Step 2. Infect it!
12
Overwrite the
magic number
`Classic` Pickle payload
Python code to execute
on load
Shell code to
run on load

13. Step 3. Upload it
13
Link to our malicious file

14. •Just one command to run from anywhere!
•torch.hub.load(“ChickenDuo/top”, “model”)
14

15. Part II
15

16. Serialization
16
Saved
Model
Graph
File
(.pb)
Variables
Assets
Constants and static Logic

17. Cross-platform -> Another approach
17

18. Custom serialization
•Protobuf format (.pb)
•~1300 operations (math, conditionals, statistics, etc.)
•Only TWO of them were found dangerous
•WriteFile (any text, any file)
•ReadFile (any file)
18
Looks like Google is
aware of them

19. Graph serialization
19
Result
Tenso
r
Some ops
Payload >
result? Result
Tenso
r
Some ops
Payload ops
True
False

20. Code
20
Read the existing graph and
rename the “ending” tensor
Execute func to determine
which route to take
(tensor or tensor)
Write it all back

21. Wrapper
21
Check if file exists
Append our payload to a file

22. Wrapper
22
Check if file exists
Append our payload to a file

23. Configs
23

24. Part III
24

25. 25
Keras model
Serialization
Saved
Model
Keras with h5
Weights onlyModel from config

26. Serialization with topology
26
- Only Keras layers (Functional model)
- … has a Lambda layer, which serialize custom
python function with marshal
(https://github.com/keras-team/keras/blob/master/k
eras/layers/core.py#L566)
- No warning on launching third-party models!
© keras.io

27. Example
27

28. 28

29. @chicken_2007
THANKS FOR
ATTENTION