The boom of artificial intelligence brought to the market a set of impressive solutions both on hardware and software sides. On the other hand, massive implementation of AI in various areas brings about problems, and security is one of the greatest concerns. The speaker will present results of hands-on vulnerability research of different components of AI infrastructure, including NVIDIA DGX GPU servers, ML frameworks, such as PyTorch, Keras, and TensorFlow, data processing pipelines and specific applications, including medical imaging and face recognition–powered CCTV. Updated Internet Census toolkit based on the Grinder framework will be introduced.
Please note, that this talk is by Sergey and Hacking Odyssey group.
We don't speak for our employers.
All the opinions and information here are of our responsibility. So, mistakes and bad
jokes are all OUR responsibilities.
Hacking Odyssey Group
Hacking Odyssey Projects
SD-WAN New Hop
James Mickens, Harvard University, USENIX Security '18-Q: Why
Do Keynote Speakers Keep Suggesting That Improving Security Is
Gapanovich, Rozenberg, Gordeychik, Signalling cyber security: the need for a mission-centric approach
a process that ensures
control object operation with
no dangerous failures or
damage, but with a set
economic efficiency and
reliability under adversarial
But what about?...
The goals of the project is to provide tools and results of passive and active fingerprinting of
Machine Learning Frameworks and Applications using a common Threat Intelligence
approach and to answer the following questions:
How to detect ML backend systems on the Internet and Enterprise network?
Are ML apps secure at Internet scale?
What is ML apps security level in a general sense at the present time?
How long does it take to patch vulnerabilities, apply security updates to the ML
backend systems deployed on the Internet?
● Sergey Gordeychik
● Anton Nikolaev
● Denis Kolegov
● Maria Nedyak
AIFinger Project Coverage
○ NVIDIA DIGITS
○ Apache mxnet
Databases with ML Content
○ Elasticsearch with ML data
○ MongoDB with ML data
○ Docker API with ML data
○ Kibana (Elasticsearch
○ Redmon (Redis Web UI)
○ Docker API
Job and Message Queues
○ Alibaba Group Holding AI Inference
○ Apache Kafka Consumer Offset Monitor
○ Apache Kafka Manager
○ Apache Kafka Message Broker
○ RabbitMQ Message Broker
○ Celery Distributed Task Queue
○ Gearman Job Queue Monitor
Interactive Voice Response (IVR)
○ Inference Solutions
Measuring Artificial Intelligence and Machine Learning Implementation Security on the Internet
Large scale campaign against Kubernetes and Kuberflow clusters
that abused exposed Kubernetes dashboards for deploying
cryptocurrency miner observed deployment of a suspect image
from a public repository on many different clusters. The image
is ddsfdfsaadfs/dfsdf:99. By inspecting the image’s layers, we can
see that this image runs an XMRIG miner:
Ok, let’s scan!
Nmap scan report for X.X.X.X
Host is up (0.010s latency).
Not shown: 991 closed ports
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 6.0p1 Debian 4 (protocol 2.0)
80/tcp open http lighttpd
427/tcp open svrloc?
443/tcp open ssl/http lighttpd
623/udp open ipmi
554/tcp filtered rtsp
1723/tcp filtered pptp
5120/tcp open barracuda-bbs?
5988/tcp open wbem-http?
5989/tcp open ssl/wbem-https?
I have only one question!
How the complex password will help?!!
Issued by Quanta Computers Inc?
128 bytes (1024) RSA key?..
Issued 17 of April 2017…
Same serial over the Internet!!!
Find and decode firmware
Google for Quanta Computers BMC firmware
Grep the cert and keys
TLS services on BMC uses RSA 1024
with weak cyphers, default Diffie-
The private/public keys are hardcoded
in firmware and are the same for many
Quanta Computers BMC, including
Public and private keys can be found
This allow passively decrypt network
communications without MITM
NetNTLMv2: 28912.2 MH/s
MD5: 450.0 GH/s
SHA-256: 59971.8 MH/s
MS Office 2013: 163.5 kH/s
bcrypt $2*$, Blowfish (Unix): 434.2 kH/s
Can we use DGX to bruteforce DGX password hash?!
BlowFish without IV is used as implemented in libblowfish.so.2.5.0
• Please don’t use one way hashing with salt. Use plaintext or reversible
• Password encryption key should be hardcoded and stored in same folder as a
• It is important to keep it like the product name.
• Store it in several places across the filesystem for resilience.
Hardcoded RC4 Key in JViewer-SOC
• JViewer-SOC (KVM and IPMI applet) use RC4 cipher with a hardcoded key for traffic
• In the JViewer-SOC java applet com.ami.kvm.jviewer.soc.video package contains Decoder
• This class defines DecodeKeys constant which is equal to “fedcba9876543210”.
• Constant is used to initialize RC4 key scheduling (expansion) algorithm.
This allows an attacker to bypass security features, decrypt traffic and extract sensitive
Insecure random number generator in RAKP/AES
• JSOL.jar/com/ami/jsol/common/Util.java defines functions random4ByteArray
• The Random function from java.util.Random class is used.
• These functions are used within RAKP crypto protocol implementation.
• According to the specification of the RAKP it is based on Bellare-Rogaway
• The issue is that the 1 protocols require random numbers in cryptographically
The same function is used to generate IV for AES encryption in the processEncryption function
of IPMISession class.
CSRF is not an issue….
A vulnerability to Cross-Site Request Forgery (CSRF) attack was found in the Nvidia BMC
Web Service. It allows an attacker to force an authenticated user to execute the API
endpoints within the web application.
There is a list of internal queries which require active session authentication and don’t
require CSRF token.
/rpc/ getsessiontoken .asp
Unrestricted SingImage key upload
SingImage upload feature in DGX-1 BMC accept any correct RSA 1024 public key without any verification.
This key is used to verify firmware signature.
SignImage upload routine, implemented in libifc.so.2.42.0 WebValidateSignImageKey function accept any
correct RSA 1024 public key without any verification of authenticity of the key and store it in the
CheckImageSign function implemented in libipmimsghndlr.so use public.pem to verify firmware signature.
Unrestricted File Upload through CSRF
Web-server handler libmodhapi.so defines stripped function at 0x8BE0
address. This function is being called when an authorized user sends POST request to
If a POST request is multipart/form-data this function checks for file argument and if its name
doesn’t end with a ‘/’ symbol¨ looks up for a file path in the hardcoded fille-argument-name-to-
However if the argument name ends with ‘/’¨ file is being saved at the file system defined as file
argument name filename.
Thus it is possible to upload custom files and overwrite existing ones with user-defined
Example attack vector - overwrite ./shadow or ./passwd file in the “/conf/” folder to create/modify
users and/or replace default shell to get remote root access via ssh.
Vulnerability can be exploited via CSRF.
Step 2. Infect it!
Python code to
execute on load
to run on
Python Pickle Injection
Pickle is a python package used to 'serialize' an
object to string format and store them to or
load from a file.
Pickle is a simple stack language, which means
pickle has a variable stack.
• Every time it finished 'deserializing' an object it
stores it on the stack.
• Every time it reaches a '.' while 'deserializing', it
pop a variable from the stack.
Besides, pickle has a temporary memo, like a
'p0', 'p1' means put the top obj on the stack to
memo and refer it as '0' or '1'
'g0', 'g1' act as get obj '0' or '1'
Pickle has two packages: pickle and cPickle,
they have some specific differences like
different methods, but most of the case they
act in the same way.
Step 3. Upload it
Link to our malicious
•Just one command to run from anywhere!
•Protobuf format (.pb)
•~1300 operations (math, conditionals, statistics, etc.)
•Only TWO of them were found dangerous
•WriteFile (any text, any file)
•ReadFile (any file)
Looks like Google
is aware of them
Read the existing graph
and rename the “ending”
Execute func to
determine which route
to take (tensor or
Write it all back
Check if file exists
Append our payload to a
Check if file exists
Append our payload to a
Keras with h5
Weights onlyModel from config
Timeo Danaos et dona ferentes
`torch.load()` uses ``pickle`` module implicitly, which is known to be
insecure. It is possible to construct malicious pickle data which will
execute arbitrary code during unpickling. Never load data that could have
come from an untrusted source, or that could have been tampered with.
**Only load data you trust**.
Hacking Medical Imaging
170 000 cameras across the city
Face recognition system based
on FindFace technology
The current face recognition
system operates on the "black
lists" (criminals, missed people)
The system does not compare
all people caught in the camera
with all residents of Moscow!
Let’s check it out!
• Segmentation dons not works
• Or works, but with poor accuracy
• The presence of a biometric DB
• The relevance of the biometric DB
• Biometric attacks
• Use of masks, etc.
• False positive handling
White List (anyone you can)
• Upload photos via the app
Blacklist (not allowed)
• Register when a COVID is
• Other citizens ???
Where to get?
How to compare with the
Jan Krissler, “Ich sehe, also bin ich ... Du”
What can we do?
AI Cybersecurity is Green Field
From SDN to Model Privacy, from Secure SDL to Adversarial
Don’t trust AI if adversarial “input” is possible
AI IS NOT spherical model traveling in a vacuum!
Centralize data and annotation
Force vendors to follow security best practices from the beginning
Detect and control AI-based abuses