The document discusses machine learning in medical devices. It describes how the FDA regulates machine learning as a medical device and proposes a framework for modifications to AI/ML models. The framework allows manufacturers to make pre-approved changes if they follow a predetermined change control plan or submit changes through an algorithm change protocol, software pre-specifications, and total product lifecycle approach. The document also discusses developing medical devices with machine learning and important safety considerations.

1. Machine Learning
in Medical Devices
June 18, 2020
Milton Yarberry, Director of Medical Programs, ICS
Stephen Olsen, Senior FAE Manager,BlackBerry QNX

2. 2
Agenda
● Machine Learning
○ What is it?
○ Why is this special?
○ The FDA’s treatment
● Use case: Medical devices
○ How to make use of AI or ML in a medical device
● Medical Device Safety Considerations
● Development lifecycle with Machine Learning
● Safeguards for Medical Devices with Machine Learning
Machine Learning in Medical Devices

3. Google’s DeepDream
What does a
network trained to
see dogs see in an
image without
dogs?
Initially it was
invented to help
scientists and
engineers to see
what a deep neural
network is seeing
when it is looking in
a given image.
3

4. Inceptionism
4

5. What is AI/ML?
Without
using explicit
instructions
Relying on
patterns and
inference
instead
AI
FDA defines ML as
“…techniques used to design and train software
algorithms to learn from and act on data…”
“…when intended to treat, diagnose, cure,
mitigate, or prevent disease or other conditions,
are medical devices…” 1
5

6. Why ML is different. What’s hard?
Wikipedia
“without using explicit
instructions, relying on patterns
and inference instead.”
Recognition, categorization but not procedural
Not ...
word processors
email clients
multimedia players
video games
Apocryphal, tanks or clouds?
Verification, local minima
Overtraining (remembering the answers)
Prove “Ground truth” (vs. gold standard)
Different from conventional algorithms
Why ML now?
ML applications
Challenges
Compute power
Since 1984,
Processor power
Up ~1,000,000X
Data storage
Up ~5,000,000X
6

7. ML products with algorithms that continually adapt
based on new data require a new
pre-market submission (510K).
The Problem
7

8. April 2, 2019
FDA proposes regulatory framework for modifications to an AI/ML
https://www.fda.gov/media/122535/download
New from the FDA
8

9. Summary,
FDA allows trusted manufacturers to make
pre-approved changes if they follow a
predetermined change control plan
FDA’s Proposed Framework
9

10. Summary,
FDA allows trusted manufacturers to make
pre-approved changes if they follow a
predetermined change control plan
or,
FDA allows changes to an ML-enabled SaMD with an SPS using ACP in a TPLC.
FDA’s Proposed Framework
10

11. FDA’s Proposed Framework
SPS
ACP
TPLC
11
SaMD Software as a Medical Device (phone, laptop, tablet)
SaMD Pre-Specifications (this is what’s going to change)
Algorithmic Change Protocol (how the change is controlled)
Total Product Life Cycle (an ML lifecycle that includes data)

12. Software as a Medical Device
● Software that runs on a generic computing platform (computer, mobile
phone, tablet computer)
● "Software intended to be used for one or more medical purposes that
perform these purposes without being part of a hardware medical
device.“(IMDRF)
● No specially made hardware, doesn’t control hardware motors, sensors,
switches
SaMD Examples: Viewer for MRI images, ECG app, radiological image analysis
(Computer Aided Detection)
SaMD
12

13. SPS – SaMD Pre-Specification
● Anticipated modifications to performance or inputs or intended use
● A region of potential changes, around specifications and labeling
ACP – Algorithm Change Protocol
● Plan to control risks while making changes
● Step-by-step process
○ Data management
○ Re-training
○ Performance evaluation
○ Update procedures
SPS & ACP
13

14. A regulatory framework
4 principles:
1. Quality systems and good machine learning practice (corporate reputation)
2. Initial premarket assurance of safety and effectiveness (premarket review)
3. Approach for modifications after initial review with an established SPS and ACP (change plan)
4. Transparency and real-time monitoring of AI/ML-based SaMD (PAY ATTENTION)
TPLC – Total Product Life Cycle
14

15. TPLC on an AI workflow
15

16. FDA’s Software Precertification (Pre-Cert) Program
https://www.fda.gov/medical-devices/digital-health/digital-health-software-precertification-pre-cert-program
16

17. Old-way vs. New-way (Pre-Cert)
Software Precertification Program: Working Model – Version 1.0 – January 2019
17

18. 18
Agenda
● Machine Learning
○ What is it?
○ Why is this special?
○ The FDA’s treatment
● Use case: Medical devices
○ How to make use of AI or ML in a medical device
● Medical Device Safety Considerations
● Development lifecycle with Machine Learning
● Safeguards for Medical Devices with Machine Learning
Machine Learning in Medical Devices

19. Software Foundation — Safe
● Safe Real Time OS (QNX)
○ Certificate IEC 62304/61508
○ Safety Manual
○ Fault Tolerant
○ APIs (POSIX)
○ Networking
○ Well Tested/Deployed
19

20. ● Security Features
○ User permissions
○ Application permissions
○ FIPS 140-2
○ OpenSSL
○ Secure File System
○ Lifecycle Management
○ Virus Detection/Eradication
Software Foundation — Security
20

21. Machine Learning in Cloud
● Collect test set data on Edge
● Propagate data securely to cloud (On/Off Premises)
● Train the model
● Distribute model securely to Edge devices
Test
Data
Machine
Learning
21

22. Security: Multilayered with Lifecycle Management
22

23. Research
Security
Lifecycle
Support
Product
Planning
Sourcing Test Production
Contract
Award
Field
Trials Maintenance
Development Integration Refresh
Security
Lifecycle
Support
Field
Updates
Penetration testing
Forensic services
Incident response
Security best practices training
Regulatory compliance support
Security tooling & testing
Secure boot
Access control
Virtualization
Secure reference implementations
Runtime separation & isolation
Secure over-the-air software
Secure manufacturing
Managed PKI
FIPS certified crypto
Complete Security Lifecycle Support
23

24. QNX Runtime Security — Layered and completely integrated
Applications
Temporal & Spatial Control
Runtime Integrity
Application Sandboxing
Authorization
Tamper Resistance
HW Validation
Medical Device Manufacturer and 3rd party apps (embedded &
connected)
Control & Restrict CPU usage, Resource Access, Protect against
defects and rogue execution
Monitor system behavior, Intrusion detection logging & reporting
Whitelisting, Pathspace control, Resource Access, Abilities, Trusted Code
Execution
Mandatory Access Control (MAC)
Secure boot, Signed Execution, Image Verification, Integrity Measurement model
Unique Certificate through Certicom Secure Manufacturing
24

25. SafeGuards
● Training wheels so you cannot fall.
● Validating the output of the model. Ensuring that it does not break
major rules.
25

26. Summary
● Machine Learning
● Use case: Medical devices
○ How to make use of AI or ML in a medical device
● Medical Device Safety Considerations
● Development lifecycle with Machine Learning
● Safeguards for Medical Devices with Machine
Learning
26