Deep learning visual recognition technology can be applied to accurately detect cancer patterns on histology tissue slides. This application of deep learning visual recognition technology opens new opportunities in personalized cancer therapy and drug discovery.

1. Visual Recognition of Tissue Patterns on
Pathology Slides
web-pathology.net
FOR HEALTHCARE AND DRUG DISCOVERY
SMART IMAGING TECHNOLOGIES

2. Personalized Cancer Therapy: Knowledge Path
Personalized
Therapy
Medical
History and
Personal
Information
Genetic
Information
Pathology
Information
(tumor
biomarkers)
Personalized cancer therapy is a treatment strategy centered on
the ability to predict which patients are more likely to respond to
specific cancer therapies.
This approach is founded upon the idea that tumor biomarkers
are associated with patient prognosis and tumor response to
therapy.
In addition, patient genetic factors can be associated with drug
metabolism, drug response and drug toxicity.
Personalized tumor molecular profiles, tumor disease site and
other patient characteristics are then potentially used for
determining optimum individualized therapy options.
Source: MD Anderson Cancer Center
Pathology, the “study of disease”,
is an essential component for
analysis of personalized cancer
therapy options
Needs for Healthcare
FOR HEALTHCARE

3. Diagnostic Pattern Library: Applications and Benefits
Classified cancer pattern library is valuable digital asset that can
be licensed to other parties to train visual recognition and image
analysis algorithms.
Visual recognition application can be used to automatically
annotate digital pathology slides and link them with the rest of
institutional cancer knowledge base. This application can be
licensed to third parties to use for the same purposes.
Research and Clinical Applications:
• Computer-assisted cancer diagnosis with pre-screening,
suggestive diagnosis options and contextual links to cancer
knowledge libraries (similar cases, experts, research, additional
tests etc.)
• Data mining and of advanced analytics of historic tissue
samples for cancer patients with known outcomes with
purpose of building predictive knowledge bases for cancer
care and drug discovery
FOR HEALTHCARE

4. Researchers need to measure drug effects on tissue.
They also need to analyze relationships between effect on tissue and
all other relevant data in the drug research project.
Drug effects on tissue are expressed via biomarker patterns on tissue
slides. Researchers need to turn visual pattern expression into usable
data.
They need technology to:
• Objectify pattern definitions to deal with subjective biases of human
observers
• Search for similar biomarker expression patterns across archives
• Aggregate, analyze and organize pattern expression information
• Link pattern information to other “omnics” and project research data
Needs for Drug Discovery
FOR DRUG DISCOVERY

5. Utilizing Pathology Knowledge: The Challenge
Traditionally, pathology diagnosis is presented in descriptive
natural language statements. Often It is verbose professional
opinion of human expert with little quantitative information
Statistical agreement between human experts is 75%-85%.
Pathology diagnosis is rendered by pathologist observing
patterns of cells on tissue slide under the microscope. In order
to be useful for comparison and analysis these observations
must be:
• Quantified
• Objectified
This can be achieved (in theory) by digitizing pathology slide
and applying image analysis algorithms to quantify cell pattern
expressions.

6. Analyzing Pathology Slides: The Real Life Issues
In practice traditional image analysis approach often fall
short of expectations for number of reasons:
• Target image features are complex and difficult for
engineers to formalize and code.
• Algorithm development process is inherently
complicated: only pathologists know that patterns
they need but only image analysis engineers may
know how to extract them
• Tissues have variability and algorithms developed on
one sample may not work on others. Algorithms have
to be redesigned by engineers every time new “out of
range” samples are encountered
• Digital tissue slides are very large and processing every
pixel is extremely computationally expensive, making
“online” analysis impossible or impractical

7. Analyzing Pathology Slides: Machine Learning
Machine Learning Neural Networks learn to recognize images in the
same way humans do – by example, rather than by formalized
“handcrafted features”.
Since 2012 major improvement in visual recognition was achieved
with so called deep learning neural networks. Latest generation of
Visual Recognition Neural Networks achieve accuracy of recognition of
natural objects similar to human observers. This area of technology is
experiencing explosive growth.
Using Machine Learning brings number of advantages to visual
recognition applications:
• No need to formalize complex “handcrafted features”, pathologist
can just point to patterns they need to recognize
• No dependency on image analysis engineers (almost)
• System can be trained on very large number of samples to achieve
robust recognition
• New data samples can be added to model easily to increase
accuracy

8. Machine Learning: Requirements
Machine Learning approach to pattern recognition creates
new functional requirements for digital pathology software
• Robust visual recognition models need large number of training images which requires more
time for annotating than single pathologist can provide. This problem can be solved by
utilizing number of pathologists creating annotations for training visual recognition models
Collaborative Model Training (Crowdsourcing)
• Digital Pathology system should have capability for extracting specially formatted image
data sets on demand for training neural networks
Training Data Extraction
• Training of Neural Network requires massive parallel GPU computing power for a short time.
This scalable computing power can be economically delivered by scalable cloud
infrastructures such as IBM, Amazon or Microsoft.
Cloud Deployment
• Digital pathology software should be able to send imaging data to neural network
application for recognition and visualize responses for user.
API Integration and Visualization Interface

9. The Solution: Pattern Recognition with Machine Learning
• Last generation deep learning convolution networks can identify target tissue patters with 95%
accuracy
Deep Learning
• Pathologists can train recognition solution by simply annotating target tissue patters on slides
in their workspace
• They can easily set up classes of patterns for identification
Easy Training
• Robust solutions can be trained from multiple slides to identify target tissue patterns reliably
across large variety of samples
• Recognition models can be retrained easily if new patterns or different samples should be
added
Robust Recognition Models
• Slides in digital archives can be processed automatically for pattern detection and labeled
based on findings
• New slides can be analyzed and classified on upload with suggestive classification available
when human expert opens the slide
Automatic Processing
• Visualization overlays help quickly locate and review target patterns
• Visualization layer provides quantitative information about patterns
Advanced Visualization
• All data is stored in the database and available for search, data mining and analytics
Powerful Analytics
Our software can train neural networks and utilize latest deep learning visual recognition solutions
from best in class solution providers

10. The Solution: How it works
• Human experts mark areas with target patterns on digital slides in the web interface
• Multiple people across geographic locations can work on the same slide library at the
same time
• Software automatically prepares and extracts imaging data for training visual recognition
model in appropriate format
• Massive amount of training data can be produced quickly and efficiently with
crowdsourcing approach
1. Preparation of Training Data
• We train visual recognition models using proven templates and best-in-class machine
learning service providers
2. Training Visual Recognition Model
• Tissues on new slides coming to server are automatically recognized against known
taxonomy and labeled according to application logic.
3. Automatic Recognition Processing
• Users can mark the area and search for similar patterns on the same slide or across entire
slide archive
4. Search by Example
• Software shows identified patterns on slides using interactive “smart” overlays and
computes statistics
5. Viewing Results
• Users can correct visual recognition results by marking area with appropriate label. New
data will be added to the training set to improve recognition
6. Continuous Learning

11. Integration: Data Mining and Discovery
• Non SQL flexible indexed database architecture allows integrated
storage of different data items across multiple locations
Distributed Database
• Flexible structure allows storing and integrating various data in the
single information store
• New data can be added to database structure at any time
Comprehensive Data
• Selection and navigation is possible for any data item in the database
• Global search on any data is instant even for millions of items
Instant Search and Navigation
• Data items can be linked with external data sources and knowledge
bases such as diagnostic codes, SNOMED classifications or
proprietary knowledge bases
Data Linking
We provide instant search, navigation and data mining ability across millions of slides