IIBMP2019 講演資料「オープンソースで始める深層学習」

1. Preferred Networks
2. Chainer
3.
4. Chainer Chemistry
5. Optuna

Preferred Networks(PFN)
Automobile
Factory
Robot
Manufacturing
Bio /
Healthcare
Personal
Robot
Creative
2014 3
N 270 2019 7
NTT Fanuc Toyota DYHD
JXTG
•
•

● Mechanical Engineering
● Electrical Engineering
● Navigation
● Manipulation
● Embedded Systems
● SLAM / 3D Reconstruction
● Computer Vision
● Simulation
● Reinforcement Learning
● Data Collection / Annotation
● Speech Recognition
● Operating System
● Natural Language Processing(NLP)
● Human–Computer Interaction(HCI)
(Machine
Learning)
(Deep
Learning)
● User Interface (UI)
● Computer Network
● Motion Planning
● IoT data analysis
● Generative models
● 3D computer
graphics (CG)
● Full-stack dev.
● Bioinformatics
● Biology/Chemistry
● Medical Images
Researcher & Engineer
+ α

● 2017 12
DL
MN-1/1b/2
2560 GPU

2019
●
2018
●
● ODSC East 2018, the Open Source Data Science
Project Award
● CEATEC Award
2017
●
● Japan-U.S. Innovation Awards Emerging Leader
Award
● FT ArcelorMittal Boldness in Business Awards
2016
● JEITA
● Forbes JAPAN’s CEO OF THE YEAR 2016
1

RSNA Pneumonia Detection Challenge
Kaggle X
X

Chainer
Preferred Networks
(oono@preferred.jp)
2019 9 10
8 IIBMP2019

ü NN Python
→ NN
ü Define-by-Run NN
→ Python
→ NN
ü CuPy NumPy-like GPU
→ CPU/GPU agnostic
Speed up research and development of deep learning and its applications.

Awards
ODSC East 2018
Open Source Data Science Project
2018

https://www.youtube.com/watch?v=yNc5N1MOOt4 https://www.youtube.com/watch?v=lGOjchGdVQs

Wikipedia: Electrocardiography
AI
AI

Design networks Train and evaluate
networks
Dataset

Neural Network as a Computational Graph
• NN
• 2 DAG
y = x1 * x2
z = y - x3
x1 mul suby
x3
z
x2

Define-By-Run
1: x = Variable(np.array(3))
3
5*2 -1
- 4
**2
x z
6
9
3
x
y
3 5*2 -1
x
62: y = 2 * x - 1
3: z = x ** 2 - y
y

9
!, "
For # = 1, … , ##)*+,)#-.:
Get a minibatch (01, 21) (1 = 1, … , #4,)5ℎ7#8*)
9 = ∑;<=
#>?@ABCDEF
∇" !(0;, 2;; ")
" ← " – K9
Output " • L
• θ
Stochastic Gradient Descent, SGD

Backpropagation.
10
y = x1 * x2
ga * z * a ga ∇a z
x1 mul suby
x3
z
x2
z = y - x3
gx1 = gy * x2
gx2 = gy * x1
gz = 1
gy = gz
gx3 = -gz
gzgy
gx3
gx1
gx2

Multi Layer Perceptron; MLP
x Linear
W1 b1
h1 ReLU a1
Linear
W2 b2
h2
Soft
max
prob
Cross
Entropy
loss
t
11
1
2

12
class MLP(chainer.Chain):
def __init__(self):
super(MLP, self).__init__()
with self.init_scope():
self.fc1 = L.Linear(None, 100)
self.fc2 = L.Linear(100, 10)
def forward(self, x):
a1 = F.relu(self.fc1(x))
h2 = self.fc2(a1)
return h2
model = MLP()
model = L.Classifier(model)
x Linear
W1 b1
h1 ReLU
a1 Linear
W2 b2
h2

# Setup Trainer
trainer = training.Trainer(
updater, stop_trigger=(100, 'epoch’))
trainer.extend(extensions.ProgressBar())
trainer.run()
# Parameter Updater
opt = optimizers.SGD(lr=0.01)
opt.setup(model)
updater = training.StandardUpdater(it, opt)
13
# Dataset Preparation
dataset = [(x1, t1), (x2, t2), ...]
it = iterators.SerialIterator(dataset)
Example
https://github.com/pfnet/chainer/tree/master/examples
• (__len__ __getitem__
)
•
• Optimizer, Updater
•
• Extension Trainer
• Extension

Caffemodel
Caffe .caffemodel
VGG16, 19, GoogLeNet, ResNet50, 101, 152
model = links.ResNet50Layers(pretrained_model=‘auto')
image = np.random.uniform(0, 1, (224, 224, 3)).astype(np.float32)
image = prepare(image)
with chainer.using_config('train', False):
with chainer.using_config('enable_backprop', False):
feature = model.extract([image], layers=['pool5'])
• NoneImageNet ResNet
•

ChainerCV
• Detection: FasterRCNNVGG16, SDD300, 512, YOLOv2, v3
• Semantic Segmentation: SegNetBasic, PSPNetResNet101
• Instance Segmentation: FCISResNet101
• Classification: SEResNet50, 101, 152, SEResNeXt50, 101

CuPy GPU
ü NumPy API
ü NVIDIA GPU
cuBLAS, cuDNN, cuRAND, cuSPARSE, and NCCL
ü
ElementwiseKernel, ReductionKernel
To GPU
To CPU

ChainerMN
optimizer = chainer.optimizers.MomentumSGD()
optimizer = chainermn.DistributedOptimizer(
chainer.optimizers.MomentumSGD())
ResNet-50 ImageNet
ImageNet 1024GPU, 15
[Akiba et al., 17] [Tokui et al. KDD19]

Chainer Family
Chainer UI Chainer Chemistry

• Chianer define-by-run
•
Caffe, Theano
• NN Chainer
https://chainer.org

Preferred Networks
(suga@preferred.jp)
2019 9 10
8 IIBMP2019

1.
u ~ DL ~
2.
u AI
u Chainer Tutorial

PFN
Humanoid Robot
Consumer Industrial
Cloud
Device
PhotoGame
Text
Speech
Infrastructure
Factory Robot
Automotive
Smart City
Industry4.0
Industrial IoT
Healthcare

DL (1)
DeepVariant ⇒ CNN
(Shanrong Zhao et al., Cloud Computing for Next-Generation Sequencing Data Analysis, 2017 )
(Poplin, Ryan et al., “A universal SNP and small-indel variant caller using deep neural networks.”, 2018 )
•
• DeepVariant (https://github.com/google/deepvariant)
Pileup image

DL (2)
AlphaFold
CASP (Critical Assessment of
techniques for protein Structure
Prediction)
2
(http://predictioncenter.org/casp13/doc/CAS
P13_Abstracts.pdf )

AI
• AI AI
• https://japan-medical-ai.github.io/medical-ai-course-materials/
• GitHub https://github.com/japan-medical-ai/medical-ai-course-materials
• AI AI https://www.japan-medical-ai.org/?page_id=26
•
• Python Google Colaboratory
1.
2.
3.

1)
• Google Colaboratory Web
• Google
• GPU
1. https://colab.research.google.com/
2. Colaboratory ( )
3. GPU
4.
5.
Colab

2)
•
•
•
•
•
•
•
•
•
•
•
• NumPy
•
• Scikit-learn
• Chainer
1
•
•
•
2
• /
• NumPy
• Scikit-learn
3
•
• NumPy
4 Deep Learning
•
• Chainer

3)
(Sequential regulatory activity prediction across chromosomes with convolutional neural networks, D. R. Kelly and et al., 2018)
5 MRI
• MRI
6
•
- “ ”
- ” ”
- ” ”
( )
7 DL
• DNA
8
• /

Chainer Tutorial
•
• Python
1.
2.
3. AI
•
• AI
• Python Pandas Matplotlib

Chainer Tutorial
• 2019 4 10
• (8 )
• 101 views
• 13 users
• AI
( )
• Python
• Cupy
• Pandas
• Matplotlib
•
• …

•
•
• AI
• Chainer Tutorial

Chainer Chemistry
Preferred Networks
2019 9 10
8 (IIBMP2019)

Chainer Chemistry: Chainer
• Deep learning ( Graph Convolution)
–
https://github.com/pfnet-research/chainer-chemistry

Chainer Chemistry: Chainer
• OSS
– https://github.com/pfnet-research/chainer-chemistry

● Data driven
● Deep learning
○ Graph Convolution
● Chainer Chemistry
○
○
●
○
○

→
D 4.4
E 6.3
F 10.5
A 1.8
B 8.8
C 3.2

VS
DFT (Density Functional Theory) MD
→ Pros: ( )
Cons:
→ Pros:
Cons: /
“Neural message passing for quantum chemistry” Justin et al

Deep
learning
Graph Convolution

Graph convolution ~ ~
CNN: Convolutional Neural Network
class label
Chemical
property
Graph convolutional network

Graph convolutional Neural Network
Graph convolution
Graph
Conv
Graph
Conv
Graph
Conv
Graph
Conv
Graph
Readout
Linear Linear
Graph convolutional Neural Network
Graph Graph→

( )
C
N
O
1.0 0.0 0.0 6.0 1.0
atom type
0.0 1.0 0.0 7.0 1.0
0.0 0.0 1.0 8.0 1.0
charge
chirality
Man-made features
Molecular Graph Convolutions: Moving Beyond Fingerprints
Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, Patrick Riley arXiv:1603.00856

Graph Convolution: ( )
Graph Convolution
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, & Vijay Pande (2017). Low Data Drug
Discovery with One-Shot Learning. ACS Cent. Sci., 3 (4)

Graph Readout: ( )
Han Altae-Tran, Bharath Ramsundar, Aneesh S. Pappu, & Vijay Pande (2017). Low Data Drug
Discovery with One-Shot Learning. ACS Cent. Sci., 3 (4)

●
○
●
○ Graph Convolution
●
○ Imbalanced
○

File Parser
(SDF file, CSV file) QM 9, Tox21, molnet dataset
Graph convolution NN
GraphConvolution (Upate)
GraphReadout
Preprocessing
(NFP, GGNN, SchNet)
Example
Train and prediction
with
QM9/tox21/molnet
dataset
Model
Layer/Function
Dataset
Pretrained
Model
(TBD)
Preprocessor (Feature Extractor)

Dataset introduction - tox21
# of Dataset: Train 11757, Validation 295, Test 645
Label - Following 12 types of toxity is included:
'NR-AR', 'NR-AR-LBD', 'NR-AhR', 'NR-Aromatase', 'NR-ER', 'NR-ER-LBD',
'NR-PPAR-gamma', 'SR-ARE', 'SR-ATAD5', 'SR-HSE', 'SR-MMP', 'SR-p53'
Example:
SMILES:
C(=O)C1(O)Cc2c(O)c3c(c(O)c2C(OC2CC
(N)C(O)C(C)O2)C1)C(=O)c1c(O)cccc1C3
=O
LABEL: [ 0 1 -1 1 -1 1 -1 -1 1 -1 1 1]
SMILES:
CCCOc1ccc(C(=O)CCN2CCCCC2)cc1.Cl
LABEL: [ 0 0 0 -1 1 0 0 -1 -1 -1 0 0]
SMILES:
CCOP(=S)(OCC)SC(CCl)N1C(=O)c2cccc
c2C1=O
LABEL: [ 0 0 1 0 1 1 0 1 0 0 -1 -1]
SMILES: O=c1c(O)c(-
c2ccc(O)cc2)oc2cc(O)cc(O)c12
LABEL: [ 0 0 1 -1 1 1 -1 0 0 0 1 0]
2948 3895 6558 7381

Dataset introduction - molnet
http://moleculenet.ai/datasets-1
Chainer Chemistry molecutenet
molnet:
• PubChem PDBbind
•

Dataset introduction - molnet
http://moleculenet.ai/datasets-1

Chainer Chemistry
As of 2019/9/9 from https://github.com/pfnet-research/chainer-chemistry#supported-models
• NFP: Neural Fingerprint [2, 3]
• GGNN: Gated Graph Neural Network [4, 3]
• WeaveNet [5, 3]
• SchNet [6]
• RSGCN: Renormalized Spectral Graph Convolutional Network [10]
• RelGCN: Relational Graph Convolutional Network [14]
• GAT: Graph Attention Networks [15]
• GIN: Graph Isomorphism Networks [17]
• MPNN: Message Passing Neural Networks [3]
• Set2Set [19]
• Graph Warp Module (GWM) [18]-attached models
… →

4 IT (2017 )
• Sirtuin 1
250
•
• 400
200
• 2-stage
– Thermal Shift Assay
– Inhibitory assay → IC50 measurement Sirtuin 1
http://www.ipab.org/eventschedule/contest/contest4

Ours Average
(18 teams in total)
1st screening (TSA) 23 / 200 (11.5%) 69 / 3559 (1.9 %)
2nd screening (IC50) 1 5
We found one hit compound and
won one of Grand prize (IPAB )

Kaggle Molecular property prediction
Kaggle:
CHAMPS
NMR Coupling constant
https://www.kaggle.com/toshik/schnet-starter-kit

BayesGrad
“BayesGrad: Explaining Predictions of Graph Convolutional Networks”
Akita et al., ICONIP 2018
https://arxiv.org/abs/1807.01985
https://github.com/pfnet-research/bayesgrad
Dropout Saliency

GWM
“Graph Warp Module: an Auxiliary Module for Boosting the Power of Graph
Neural Networks in Molecular Graph Analysis” Ishiguro et al.
https://github.com/pfnet-research/chainer-chemistry
Graph Convolution Neural Network

GraphNVP
“GraphNVP: An Invertible Flow Model for Generating Molecular Graphs”
Madhawa et al.
https://github.com/pfnet-research/graph-nvp
Flow

Optuna
Preferred Networks
2019 9 10
8 ( IIBMP2019 )

●
● :
○ Define-by-Run API
○
○
● Chainer
2
https://optuna.org

●
● Optuna
● :
○ Define-by-Run
○
○
●
3

8
● SGD
● Momentum SGD
● Adam
● etc

9
● SGD
● Momentum SGD
○ learning rate (LR) = ?
○ momentum = ?
● Adam
● etc

:
• LR: 0.1
• Dropout: 0.5
… !
Accuracy: 0.6
Trial 1
?
• LR: 0.01
• Dropout: 0.0
… !
Accuracy: 0.5
Trial 2
• LR: 0.05
• Dropout: 0.3
… !
Accuracy: 0.8
Trial 3
10

:
• LR: 0.1
• Dropout: 0.5
… !
Accuracy: 0.6
Trial 1
?
• LR: 0.01
• Dropout: 0.0
… !
Accuracy: 0.5
Trial 2
?
• LR: 0.05
• Dropout: 0.3
… !
Accuracy: 0.8
Trial 3
11

13
●
○ : f(X) → y
■ X:
■ y:
○ : argmin f(X)
■ X f(X)
■ f(X) X
f

: (x - 2)2
21
● :
○ (x): 1.99016
○ : 0.00009

: (x - 2)2
22
● :
○ TPE: Tree-structured Parzen Estimator
○
○

Define-by-Run API
24
● Define-by-Run:
○
○ : Chainer, PyTorch, TensorFlow Eager
●

:
25
0, 5, 3, 8, …
MNIST ( )
From: https://en.wikipedia.org/wiki/MNIST_database

Define-by-Run
28
:
● Python
●

31
• SVHN CNN
•
(Successive Halving)
SVHN: The Street View Housing Numbers Dataset
CNN: Convolutional Neural Network

• study :
○ study_name: study
○ storage: (RDB) URL
⇒ study
35

36
RDB
Storage
(e.g., MySQL)
RDBURLStudy

Kaggle: Open Images Challenge 2018
38
•
• NMW
mAP
○ NMW: Non-Maximum Weighted suppression
○ mAP: mean Average Precision

: HPL
39
• HPL (High Performance Linpack)
○
○ (MN-1b)
• :
○ E.g., LU Broadcast
○
⇒ Optuna

Make your optimization optimized!
42

IIBMP2019 講演資料「オープンソースで始める深層学習」

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IIBMP2019 講演資料「オープンソースで始める深層学習」

Similar to IIBMP2019 講演資料「オープンソースで始める深層学習」 (20)

More from Preferred Networks

More from Preferred Networks (20)

Recently uploaded

Recently uploaded (20)

IIBMP2019 講演資料「オープンソースで始める深層学習」