IBM Think Summit 2018 Presentation AI at Speed: Scaling GPU processing for Distributed Deep learning and AI - Anand Haridass https://www-07.ibm.com/events/in-en/think-summit/agenda.html

1. Summit
Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation
AI at Speed:
Scaling GPU processing for Distributed Deep learning & AI
—
Anand Haridass
Senior Technical Staff Member,
Cognitive Systems Development,
IBM India

2. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 2
AI at Speed: Scaling GPU processing for Distributed Deep Learning and AI
Speaker: Anand Haridass, STSM, Cognitive Systems, IBM India
Outline:
 AI Workflow & Complexities
 Infrastructure role in AI Workloads
 IBM Accelerated Computing System (AC922)
 Distributed Deep Learning
 PowerAI, Advanced Solutions

3. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 3
From Raw Data to Actionable Insights
AI growth in the last few years due to
 Availability of Data
 Better Algorithms
 Infrastructure – Computational power

4. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 4
Data Source
New Data
Years of
Data
Work Flow & Data Flow Is Complex
Inference
Trained Model
Deploy in
Production using
Trained Model
Seconds
to results
Data Preparation
Data Cleansing &
Pre-Processing
Training
Dataset
Testing
Dataset
Weeks &
months
Heavy IO
Iterate
Build, Train, Optimize Models
AI Deep Learning
Frameworks
(Tensorflow & Caffe)
Monitor
& Advise
Instrumentation
Distributed &
Elastic Deep Learning
Parallel Hyper-Parameter
Search & Optimization
Network
Models
Hyper-
Parameters
Days & weeks
Traditional
Business
IoT &
Sensors
Collaboration
Partners
Mobile Apps &
Social Media
Legacy

5. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 5
ImageNet22K : 7.5M images, 22K classes , Resnet101
Total training time for 100 epoch = 35.2 days
Increasing
Compute
Requirements
Data vs. Accuracy vs. Compute Power
ImageNet1K : 1.2M images, 1K classes, Resnet101
Batch-size = 32 (limited by GPU memory)
Iteration time = 300ms
# iterations per epoch = 38000
Total training time for 100 epoch = 13.2 days

6. 6Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation
What’s Involved In Training – How Infrastructure helps ?
Training  Parameter optimization to make model fit data
Data
Millions of Txt / Images (TB)
Videos (PB)
(Storage / Networking)
Models
Billions of Parameters
(GB)
(Memory / Coherence/ IO
/Networking)
Computation
Heterogenous Compute
Millions of iterations
Mainly matrix operations
(CPU + Accelerator /
Memory/ IO)
Workload Characteristics  Compute & Data Intensive

7. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 7
Faster Training Time with Distributed Deep Learning
Recognition
Recognition
>50x
What will you do?
Iterate more and create more accurate models?
Create more models?
Both?
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
4Hours
9 Days

8. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 8
POWER AC922 System
POWER9 – Leadership I/O
On Chip
Accel
25G
Link
Nvidia
GPUs
I/O
CAPI
NVLink
New CAPI
PCIe
G4
CAPI 2.0
PCIe G4
ASIC /
FPGA
Devices
ASIC /
FPGA
Devices
PCIe
Devices
NVLink 2.0
Open CAPI

9. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 9
Balanced System Design For AI
IBM AC922 – 4 GPU Server

10. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 10
Balanced System Design For AI
IBM AC922 – 4 GPU Server
PCIe G3
16GB/s+
16GB/s
128GB/s
CPU
GPU
Competition (1 socket shown)
PCIe G3
QPI

11. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 11

12. PowerAI: Open-Source
Based Enterprise AI Platform
12Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation
• Software distribution of open-
source AI frameworks
(TensorFlow, PyTorch, Keras ..)
• Open-source enhanced for
Ease of Use & Faster AI
Training Times
• 3-4x Faster Training on
POWER-GPU Servers - HW/SW
co-optimization
Open Source Frameworks:
Supported Distribution
Developer Ease-of-Use Tools
Faster Training Times via
HW & SW Performance Optimizations
GPU-Accelerated
Power Servers
Storage

13. 13
Faster Training Time with Distributed Deep Learning
Communication Library for Distributed Deep
Learning Training
• Enables deep learning software to scale to 100s of
servers with GPUs
• Works across variety of system sizes
• Works with variety of network types, switch topologies
Released Results @256 P100 GPUs
• Better scaling efficiency than Facebook AI Research
95% (IBM) vs < 90% (FB) [ResNet-50, ImageNet-1K ]
• Higher image recognition accuracy than Microsoft
33.8% (IBM) vs. 29.8 (MS) [ImageNet-22K using
ResNet-101
PowerAI DDL: https://arxiv.org/abs/1708.02188
PowerAI DDL training using Caffe
Using S822LC Cluster
Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation

14. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 14
Distributed Deep Learning (DDL)
DDL splits reductions into different dimensions, using different algorithms for each dimension  in the order to cause lowest
communication overhead (It takes into account the fact that not all nodes are connected with the same interface)
DDL performs best compared to other allreduce libraries when used in a cluster with a non-flat topology.

15. Large Model Support (LMS)
1
5
CPUDDR4
GPU
PCIe
Graphics
Memory
System
Bottleneck
Here
POWER
CPU
DDR4
GPU
NVLink
POWER NVLink
Data Pipe
Graphics
Memory
1
5
15Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation
Large Model SupportTraditional Model Support
Limited memory on GPU forces trade-
off in model size / data resolution
Use system memory and GPU to support more
complex and higher resolution data
SysML Conference : https://www.sysml.cc/doc/127.pdf
128GB/s
170GB/s

16. Large Model Support (LMS)
16Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation
Large AI Models Train
~4 Times Faster
3.1 Hours
49 Mins
0
2000
4000
6000
8000
10000
12000
Xeon x86 2640v4 w/
4x V100 GPUs
Power AC922 w/ 4x
V100 GPUs
Time(secs)
Caffe with LMS (Large Model Support)
Runtime of 1000 Iterations
GoogleNet model on Enlarged
ImageNet Dataset (2240x2240)
 TFLMS modifies the TensorFlow graph prior to training to
inject swap nodes that will swap tensors in and out of
GPU memory to system memory.
 Contributed to the community (not accepted yet):
https://github.com/tensorflow/tensorflow/pull/19845

17. 17Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation
libGLM (C++ / CUDA
Optimized Primitive Lib)
Distributed Training
Logistic Regression Linear Regression
Support Vector
Machines (SVM)
Distributed Hyper-
Parameter Optimization
More Coming Soon
APIs for Popular ML
Frameworks
Snap ML
Distributed GPU-Accelerated Machine Learning Library
(coming
soon)
Snap Machine Learning (ML) Library
SnapML : https://arxiv.org/abs/1803.06333

18. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 18
46x faster than previous
record set by Google
Workload: Click-through rate
prediction for advertising
Logistic Regression Classifier in
SnapML using GPUs vs
TensorFlow using CPU-only
Snap ML: Training Time Goes
From An Hour to Minutes
Model: Logistic Regression: TensorFlow vs Snap ML
Test LogLoss: 0.1293 (Google using Tensorflow), 0.1292 (Snap ML)
Platform: 89 CPU-only machines in Google using Tensorflow versus
4 AC922 servers (each 2 Power9 CPUs + 4 V100 GPUs) for Snap ML
Google data from this Google blog
Dataset: Criteo Terabyte Click Logs
(http://labs.criteo.com/2013/12/download-terabyte-click-logs/)
4 billion training examples, 1 million features

19. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 19
Self-defined Training
with visualized
monitoring
Custom Learning for
Image Classification
Inference API
deployment
Image Labeling and
Preprocessing
Computer Vision Services
Video Labeling
Service
Custom Learning for
Object Detection
PowerAI Vision: End to End Computer Vision Platform
Data
Up &
Running
Data Pre-
Processing
Build, Train,
Optimize
Deploy &
Infer
Maintain
Model
Accuracy
 GUI-based tool
 Data Exploration and Labeling
 Semi-automated Labeling for Video streams – label 10-15%* of data and let the pre-packaged models do the remaining labeling
 Pull in custom packages for Data Pre-processing (like OpenCV, etc.)
 Built-in Image Classifier and Object Detection modules
 Self-defined training (with customizable parameters) with visualized monitoring
 Inference model (post-training) deployment for multiple inference engines (CPU, GPU & FPGA)

20. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 20
IBM PowerAI Vision – “Semi-automatic” labelling
Train DL Model
Define Labels
Manually Label Some Images /
Video Frames
Manually Label
Use Trained DL
Model
Run Trained DL Model on
Entire Input Data to Generate
Labels
Correct Labels on
Some Data
Manually Correct Labels on
Some Data
Repeat Till Labels Achieve Desired
Accuracy

21. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 21
PowerAI: Open Source AI Frameworks
H2O ML
Software
GPU-Accelerated
POWER9 Servers
Storage
IBM IVA
Video
Analytics
IBM Data
Science
Experience
Foundational Software
GPU Software
Caffe
Snap ML
ICp: IBM Cloud Private (Kubernetes, Containers)
PowerAI
Vision
PowerAI Enterprise (Spark integration, Cluster Mgmt,
LMS, Auto-Tuning, DDL, Elastic Training)
PowerAI: Open-Source based Enterprise AI Platform

22. Think Summit / Mumbai / Sept 12, 2018 / © 2018 IBM Corporation 22
Thank You
Charts sourced from
Florin Manaila
Sumit Gupta
Subramaniam Meenakshisundaram
Ashwin Srinivas
IBM Research Team