3. OCT 26, 2017 - 3Copyright 2017 ITRI 工業技術研究院
自主智慧車深度學習測試驗證平台
4. OCT 26, 2017 - 4Copyright 2017 ITRI 工業技術研究院
ITRI Autonomous Vehicle Platforms
• AI Development
• Data mining
• Data augmentation
• CNN/DNN model training
• NVidia Platform Development
• PilotNet
• DetectNet
• OpenroadNet (SpaceNet)
NVidia PX2 IPC
• Perception Development
• Lane detection
• Road mark detection
• Object detection
• Localization
• Decision Development
• Lane keeping
• Lane following
• Stop and Go
• Auto parking
• Call car
5. OCT 26, 2017 - 5Copyright 2017 ITRI 工業技術研究院
Deep Learning for Autonomous Vehicles
Automated Vehicle
Real Driving Scene
Up to 50 kph
Up to 10 scenes
DL Driving Controller
On-Board Driving Control Systems
Visual Sensor
DL Perception Processor
720PX480P/ 30fps X3
Traffic Scenario Database
Deep Learning (DL) Cloud System
Sensed Data
Driving Teaching
DL Modeling & Computing Center
Traffic Scene Video
DL model
Up to 30
scenes,
include
10 urban
scenes
750Mb/s
X 10 vehicles
750Mb/s
Peak computing performance
> 3 TFLOPS
decision @ 0.1 sec
6. OCT 26, 2017 - 6Copyright 2017 ITRI 工業技術研究院
With NVIDIA PX2 for Autonomous Vehicles
Data server
Label Tool: Vatics
Labeled data: >100k images
Object type : Pedestrian, Bike,
Motorcycle, Vehicle, Bus
Data Mining
Training Server-NVIDIA
DIGITS DevBox *2+
DGX-1
CNN model:
Object Detection:
Faster R-CNN, YOLO, SSD
Datasets:
Pascal + MSCOCO +
KITTI+ Self-collected
Data Training
Ground Truth
Testing
DrivePX2
Detect NET
Products
Space NETPilot NET
7. OCT 26, 2017 - 7Copyright 2017 ITRI 工業技術研究院
NVIDIA DetectNet for Autonomous Vehicles
• Object detection
• Object Classes:
Vehicle, Bus, Motorcycle, Bike, Pedestrian.
• 20 fps @ DrivePX2
Model
Camera Input
8. OCT 26, 2017 - 8Copyright 2017 ITRI 工業技術研究院
NVIDIA SpaceNet for Autonomous Vehicles
• Distinguish the three characteristic areas: free-space, car, and person
9. OCT 26, 2017 - 9Copyright 2017 ITRI 工業技術研究院
NVIDIA PilotNet for Autonomous Vehicles
• Using pixel changes to control steering wheel
Steering
Wheel Angle θ
PilotNet demonstration on driving simulator
Input Sequences
from TriCam
10. OCT 26, 2017 - 10Copyright 2017 ITRI 工業技術研究院
NVIDIA PilotNet for Autonomous Vehicles
Camera View Steering Wheel View Vehicle View
End2End:
Input
Output
12. OCT 26, 2017 - 12Copyright 2017 ITRI 工業技術研究院
Testing on PilotNet, DetecNet, SpaceNet
• Test PilotNet
for lane
following and
right turn
• Test PilotNet
under rainy
conditions
• Test DetecNet
and SpaceNet
13. OCT 26, 2017 - 13Copyright 2017 ITRI 工業技術研究院
Testing on PilotNet, DetecNet, SpaceNet
16. OCT 26, 2017 - 16Copyright 2017 ITRI 工業技術研究院
車輛的出現到行駛策略
17. OCT 26, 2017 - 17Copyright 2017 ITRI 工業技術研究院
From Horse Carriage to First Automobile
• Britain’s 1865 Locomotive Act:
• Drivers were limited to 4mph and
• Driver were required to follow a man on foot
carrying a red warning flag.
• Fortunately it was amended in 1896
to ditch that nonsense.Source: https://www.youtube.com/watch?v=WBzDrpTOXi8
Source: https://www.crazyengineers.com/threads/a-brief-history-on-automobile-in-the-world.70397
Source: http://www.carmagazine.co.uk/features/top-10s/the-car-top-10-reasons-why-youve-never-had-it-so-easy/
18. OCT 26, 2017 - 18Copyright 2017 ITRI 工業技術研究院
SAE International J3016 Standard
Source: http://www.sae.org/misc/pdfs/automated_drivingpdf
Source: http://www.birmingham.ac.uk/news/thebirminghambrief/items/2016/11/driving-the-revolution.aspx
Source: http://www.businessinsider.com/what-are-the-different-levels-of-driverless-cars-2016-10/#-6
19. OCT 26, 2017 - 19Copyright 2017 ITRI 工業技術研究院
Vehicle Sensor Fusion and Control
Source: http://www.businessinsider.com/what-are-the-different-levels-of-driverless-cars-2016-10/#-6
Source: http://www.nybooks.com/articles/2016/11/24/driverless-intelligent-cars-road-ahead/
20. OCT 26, 2017 - 20Copyright 2017 ITRI 工業技術研究院
Sensing & Perception for Automated Driving
Where is the car? What is around the car? What will happen next?
What should the car DO?Source: GTC Taipei 2016
21. OCT 26, 2017 - 21Copyright 2017 ITRI 工業技術研究院
工研院的全方位測試驗證研發平台
22. OCT 26, 2017 - 22Copyright 2017 ITRI 工業技術研究院
R&D Process for Autonomous Vehicles
Computing System
(Real-Time Driving Decision)
Sensing System
(Master Surrounding Condition)
Driving Control
(Safe and Smooth Maneuver)
System
Requirement
Core
Modules
Camera Lidar Chip Computer Steering Braking Panel
(經濟部技術處商用電動車平台)
• Our research focus is
to develop novel visual/range sensing technology and perception algorithm
as well as related integrating sensing/computing/control.
23. OCT 26, 2017 - 23Copyright 2017 ITRI 工業技術研究院
R&D Process for Autonomous Vehicles
• We have a strong and solid V-Shape R&D process for typical vehicle development,
from the virtual simulation of prototype design to the verification of real vehicles
Virtual Simulation Real Verification
Validation
Sensors in the Loop
Controller in the Loop
Sensor Modeling
Controller Modeling
• Model-based (PreScan)
Dynamics (CarSIM)
PreScan
Matlab/Simulink
Driving Simulator
Real World
Calibration
dSPACE
MicroAutoBox
Human Driving Experience Learning
Passenger Response Investigation
Target Field &
Scenario
Rapid
Prototyping
Vehicle/Driver
in the Loop
Hardware
in the Loop
CPEV
Virtual Car
Automated
Control System
Field Test
24. OCT 26, 2017 - 24Copyright 2017 ITRI 工業技術研究院
Real Vehicle & Virtual Simulator
Real Vehicle
Test Filed (Bldg. 58) Scenario Model
Vision set Lidar/Radar
Real Vehicle Prototyped Controller Virtual Vehicle Control Block Diagram
Multi-Camera TIS Model
Comparison between PreScan simulation and vehicle testing
Virtual Vehicle
25. OCT 26, 2017 - 25Copyright 2017 ITRI 工業技術研究院
Versatile Research Development Stages
Virtual
Simulation
Model
Virtual
Simulation
Model
Virtual
Simulation
Model
Virtual
Simulation
Model
Model
in the Loop
Hardware
in the Loop
Driver
in the Loop
Vehicle
in the Loop
Stage 1 Stage 2 Stage 3
Controller
Vehicle
Scenario
System
Sensor
Stage 0
26. OCT 26, 2017 - 26Copyright 2017 ITRI 工業技術研究院
Data Fusion and Control on Vehicle Platform
Ibeo Laser
Vision set
DGPS
30 Hz
25 Hz
10 Hz
障礙物資訊
(dx, dy, dθ)
位置經緯度(EL/NL)
車輛方位角(θheading)
720×480(Pixel)
Stop line (dx,dy)
Sensing (Step 1)
Fusion results
Ackermann
steering geometry
Perception (Step 2)
𝑅"#
Bird view of ROI
Plan driving zone
Clearanc
e limit
𝑥, 𝑦 = ((𝑑𝑥+, 𝑑𝑦+)
Vehicle
heading
angle
Target
heading
angle
Weighting
procedure
Steering
angle limit
Decision (Step 3)
Navigation
Decide path
𝑙𝑖𝑚𝑖𝑡 𝜑3, 𝜑4
= (40, −40)
δ
δ_target
𝑃9 = (𝐸𝐿, 𝑁𝐿, 𝜃>?@A+BC)
y: 路徑偏移量
ω: 轉角變化率
υ: 車速
φ: 前輪轉向角Control (Step 4)
Steering
Control
command
Vehicle
dynamics
𝛼 = 𝑓(𝑦, 𝜔, 𝑣, 𝜑)
𝛿A = 𝛼𝛿 δd: 車輛目標方向角
δ: 車輛方向角
Final result
𝑋′
𝑌′
= 𝑓
𝑋
𝑌
Perspective
projection
27. OCT 26, 2017 - 27Copyright 2017 ITRI 工業技術研究院
Versatile Testing Scenarios – Simulator
• Virtual ADAS integration on sensor & controller testing in simulated scenarios
• Self-driving controller development
Vehicle Model
Sensor Model
Brake Pedal
Steer Angle
Acc. Pedal
Speed
R_Line
L_Line
Stop Line
GPS
Motion Control Model
LKA: Lane keeping Assistance
GPS-based Steering Control
Stop Line Recognition & Braking
Deep Learning Model
Scenario Model
Shadow
Rain
Light
Darkness
