SlideShare a Scribd company logo

Sensor fusion of LiDAR and Camera for real time object detection - talk version

•
•
NECST Lab @ Politecnico di Milano
NECST Lab @ Politecnico di Milano

Autonomous vehicles primary need is to have a robust and precise perception of the surrounding environment. To achieve accurate results in autonomous driving a combination of several different sensors is used. This technique is commonly known as sensor fusion. However, a critical aspect of autonomous driving perception is also the reactivity: we do not only need to perform sensor fusion properly, but we also have to respect real time requirements. The latter are fundamental to guarantee predictability of the system, avoid anomalous situations and prevent hazards. Performing fusion online is not trivial because sensors returns a lot of data and process them may be time consuming. The solution we present is an alignment of the sensors, which combines information from LiDAR and cameras, that is performed once in preprocessing and allow us to exploit the precomputed matching in real time.

Automotive
1 of 22
Download to read offline
Sensor fusion for autonomous driving perception Bagni Fabio 228594@studenti.unimore .it San Francisco - 15 May 2019 1
Fabio Bagni San Francisco - 15 May 2019 Autonomous driving perception Precision and reactivity Risks of accidents, traffic laws respect. 2
Fabio Bagni San Francisco - 15 May 2019 Embedded platforms 3
Fabio Bagni San Francisco - 15 May 2019 Goals Real time 3D detection ● High performances ○ required 10 Hz frequency ● High precision ● Real time on embedded platforms 4
Fabio Bagni San Francisco - 15 May 2019 UNIMORE HiPeRT prototype 5
Fabio Bagni San Francisco - 15 May 2019 Sensors ● LiDAR ○ 3D point cloud ● Cameras ○ images Heterogeneous outputs 6
Fabio Bagni San Francisco - 15 May 2019 Sensor fusion What is sensor fusion: ● Matching of different sensors views Why sensor fusion: ● Heterogeneous information of environment 7
Fabio Bagni San Francisco - 15 May 2019 Fields of view LiDAR: 360 degrees around the vehicle Camera: wide angle, disposed do cover 360 degrees. 8 Cameras LiDAR
Fabio Bagni San Francisco - 15 May 2019 Deep learning approach Camera: ● Bird’s Eye View estimation ● Object detection LiDAR ● Bird’s Eye View calculation ● Object detection 9 Image Point cloud 3D detection 3D detection Fusion
Fabio Bagni San Francisco - 15 May 2019 Deep learning approach time Image stitching for multiple camera fusion 10 ➔ Under10 Hz Notenoughresponsive 1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).
Fabio Bagni San Francisco - 15 May 2019 Static alignment sensor fusion Alignment of LiDAR with each camera individually. 11 Calibration Preprocessing
Fabio Bagni San Francisco - 15 May 2019 Features extraction Matching is made by a calibration with a perforated panel. Holes are detected by all sensors. Holes’ centers are used as features to be matched. 12
Fabio Bagni San Francisco - 15 May 2019 Cylindrical projection Fields of view matching is made projecting sensors’ outputs on the surface of a cylinder. 13
Fabio Bagni San Francisco - 15 May 2019 Projections complexity LiDAR cylindrical projection: Depthmap O(n) : one operation per point 14 Camera cylindrical projection O(n) : one operation per pixel PARALLELIZABLE
Fabio Bagni San Francisco - 15 May 2019 Calibration results 15
Fabio Bagni San Francisco - 15 May 2019 Colored points Alignment of each point with the corresponding pixel on the image. 16
Fabio Bagni San Francisco - 15 May 2019 Object detection Neural network for object detection on camera frame Clustering of LiDAR points Bounding boxes matching 17
Fabio Bagni San Francisco - 15 May 2019 Object detection results 18
Fabio Bagni San Francisco - 15 May 2019 Object detection average time 19 LiDAR output : 10 Hz Time limit : 100 ms
Thanks for your attention Fabio Bagni - 228594@studenti.unimore.it 20 Fabio Bagni San Francisco - 15 May 2019
21
Fabio Bagni San Francisco - 15 May 2019 Homography Homography allow to translate camera projection plane on the LiDAR projection plane. This method make cylindrical surfaces match. 22

Recommended

fusion of Camera and lidar for autonomous driving II
fusion of Camera and lidar for autonomous driving IIfusion of Camera and lidar for autonomous driving II
fusion of Camera and lidar for autonomous driving IIYu Huang
 
fusion of Camera and lidar for autonomous driving I
fusion of Camera and lidar for autonomous driving Ifusion of Camera and lidar for autonomous driving I
fusion of Camera and lidar for autonomous driving IYu Huang
 
Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Yu Huang
 
Introduction to object detection
Introduction to object detectionIntroduction to object detection
Introduction to object detectionBrodmann17
 
Object Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksObject Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksUsman Qayyum
 
Driver drowsiness detection
Driver drowsiness detectionDriver drowsiness detection
Driver drowsiness detectionConnecting Point
 
Computer Vision
Computer VisionComputer Vision
Computer VisionAmeer Mohamed Rajah
 
Object tracking presentation
Object tracking presentationObject tracking presentation
Object tracking presentationMrsShwetaBanait1
 

Recommended

fusion of Camera and lidar for autonomous driving II
fusion of Camera and lidar for autonomous driving IIfusion of Camera and lidar for autonomous driving II
fusion of Camera and lidar for autonomous driving IIYu Huang
 
fusion of Camera and lidar for autonomous driving I
fusion of Camera and lidar for autonomous driving Ifusion of Camera and lidar for autonomous driving I
fusion of Camera and lidar for autonomous driving IYu Huang
 
Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Lidar for Autonomous Driving II (via Deep Learning)
Lidar for Autonomous Driving II (via Deep Learning)Yu Huang
 
Introduction to object detection
Introduction to object detectionIntroduction to object detection
Introduction to object detectionBrodmann17
 
Object Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksObject Detection using Deep Neural Networks
Object Detection using Deep Neural NetworksUsman Qayyum
 
Driver drowsiness detection
Driver drowsiness detectionDriver drowsiness detection
Driver drowsiness detectionConnecting Point
 
Computer Vision
Computer VisionComputer Vision
Computer VisionAmeer Mohamed Rajah
 
Object tracking presentation
Object tracking presentationObject tracking presentation
Object tracking presentationMrsShwetaBanait1
 
Traffic sign detection
Traffic sign detectionTraffic sign detection
Traffic sign detectionAvijit Rai
 
LiDAR-based Autonomous Driving III (by Deep Learning)
LiDAR-based Autonomous Driving III (by Deep Learning)LiDAR-based Autonomous Driving III (by Deep Learning)
LiDAR-based Autonomous Driving III (by Deep Learning)Yu Huang
 
3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -Kazuyuki Miyazawa
 
Deep learning for object detection
Deep learning for object detectionDeep learning for object detection
Deep learning for object detectionWenjing Chen
 
=SLAM ppt.pdf
=SLAM ppt.pdf=SLAM ppt.pdf
=SLAM ppt.pdfusmanarif88
 
3-d interpretation from single 2-d image for autonomous driving II
3-d interpretation from single 2-d image for autonomous driving II3-d interpretation from single 2-d image for autonomous driving II
3-d interpretation from single 2-d image for autonomous driving IIYu Huang
 
Introduction of slam
Introduction of slamIntroduction of slam
Introduction of slamHung-Chih Chang
 
Depth Fusion from RGB and Depth Sensors by Deep Learning
Depth Fusion from RGB and Depth Sensors by Deep LearningDepth Fusion from RGB and Depth Sensors by Deep Learning
Depth Fusion from RGB and Depth Sensors by Deep LearningYu Huang
 
Human Action Recognition
Human Action RecognitionHuman Action Recognition
Human Action RecognitionNAVER Engineering
 
Object detection
Object detectionObject detection
Object detectionROUSHAN RAJ KUMAR
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingPreferred Networks
 
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), FooladSuperpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), FooladShima Foolad
 
Image segmentation
Image segmentationImage segmentation
Image segmentationKuppusamy P
 
Object detection presentation
Object detection presentationObject detection presentation
Object detection presentationAshwinBicholiya
 
"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from CadenceEdge AI and Vision Alliance
 
Image captioning
Image captioningImage captioning
Image captioningMuhammad Zbeedat
 
Computer Vision for Beginners
Computer Vision for BeginnersComputer Vision for Beginners
Computer Vision for BeginnersSanghamitra Deb
 
Vehicle detection
Vehicle detectionVehicle detection
Vehicle detectionEvangelos Skodras
 
Object Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning FrameworkObject Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning FrameworkNader Karimi
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Simplilearn
 
Mesticheria Team - WiiReflex
Mesticheria Team - WiiReflexMesticheria Team - WiiReflex
Mesticheria Team - WiiReflexNECST Lab @ Politecnico di Milano
 
Punto e virgola Team - Stressometro
Punto e virgola Team - StressometroPunto e virgola Team - Stressometro
Punto e virgola Team - StressometroNECST Lab @ Politecnico di Milano
 

More Related Content

What's hot

Traffic sign detection
Traffic sign detectionTraffic sign detection
Traffic sign detectionAvijit Rai
 
LiDAR-based Autonomous Driving III (by Deep Learning)
LiDAR-based Autonomous Driving III (by Deep Learning)LiDAR-based Autonomous Driving III (by Deep Learning)
LiDAR-based Autonomous Driving III (by Deep Learning)Yu Huang
 
3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -Kazuyuki Miyazawa
 
Deep learning for object detection
Deep learning for object detectionDeep learning for object detection
Deep learning for object detectionWenjing Chen
 
3-d interpretation from single 2-d image for autonomous driving II
3-d interpretation from single 2-d image for autonomous driving II3-d interpretation from single 2-d image for autonomous driving II
3-d interpretation from single 2-d image for autonomous driving IIYu Huang
 
Depth Fusion from RGB and Depth Sensors by Deep Learning
Depth Fusion from RGB and Depth Sensors by Deep LearningDepth Fusion from RGB and Depth Sensors by Deep Learning
Depth Fusion from RGB and Depth Sensors by Deep LearningYu Huang
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingPreferred Networks
 
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), FooladSuperpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), FooladShima Foolad
 
Image segmentation
Image segmentationImage segmentation
Image segmentationKuppusamy P
 
Object detection presentation
Object detection presentationObject detection presentation
Object detection presentationAshwinBicholiya
 
"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from CadenceEdge AI and Vision Alliance
 
Computer Vision for Beginners
Computer Vision for BeginnersComputer Vision for Beginners
Computer Vision for BeginnersSanghamitra Deb
 
Object Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning FrameworkObject Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning FrameworkNader Karimi
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Simplilearn
 

What's hot (20)

Traffic sign detection
Traffic sign detectionTraffic sign detection
Traffic sign detection
 
LiDAR-based Autonomous Driving III (by Deep Learning)
LiDAR-based Autonomous Driving III (by Deep Learning)LiDAR-based Autonomous Driving III (by Deep Learning)
LiDAR-based Autonomous Driving III (by Deep Learning)
 
3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -3D Perception for Autonomous Driving - Datasets and Algorithms -
3D Perception for Autonomous Driving - Datasets and Algorithms -
 
Deep learning for object detection
Deep learning for object detectionDeep learning for object detection
Deep learning for object detection
 
=SLAM ppt.pdf
=SLAM ppt.pdf=SLAM ppt.pdf
=SLAM ppt.pdf
 
3-d interpretation from single 2-d image for autonomous driving II
3-d interpretation from single 2-d image for autonomous driving II3-d interpretation from single 2-d image for autonomous driving II
3-d interpretation from single 2-d image for autonomous driving II
 
Introduction of slam
Introduction of slamIntroduction of slam
Introduction of slam
 
Depth Fusion from RGB and Depth Sensors by Deep Learning
Depth Fusion from RGB and Depth Sensors by Deep LearningDepth Fusion from RGB and Depth Sensors by Deep Learning
Depth Fusion from RGB and Depth Sensors by Deep Learning
 
Human Action Recognition
Human Action RecognitionHuman Action Recognition
Human Action Recognition
 
Object detection
Object detectionObject detection
Object detection
 
Introduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable RenderingIntroduction to 3D Computer Vision and Differentiable Rendering
Introduction to 3D Computer Vision and Differentiable Rendering
 
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), FooladSuperpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
Superpixel algorithms (whatershed, mean-shift, SLIC, BSLIC), Foolad
 
Image segmentation
Image segmentationImage segmentation
Image segmentation
 
Object detection presentation
Object detection presentationObject detection presentation
Object detection presentation
 
"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence"Fundamentals of Monocular SLAM," a Presentation from Cadence
"Fundamentals of Monocular SLAM," a Presentation from Cadence
 
Image captioning
Image captioningImage captioning
Image captioning
 
Computer Vision for Beginners
Computer Vision for BeginnersComputer Vision for Beginners
Computer Vision for Beginners
 
Vehicle detection
Vehicle detectionVehicle detection
Vehicle detection
 
Object Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning FrameworkObject Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning Framework
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
 

More from NECST Lab @ Politecnico di Milano

Embedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposingEmbedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposingNECST Lab @ Politecnico di Milano
 
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...NECST Lab @ Politecnico di Milano
 
EMPhASIS - An EMbedded Public Attention Stress Identification System
EMPhASIS - An EMbedded Public Attention Stress Identification System EMPhASIS - An EMbedded Public Attention Stress Identification System
EMPhASIS - An EMbedded Public Attention Stress Identification SystemNECST Lab @ Politecnico di Milano
 
Maeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matchingMaeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matchingNECST Lab @ Politecnico di Milano
 

More from NECST Lab @ Politecnico di Milano (20)

Mesticheria Team - WiiReflex
Mesticheria Team - WiiReflexMesticheria Team - WiiReflex
Mesticheria Team - WiiReflex
 
Punto e virgola Team - Stressometro
Punto e virgola Team - StressometroPunto e virgola Team - Stressometro
Punto e virgola Team - Stressometro
 
BitIt Team - Stay.straight
BitIt Team - Stay.straight BitIt Team - Stay.straight
BitIt Team - Stay.straight
 
BabYodini Team - Talking Gloves
BabYodini Team - Talking GlovesBabYodini Team - Talking Gloves
BabYodini Team - Talking Gloves
 
printf("Nome Squadra"); Team - NeoTon
printf("Nome Squadra"); Team - NeoTonprintf("Nome Squadra"); Team - NeoTon
printf("Nome Squadra"); Team - NeoTon
 
BlackBoard Team - Motion Tracking Platform
BlackBoard Team - Motion Tracking PlatformBlackBoard Team - Motion Tracking Platform
BlackBoard Team - Motion Tracking Platform
 
#include<brain.h> Team - HomeBeatHome
#include<brain.h> Team - HomeBeatHome#include<brain.h> Team - HomeBeatHome
#include<brain.h> Team - HomeBeatHome
 
Flipflops Team - Wave U
Flipflops Team - Wave UFlipflops Team - Wave U
Flipflops Team - Wave U
 
Bug(atta) Team - Little Brother
Bug(atta) Team - Little BrotherBug(atta) Team - Little Brother
Bug(atta) Team - Little Brother
 
#NECSTCamp: come partecipare
#NECSTCamp: come partecipare#NECSTCamp: come partecipare
#NECSTCamp: come partecipare
 
NECSTCamp101@2020.10.1
NECSTCamp101@2020.10.1NECSTCamp101@2020.10.1
NECSTCamp101@2020.10.1
 
NECSTLab101 2020.2021
NECSTLab101 2020.2021NECSTLab101 2020.2021
NECSTLab101 2020.2021
 
TreeHouse, nourish your community
TreeHouse, nourish your communityTreeHouse, nourish your community
TreeHouse, nourish your community
 
TiReX: Tiled Regular eXpressionsmatching architecture
TiReX: Tiled Regular eXpressionsmatching architectureTiReX: Tiled Regular eXpressionsmatching architecture
TiReX: Tiled Regular eXpressionsmatching architecture
 
Embedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposingEmbedding based knowledge graph link prediction for drug repurposing
Embedding based knowledge graph link prediction for drug repurposing
 
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
PLASTER - PYNQ-based abandoned object detection using a map-reduce approach o...
 
EMPhASIS - An EMbedded Public Attention Stress Identification System
EMPhASIS - An EMbedded Public Attention Stress Identification System EMPhASIS - An EMbedded Public Attention Stress Identification System
EMPhASIS - An EMbedded Public Attention Stress Identification System
 
Luns - Automatic lungs segmentation through neural network
Luns - Automatic lungs segmentation through neural networkLuns - Automatic lungs segmentation through neural network
Luns - Automatic lungs segmentation through neural network
 
BlastFunction: How to combine Serverless and FPGAs
BlastFunction: How to combine Serverless and FPGAsBlastFunction: How to combine Serverless and FPGAs
BlastFunction: How to combine Serverless and FPGAs
 
Maeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matchingMaeve - Fast genome analysis leveraging exact string matching
Maeve - Fast genome analysis leveraging exact string matching
 

Recently uploaded

What Causes BMW Chassis Stabilization Malfunction Warning To Appear
What Causes BMW Chassis Stabilization Malfunction Warning To AppearWhat Causes BMW Chassis Stabilization Malfunction Warning To Appear
What Causes BMW Chassis Stabilization Malfunction Warning To AppearJCL Automotive
 
Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...
Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...
Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...Delhi Call girls
 
Call me @ 9892124323 Call Girl in Andheri East With Free Home Delivery
Call me @ 9892124323 Call Girl in Andheri East With Free Home DeliveryCall me @ 9892124323 Call Girl in Andheri East With Free Home Delivery
Call me @ 9892124323 Call Girl in Andheri East With Free Home DeliveryPooja Nehwal
 
Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!
Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!
Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!AutoScandia
 
Delhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Callshivangimorya083
 
Greenery-Palette Pitch Deck by Slidesgo.pptx
Greenery-Palette Pitch Deck by Slidesgo.pptxGreenery-Palette Pitch Deck by Slidesgo.pptx
Greenery-Palette Pitch Deck by Slidesgo.pptxzohiiimughal286
 
John Deere Tractors 6130M 6140M Diagnostic Manual
John Deere Tractors 6130M 6140M Diagnostic ManualJohn Deere Tractors 6130M 6140M Diagnostic Manual
John Deere Tractors 6130M 6140M Diagnostic ManualExcavator
 
Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...
Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...
Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...amitlee9823
 
Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...
Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...
Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...anilsa9823
 
Business Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay Dubai
Business Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay DubaiBusiness Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay Dubai
Business Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay DubaiAroojKhan71
 
83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar
83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar
83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagardollysharma2066
 
Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...
Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...
Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...amitlee9823
 
Delhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Callshivangimorya083
 
Tata_Nexon_brochure tata nexon brochure tata
Tata_Nexon_brochure tata nexon brochure tataTata_Nexon_brochure tata nexon brochure tata
Tata_Nexon_brochure tata nexon brochure tataaritradey27234
 
design a four cylinder internal combustion engine
design a four cylinder internal combustion enginedesign a four cylinder internal combustion engine
design a four cylinder internal combustion enginepiyushsingh943161
 
Chapter-1.3-Four-Basic-Computer-periods.pptx
Chapter-1.3-Four-Basic-Computer-periods.pptxChapter-1.3-Four-Basic-Computer-periods.pptx
Chapter-1.3-Four-Basic-Computer-periods.pptxAnjieVillarba1
 
Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...
Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...
Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...amitlee9823
 

Recently uploaded (20)

What Causes BMW Chassis Stabilization Malfunction Warning To Appear
What Causes BMW Chassis Stabilization Malfunction Warning To AppearWhat Causes BMW Chassis Stabilization Malfunction Warning To Appear
What Causes BMW Chassis Stabilization Malfunction Warning To Appear
 
Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...
Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...
Call Girls in Malviya Nagar Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts Ser...
 
Call Girls in Shri Niwas Puri Delhi 💯Call Us 🔝9953056974🔝
Call Girls in Shri Niwas Puri Delhi 💯Call Us 🔝9953056974🔝Call Girls in Shri Niwas Puri Delhi 💯Call Us 🔝9953056974🔝
Call Girls in Shri Niwas Puri Delhi 💯Call Us 🔝9953056974🔝
 
(ISHITA) Call Girls Service Jammu Call Now 8617697112 Jammu Escorts 24x7
(ISHITA) Call Girls Service Jammu Call Now 8617697112 Jammu Escorts 24x7(ISHITA) Call Girls Service Jammu Call Now 8617697112 Jammu Escorts 24x7
(ISHITA) Call Girls Service Jammu Call Now 8617697112 Jammu Escorts 24x7
 
Call me @ 9892124323 Call Girl in Andheri East With Free Home Delivery
Call me @ 9892124323 Call Girl in Andheri East With Free Home DeliveryCall me @ 9892124323 Call Girl in Andheri East With Free Home Delivery
Call me @ 9892124323 Call Girl in Andheri East With Free Home Delivery
 
Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!
Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!
Why Won't Your Subaru Key Come Out Of The Ignition Find Out Here!
 
Delhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Vikaspuri 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
 
Greenery-Palette Pitch Deck by Slidesgo.pptx
Greenery-Palette Pitch Deck by Slidesgo.pptxGreenery-Palette Pitch Deck by Slidesgo.pptx
Greenery-Palette Pitch Deck by Slidesgo.pptx
 
John Deere Tractors 6130M 6140M Diagnostic Manual
John Deere Tractors 6130M 6140M Diagnostic ManualJohn Deere Tractors 6130M 6140M Diagnostic Manual
John Deere Tractors 6130M 6140M Diagnostic Manual
 
Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...
Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...
Top Rated Call Girls Mumbai Central : 9920725232 We offer Beautiful and sexy ...
 
Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...
Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...
Lucknow 💋 (Genuine) Escort Service Lucknow | Service-oriented sexy call girls...
 
Business Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay Dubai
Business Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay DubaiBusiness Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay Dubai
Business Bay Escorts $#$ O56521286O $#$ Escort Service In Business Bay Dubai
 
83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar
83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar
83778-77756 ( HER.SELF ) Brings Call Girls In Laxmi Nagar
 
Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...
Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...
Call Girls Kadugodi Just Call 👗 7737669865 👗 Top Class Call Girl Service Bang...
 
Delhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip CallDelhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
Delhi Call Girls Mayur Vihar 9711199171 ☎✔👌✔ Whatsapp Hard And Sexy Vip Call
 
Tata_Nexon_brochure tata nexon brochure tata
Tata_Nexon_brochure tata nexon brochure tataTata_Nexon_brochure tata nexon brochure tata
Tata_Nexon_brochure tata nexon brochure tata
 
design a four cylinder internal combustion engine
design a four cylinder internal combustion enginedesign a four cylinder internal combustion engine
design a four cylinder internal combustion engine
 
(INDIRA) Call Girl Surat Call Now 8250077686 Surat Escorts 24x7
(INDIRA) Call Girl Surat Call Now 8250077686 Surat Escorts 24x7(INDIRA) Call Girl Surat Call Now 8250077686 Surat Escorts 24x7
(INDIRA) Call Girl Surat Call Now 8250077686 Surat Escorts 24x7
 
Chapter-1.3-Four-Basic-Computer-periods.pptx
Chapter-1.3-Four-Basic-Computer-periods.pptxChapter-1.3-Four-Basic-Computer-periods.pptx
Chapter-1.3-Four-Basic-Computer-periods.pptx
 
Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...
Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...
Sanjay Nagar Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalor...
 

Sensor fusion of LiDAR and Camera for real time object detection - talk version

  • 1. Sensor fusion for autonomous driving perception Bagni Fabio 228594@studenti.unimore .it San Francisco - 15 May 2019 1
  • 2. Fabio Bagni San Francisco - 15 May 2019 Autonomous driving perception Precision and reactivity Risks of accidents, traffic laws respect. 2
  • 3. Fabio Bagni San Francisco - 15 May 2019 Embedded platforms 3
  • 4. Fabio Bagni San Francisco - 15 May 2019 Goals Real time 3D detection ● High performances ○ required 10 Hz frequency ● High precision ● Real time on embedded platforms 4
  • 5. Fabio Bagni San Francisco - 15 May 2019 UNIMORE HiPeRT prototype 5
  • 6. Fabio Bagni San Francisco - 15 May 2019 Sensors ● LiDAR ○ 3D point cloud ● Cameras ○ images Heterogeneous outputs 6
  • 7. Fabio Bagni San Francisco - 15 May 2019 Sensor fusion What is sensor fusion: ● Matching of different sensors views Why sensor fusion: ● Heterogeneous information of environment 7
  • 8. Fabio Bagni San Francisco - 15 May 2019 Fields of view LiDAR: 360 degrees around the vehicle Camera: wide angle, disposed do cover 360 degrees. 8 Cameras LiDAR
  • 9. Fabio Bagni San Francisco - 15 May 2019 Deep learning approach Camera: ● Bird’s Eye View estimation ● Object detection LiDAR ● Bird’s Eye View calculation ● Object detection 9 Image Point cloud 3D detection 3D detection Fusion
  • 10. Fabio Bagni San Francisco - 15 May 2019 Deep learning approach time Image stitching for multiple camera fusion 10 ➔ Under10 Hz Notenoughresponsive 1. Ming Liang, Bin Yang, Shenlong Wang and Raquel Urtasun: Deep Continuous Fusion for Multi-Sensor3D ObjectDetection.In: ECCV (2018).
  • 11. Fabio Bagni San Francisco - 15 May 2019 Static alignment sensor fusion Alignment of LiDAR with each camera individually. 11 Calibration Preprocessing
  • 12. Fabio Bagni San Francisco - 15 May 2019 Features extraction Matching is made by a calibration with a perforated panel. Holes are detected by all sensors. Holes’ centers are used as features to be matched. 12
  • 13. Fabio Bagni San Francisco - 15 May 2019 Cylindrical projection Fields of view matching is made projecting sensors’ outputs on the surface of a cylinder. 13
  • 14. Fabio Bagni San Francisco - 15 May 2019 Projections complexity LiDAR cylindrical projection: Depthmap O(n) : one operation per point 14 Camera cylindrical projection O(n) : one operation per pixel PARALLELIZABLE
  • 15. Fabio Bagni San Francisco - 15 May 2019 Calibration results 15
  • 16. Fabio Bagni San Francisco - 15 May 2019 Colored points Alignment of each point with the corresponding pixel on the image. 16
  • 17. Fabio Bagni San Francisco - 15 May 2019 Object detection Neural network for object detection on camera frame Clustering of LiDAR points Bounding boxes matching 17
  • 18. Fabio Bagni San Francisco - 15 May 2019 Object detection results 18
  • 19. Fabio Bagni San Francisco - 15 May 2019 Object detection average time 19 LiDAR output : 10 Hz Time limit : 100 ms
  • 20. Thanks for your attention Fabio Bagni - 228594@studenti.unimore.it 20 Fabio Bagni San Francisco - 15 May 2019
  • 21. 21
  • 22. Fabio Bagni San Francisco - 15 May 2019 Homography Homography allow to translate camera projection plane on the LiDAR projection plane. This method make cylindrical surfaces match. 22