Presentation at Autosens 2016

1. Building ADAS
SYSTEM from scratch
Alex Myakov, Chief CV Advocate
Yury Gorbachev, CV Integration & Optimization lead
September, 2016

2. Who we are ?
Itseez was acquired by IoTG/Intel in July, 2016
Itseez was founded in 2005:
• 3 ex- Intel co-Founders + 1 Principal Engineer
• OpenCV development and support (2005-present)
• OpenVX initiative leaders: v1.0 and v1.1 were published in
October, 2014 and in May, 2016
SW Products: ADAS, Facense, AcceleratedCV (ACV)
Skills: CV algorithms, HW specific optimization, 3+
years of deep learning (DL)
Industries: automotive, security, robotics, wearables,etc

3. Building ADAS System from Scratch
The Dream and Ambition:
• Create state-of-the-art software based front
camera ADAS algos
• License such algos to Tier-1s and OEMs
Starting point (late 2013):
• Strong knowledge of CV
• Strong knowledge of embedded/optimization
• Good knowledge of cameras/sensors/optics
• No ADAS/automotive specific knowledge

4. Building ADAS solution from scratch
Strategy:
• Highly portable CV algos (pure ARM optimized code)
• Open SW platform (Android, Linux)
• Simplest system possible
• Cheapest COTS components
• Camera (optics, sensor, packaging)
• HW platform
• Easy/fast installation in any car with no dependence
on car parameters
• Automatic or simple calibration

5. ADAS Project Timeline
2013
• TSR
• Demo platform:
Nexus 4
2014
• +LDW
• +FCW
• Demo Platform:
Asus Transformer
Tablet + Android OS
+ Standalone
Camera
2015
• +PD
• Demo Platform:
TK1+ Linux OS +
Standalone Camera
• Demos with QNX:
• CES 2015
• TU Update 2015
2016
• PoCs:
• +Semantic Road
Segmentation
• +Obstacle Detection
• +Driver Monitoring
• Demos with QNX:
• CES 2016

6. Demo Setup
Camera:
• 1M
• 1280H x 800V
• HDR/WDR
• 30 fps
Embedded
Platform
USB 3.0
Snap-and-go concept:
• Simple and fast installation in any car
• No dependence on car parameters
• Automatic calibration or simple
calibration

7. Lessons Learned
We expected ADAS to be just another CV application !
• we ended up running into and solving lots of issues
• SW development/testing paradigm
• HW issues
• Datasets

8. ADAS Solution SW Architecture
TSR LDW FCW PD
Common Image Processing Pipeline + Autocalibration
OpenCV
IPP AcceleratedCV (ACV)
x86 ARM
CV algo prototyping on desktop
Lab testing/CI on server
Execution on target
Live test/benchmarking

9. SW design approach
ADAS algorithms are purely software based:
• Possible to design and test on desktops
• Purely based on OpenCV
• No special software skills are required (GPU, DSP,
etc)
• Flexible and upgradable
Solved platform compatibility issues
• No vendor provides cross-platform CV framework
• OpenCV is limited in supporting this
Created AcceleratedCV (ACV) library to address
platform compatibility issue

10. Continuous integration
Any change in ADAS algorithms or processing
pipeline requires complete re-evaluation
• Detection/processing quality on the entire dataset
• Performance figures for each ADAS algorithm
Benefits from pure SW based approach
• Quality evaluation on servers/cloud for entire dataset
• Performance benchmarking on multiple HW targets
• Reduces test time from days to hours!

11. Datasets
CV algorithms require datasets for design and
testing:
• No available commercial datasets
• Research datasets cannot be used for products
Our own datasets for each ADAS algo were created:
• Different conditions (rain, snow, sun)
• Geographical locations
• Dataset annotation and management tools
• Many days of driving + many months of annotation
The market offering of quality annotated datasets is
still very limited !

12. Datasets stats
TSR: 2.5K good unique signs
PD: 83K+ pedestrian bounding boxes
FCW: 5K+ different cars and ca 1K trucks
LDW: 0.5M+ boundaries

13. HW Issues
HW issues are caused by consumer “gradeness” of
components
Temperature issues:
• Camera overheating -> skipped or corrupted frames
• HW platform overheating -> throttling
System issues:
• throttling and unpredicted system behavior under
heavy processing loads
Mechanical issues:
• USB 3.0 cable connectors get loose and break

14. Where is Deep Learning in our algos?
Original ADAS algos were based on classical CV
• Embedded platforms were too weak and not able to
provide required performance Gflops
• Datasets were too small to yield quality DL results
• DL technology was not fully there
Conventional CV + small DL networks:
• PD: validation of PD results based on classical CV –
increase DR and reduce FA rate
• Driver monitoring: conventional face detector + DL
based headpose estimation

15. Semantic Road Segmentation - PoC

16. Obstacle Detection using SfM - PoC
•We estimate 3D coordinates using
points tracking and vehicle speed.
•Obstacles are calculated as
clusters of points above the road
plane.

17. Semantic segmentation + Obstacle Detection

18. Deep Learning - Silver Bullet ?
• Great new CV tool
• Large datasets
• A few orders of magnitude more compute
than classic CV
• FuSa implications

19. Questions ?
Contact Details:
Alex Myakov, Chief CV Advocate
Email: alex.myakov@intel.com