The use of multiple cameras applications is exploding. We're not just talking about 2 cameras for 3D or depth sensing, but 3-12 cameras for applications like drones, robotics, automotive, etc. The increasing use of multiple cameras combined with the growing use of mobile components such as apps processors, image sensors, displays, etc., being used by the broad market requires logic to connect these devices. In this presentation, Ted Marena of Microsemi explains how FPGAs can be used to leverage mobile components to aggregate a large number of MIPI CSI-2 camera interfaces.
2. Agenda
• History & adoption of MIPI CSI-2 image sensors
• MIPI CSI-2 interface primer
• FPGA usage models
• Applications for multiple MIPI CSI-2 image sensors with
FPGAs
• Summary
2
3. Evolution of Image Sensor Interfaces
Parallel CMOS interface subLVDS / HiSPi interface / etc.
In 90s & 2000s, parallel interface was the norm
In the next decade, various proprietary interfaces were introduced
~ 2-3MP
5. Because of mobile popularity, everyone
drifted to CSI-2
• Just like a decade ago when PC components were used
broadly, as mobile adoption exploded, so did the
acceptance of MIPI CSI-2
5
10. D-PHY vs. C-PHY
• The majority of MIPI CSI-2 cameras use the D-PHY.
C-PHY, 3-wire, embedded clock, 3-phase coding
D-PHY, diff-pairs, source synchronous, no coding
TransiUon at every symbol, used for clock recovery
11. Emulating D-PHY in FPGA
• Normally FPGAs do not have a D-PHY hard I/O block
• They usually interface via translation resistors
11
FPGA I/Os
Image
Sensor I/Os
12. Key Blocks Used in FPGAs for Imaging
12
AP/uP/uC
Math Blocks/DSP Memory Blocks
I/O Gearing Processor/Micro
13. FPGAs in Imaging/Video Applications
Bridging is the simplest design.
Acceleration requires more performance & capabilities.
13
FPGA
A B FPGA
HW ACC
A A’
DRAM
Op9onal
Bridging
AcceleraUon
14. FPGAs Also Do Processing & Aggregating
Processing could be in an embedded processor or with FPGA fabric,
memory and math blocks (DSP blocks).
Aggregation leverages the large I/O capability of FPGAs and the fabric.
14
FPGA
Aggrega9on
A
A’
DRAM
Processing AggregaUon
FPGA
Processing
A A’
B’
DRAM
B
C
15. Why Use an FPGA?
• Not enough MIPI CSI-2 or other inputs on an AP/ISP
• Limited numbers of ISP engines on an AP/ISP
• FPGA may perform some processing allowing for a lower
cost AP/ISP
• Newer capabilities that are not available with an AP/ISP
• An FPGA can implement a complete ISP
15
16. Multi MIPI CSI-2 Camera Applications
• 3D camera / virtual reality
• Dual surveillance
• Multiple-image sensor HDR
• 180-degree surveillance
• 360-degree panorama
• Surround view automotive
• Depth-detection applications
• Drone usage
16
18. 3D Camera FPGA Implementation
Dual Image Sensors
Bridge
CSI-2
CSI-2
1-4
Data
Lanes
Dual CSI-2 Bridge
CSI-2
1-4
Data
Lanes
Sync and
format logic
The FPGA can arrange the image in a side-by-side or a top-bonom configuraUon.
This makes it easier for the ISP or AP to process the image.
20. Dual Surveillance FPGA Function
This soluUon allows an ISP to process both images.
• Both images are recovered in the FPGA
• The FPGA combines the two images into
one for the ISP – oren a top-bonom
configuraUon
• This allows the ISP to process the two
images as one, but the output can be
split into two images
21. Image Sensor HDR Processing
CSI-2
May go to ISP/AP or storage
Each image sensor captures frames at exactly the
same time. A short, medium and long exposure is
used for each.
Image Buffer
Memory
FPGA with
HDR engine
22. Image Sensor HDR Processing
CSI-2
Short-, medium- and long-exposure images. Processed using local and global
tone mapping, motion artifact correction, etc.
Image Buffer
Memory
FPGA with
HDR engine
24. 180-Degree Surveillance FPGA Function
• MulUple images are recovered in the
FPGA
• The frames are stored likely in external
memory
• The FPGA performs an analysis to
determine where to merge the image
• The images are sUtched together
• Also likely a smoothing technique is used
• The image output is then processed in
the FPGA or formaned and passed onto
the ISP or AP
25. 360-Degree Cameras
CSI-2
Each image sensor frames are captured and combined.
Image processing could be in the FPGA or AP/ISP.
Image Buffer
Memory
FPGA with
sUtching,
fisheye
correcUon, etc
.
.
.
ISP or AP
26. 360-Degree Cameras
• The FPGA performs an analysis to determine
where to merge the images
• The images are stitched together
• Depending on the output format, fisheye
correction may be implemented
• The image output is then processed in the FPGA
or formatted and passed onto the ISP or AP
26
28. Surround View Application
• The FPGA implements the stitching of the images
• It formats the image for the ISP/AP
• Or FPGA processes the image and drives the display
• FPGA could add overlay such as directional lines
29. Multi Camera for Depth Detection
CSI-2
The FPGA processes all the images and
provides the lowest possible latency for
highest accuracy & response.
Image Buffer
Memory
FPGA syncs
dual camera
pairs & does
the processing
.
.
.
CSI-2
Could be one or more pairs
30. Multi Cameras for Depth-Based Analytics
Dual Image Sensors
The FPGA or AP computes 3D point cloud from a top-down stereo pair. Depth-based
analyUcs can disUnguish adults, kids, people from shopping carts for accurate people
counUng.
Top-Down
Depth
Image
FlowMetrics™ by PercepTonic
31. Depth Detection for Robotics
• The FPGA synchronizes each camera pair and processes what each
camera pair sees
• The robot can preciously determine the distance for each axis that
has a camera pair
• Parallel processing of the FPGA gives the robot the quickest
response & accuracy
32. Multi Camera Drone Application
CSI-2
These dual cameras allow a drone to “see.”
Image Buffer
Memory
FPGA syncs
dual camera
pairs, muxes,
encrypUon,
etc. .
.
.
CSI-2
GigE/10GE
Encrypted
video (wifi or
other RF)
PCIe
33. Multi Camera Drone Application
• The FPGA can merge, mux, pre-process, and run
analytics pre-processing before the AP gets involved for
3D point cloud.
Adds situational awareness so that drones do not crash into anything.
34. Summary
• Multiple camera applications continue to grow.
• MIPI CSI-2 adoption will grow and be further adopted
outside of mobile applications.
• An increasing number of FPGAs will support MIPI CSI-2
image sensors.
• The parallel architecture of the FPGA is ideal for multiple
camera applications.
34