A survey of recent research in image fusion for infra red sensors is presented. An appropriate approach is suggested based on requirements and related work.

1. Image Fusion:
Approaches in Hardware
- Kshitij Agrawal
Computer Vision Engineer
1
Github | Linkedin | Twitter

2. Image Fusion Approaches
• Fusion Algorithms
• FPGA Based Architectures
• Hybrid Architectures
2

3. FUSION ALGORITHMS
3

4. 4

5. HARDWARE ARCHITECTURRES
5

6. A Multi-sensor Image Fusion and Enhancement System for
Assisting Drivers in Poor Lighting Conditions
- Li Tao, Hau Ngo, Ming Zhang, Adam Livingston and Vijayan
Asari (2005)
Flow
• DRC + Contrast
enhancement – prior to
fusion
• Color restoration – with
chromatic info from CCD
post fusion
• Nonlinear neighborhood
dependent image
enhancement
• wavelet transform based
multiresolution image fusion
6

7. Hardware Utilization
• Pipelined enhancement
architecture utilizes about 50%
of the logic slices in Xilinx Virtex
II XC2V2000-4FF896 FPGA
• At 67 MHz sustain a throughput
rate of about 67 Mpx/sec
or 63.9 1024×1024
• Driving applications – night or
bad weather
7

8. Results of image
enhancement, image
alignment and image fusion:
(a) original visible image;
(b) enhanced visible image;
(c) original thermal image;
(d) ‘projective’ transformed
visible image (enhanced)
aligned with thermal image;
(e) fused image with color
restoration
8

9. Method of Image Fusion and Enhancement Using Mask
Pyramid
- David C. Zhang, Sek Chai and Gooitzen Van der Wal (SRI , 2011)
Flow
• Mask pyramid based selection
rule for LAP
• Acadia® II vision processor can
align and fuse three 1280x1024
images in real time
• If any intelligent analysis from
the source imager needs to be
used, it can be programmed into
mask pyramid.
• Flicker is a flashing effect occurs in fusion. This artifact is often seen
in the fusion of LWIR and SWIR or Visible images, when both image
sources contain reverse intensities at the same pixel locations. To
improve temporal coherence and reduce flicker artifacts, a hysteresis
mask is defined
9

10. FPGA System
10

11. (a) Visible image.
(b) IR image noise.
(c) Fusion without noise
mask. (d) Fusion with
noise mask. Note the
increased contrast and
speckle free sky.
11

12. Real-time Single FPGA-Based Multimodal Image Fusion System
& Fast and Adaptive Bi-dimensional Empirical
Mode Decomposition for the Real-time Video Fusion
- Michał Bartyś, Barbara Putz, Adrian Antoniewicz (2012)
Flow
• Real-time fusion algorithm goals - effective data processing, smooth fused video output & low
complexity allowing for direct low-level implementation.
• Candidates: Laplacian pyramid method [14], shift-invariant discrete wavelet transform (SIDWT)
method & FABEMD fusion algorithm [15, 16] based on the fast and adaptive bidimensional
mode decomposition.
12

13. 13
•low cost, low power device
•150K logic elements,
•6.48 Mb of embedded memory,
•360 18 × 18 multipliers,
•475 user I/O arranged in 11 banks.
•Running at 150MHz clock frequency.
•Two banks of external DDRAM2
memories - 2x64Mx32b.
•Synchronous burst SRAM (2Mx18b)
FPGA System

14. 14
Performance
•Tests have been done with means of COTS
kit based on Altera Cyclone IV FPGA
•Power dissipation of FPGA image fusion
system is rated <4W.
•UFO intended to manage real-time image
alignment and fusion operating at 50Hz and
above.

15. Hardware Utilization
• Al times are achieved for
input image resolution equal
640x480 pixels. FPGA is
synchronized by 150MHz
clock.
• Developed prototype of
real-time fusion system is
capable to process at 25 fps
• FPGA power dissipation
rated <1W while performing
real-time multimodal image
fusion
15
*Antoniewicz “FPGA Implementation of Decomposition Methods for
Real-Time Image Fusion”, Img Pr Com Challenges 4, Springer, 2013
(Detailed implementation)

16. Examples of multimodal image fusion of images
a:) TV image; b) IR image; c) TV image after morphological edge extraction;
d) IR image after morphological edge extraction; e) fusion by averaging; f) fusion
by SIDWT method; g) fusion by Laplacian pyramid; h) fusion by FABEMD method.
16

17. Others
Implementation of Image Fusion Techniques for Multi-Focus Images Using FPGA
- MA. Mohamed and R.M EI-Den2 (2011)
• DCT, DWT and PCNN based Fusion
• Xilinx Spartan 3AN
• (1) "lab" of size 640x480; (2) "Pepsi" of size 512x512, and (3) "Toy" of size 512x512.
17
Design and Implementation of Image Fusion Technique Using DWT for Micro Air
Vehicle - Dr. S. Narayana Reddy, C. Chandrasekhar (2014)
•Power consumption 0.10774W
•Operates at a maximum frequency of 456.73MHz
•Input 100x100 px @ 40Mhz

18. Hardware Utilization
18

19. HYBRID APPROACHES
19

20. • A mixed quasi-parallel structure with the DSP processors as well as FPGA matrices
has been seriously taken into account.
• Evaluation of computational power of DSP processor chosen for eventual
application in UFO system was done experimentally by application of special
software tests. Investigations were arranged by means of the off-the-shelf video
kit based on TMS320C6446 processor
20
Real-time Single FPGA-Based Multimodal Image Fusion System
- Michał Bartyś, Barbara Putz, Adrian Antoniewicz (2012)

21. Performance
• Application of chosen DSP processor does not guarantee achievement of
satisfactory processing times acceptable for implementation of image
registering and fusion at video frame rates equal 40ms.
21

22. Image Fusion Real-time System Based on FPGA and Multi-DSP
- Feng Qu, Bochao Liu, Jian Zhao, Qiang Sun (2013)
• In the final experiment, the image fusion of the five-band image with a frame rate
of 15Hz, and a resolution of 1392 × 1040, is successfully completed in 41ms by this
system.
• FPGA Alter Cyclone II - EP2C70F896, and DSP TI - TMS320C6416.
22

23. 23

24. QUALITATIVE ANALYSIS
24

25. • Experimental Tests of Image Fusion for Night Vision - Yin Chen RS Blum
(2005)
LAP and SIDWT gives best performance for Night vision
• Implementation of Image Fusion Techniques for Multi-Focus Images
Using FPGA - MA. Mohamed and R.M EI-Den2 (2011)
25

26. Conclusion
• Best algorithm - Gradient pyramid, mask pyramid,
FABEMD, SiDWT
• Least Hardware Complexity – LAP
• FPGA – low power (<1W), high throughput
(25fps, 1208x1024)
• High Bandwidth - Need multiple DDR connected
in parallel, SRAM is bonus
26