An error measure for evaluating disparity maps is presented. It offers advantages over conventional ground-truth based error measures. Cabezas, I.; Padilla, V. & Trujillo, M. (2011), A Measure for Accuracy Disparity Maps Evaluation., in César San Martín & Sang-Woon Kim, ed., 'CIARP' , Springer, , pp. 223-231 .

1. A Measure for Accuracy
Disparity Maps Evaluation
Ivan Cabezas, Victor Padilla and Maria Trujillo
ivan.cabezas@correounivalle.edu.co
November 16th 2011
16th Iberoamerican Congress on Pattern Recognition, CIARP 2011, Pucón, Chile

2. Multimedia and Vision Laboratory
 MMV is a research group of the Universidad del Valle in Colombia
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 2

3. Content
 Multimedia and Vision Laboratory
 Stereo Vision
 Disparity
 Disparity Maps Evaluation
 Error Measures
 Problem Statement
 Illustration of the BMP
 The Sigma Z Error Measure
 Illustration of the SZE
 Impact on Evaluation Results
 Final Remarks
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 3

4. Stereo Vision
 The stereo vision problem is to recover the 3D structure of a scene using
two or more images
Stereo Images Correspondence
Algorithm
Disparity Map Reconstruction
Algorithm
Left Right
3D World
3D Model
Optics
Problem
Camera Inverse
System Problem
2D Images
Yang Q. et al., Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling, IEEE PAMI 2009
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 4

5. Disparity
 Disparity is the distance between corresponding points
Trucco, E. and Verri A., Introductory Techniques for 3D Computer Vision, Prentice Hall 1998
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 5

6. Disparity Maps Evaluation
 The evaluation of stereo correspondence algorithms is based on obtained
disparity maps
Ground-truth Map Error Criteria
all nonocc disc
Scharstein, D. and Szeliski, R., A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, IJCV 2002
Scharstein, D. and Szeliski, R., High-accuracy Stereo Depth Maps using Structured Light, CVPR 2003
Mei, X., et al., On Building an Accurate Stereo Matching System on Graphics Hardware, GPUCV 2011
Cabezas, I. and Trujillo M., A Non-Linear Quantitative Evaluation Approach for Disparity Estimation, VISAPP 2011
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 6

7. Error Measures
 The selection of an error measure has an impact on evaluation results
Van der Mark, W., Gavrila, D., Real-time Dense Stereo for Intelligent Vehicles IEEE Trans. on Intelligent Transportation Systems, 2006
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 7

8. Problem Statement
 Conventional error measures have different drawbacks:
 The use of the mean, which is sensitive to extreme values
 Ignoring the inverse relation between depth and disparity
• Disparity errors of the same magnitude may have different impacts on
depth reconstruction
• Small disparities are difficult to estimate and sensitive to errors
 In particular, the BMP measure, which is widely used:
 It ignores the error magnitude
 It is sensitive to the error threshold selection
 It measures the quantity of errors
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 8

9. Illustration of the BMP
 A low percentage of disparity estimation errors does not imply an
accurate depth recovering
Scharstein, D. and Szeliski, R., A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, IJCV 2002
Scharstein, D. and Szeliski, R., High-accuracy Stereo Depth Maps using Structured Light, CVPR 2003
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 9

10. The Sigma Z Error Measure
 We propose the Sigma Z Error (SZE) measure
 The proposed measure has the following properties:
 It is inherently related to depth reconstruction in a
stereo system
 It is based on the inverse relation between depth and
disparity
 It considers the magnitude of the estimation error
 It is threshold free
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 10

11. Illustration of the SZE
 The SZE measures the impact of disparity estimation errors in terms of
distance along the Z axis of the stereo system
ADCensus Teddy ADCensus Cones RDP Teddy RDP Cones
Mei, X., et al., On Building an Accurate Stereo Matching System on Graphics Hardware, GPUCV 2011
Sun, X., et al., Stereo Matching with Reliable Disparity Propagation, 3DIMPVT 2011
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 11

12. Impact on Evaluation Results
 Two different evaluation models were used in the experimental validation
Cabezas, I. and Trujillo M., A Non-Linear Quantitative Evaluation Approach for Disparity Estimation, VISAPP 2011
Scharstein, D. and Szeliski, R., http://vision.middlebury.edu/stereo/, 2011
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 12

13. Final Remarks
 The SZE offers advantages over conventional error
measures such as the BMP since it considers the inverse
relation between depth and disparity
 A better judging of algorithms behaviour is obtained using
the SZE
 The SZE is suited to evaluate disparity estimations in
different application domains, such as: robotics, unmanned
navigation, and automatic inspection, among others
A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011 Slide 13

14. A Measure for Accuracy
Disparity Maps Evaluation
Ivan Cabezas, Victor Padilla and Maria Trujillo
ivan.cabezas@correounivalle.edu.co
November 16th 2011
16th Iberoamerican Congress on Pattern Recognition, CIARP 2011, Pucón, Chile