VRST2014 presentation Marker tracking is used in numerous applications. Depending on the context and its constraints, tracking accuracy can be a crucial component of the application. In this paper, we firstly highlight that the tracking accuracy depends on the illumination, which is usually not controlled in most applications. Particularly, we show how corner detection can shift of several pixels when light power or background context change, even if the camera and the marker are static in the scene. Then, we propose a method, based on the cross ratio invariance, that allows to re-estimate the corner extraction so that the cross ratio of the marker model corresponds to the one computed from the extracted corners in the image. Finally, we show on real data that our approach improves the tracking accuracy, particularly along the camera depth axis, up to several millimeters, depending on the marker depth.

1. Illumination Independent Marker Tracking using
Cross-Ratio Invariance
Vincent Agnus Stéphane Nicolau Luc Soler
Institut de Recherche Contre les Cancers de l'Appareil Digestif
ACM Symposium on Virtual Reality Software and Technology
(VRST2014)
November 11, 2014
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 1 / 18

2. Outline
Causes & Eects of light condition on corner detection
How cross-ratio can be used to correct corner shifting
Method validation
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 2 / 18

3. Motivation
Corner Detection : Illumination Conditions
Example
Light conditions inuence the estimation of corner position
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

4. Motivation
Corner Detection : Illumination Conditions
Example
Light conditions inuence the estimation of corner position
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

5. Motivation
Corner Detection : Illumination Conditions
Example
Light conditions inuence the estimation of corner position
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

6. Motivation
Corner Detection : Illumination Conditions
Example
Light conditions inuence the estimation of corner position
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

7. Motivation
Corner Detection : Illumination Conditions
Example
Light conditions inuence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ) World : 3 mm depth ) medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18

8. Motivation
Medical applications using markers
A low cost and accurate guidance system for
laparoscopic surgery : Validation on an
abdominal phantom [Nicolau VRST2005]
A low cost simulator to practice ultrasound
image interpretation and probe manipulation :
Design and rst evaluation [Nicolau ISVRI2011]
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18

9. Motivation
Medical applications using markers
Error on tag position
) Larger position errors for :
Needle tip
UltraSound virtual plane
A low cost and accurate guidance system for
laparoscopic surgery : Validation on an
abdominal phantom [Nicolau VRST2005]
A low cost simulator to practice ultrasound
image interpretation and probe manipulation :
Design and rst evaluation [Nicolau ISVRI2011]
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18

10. Motivation
Corner Detection
Shift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance
state of the art
Studies : marker detection , shifting ) Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

11. Motivation
Corner Detection
Shift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Radiometric
Response
Scene Radiance Measured Brightness
state of the art
Studies : marker detection , shifting ) Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

12. Motivation
Corner Detection
Shift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Radiometric
Response
Scene Radiance Measured Brightness
state of the art
Studies : marker detection , shifting ) Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

13. Motivation
Corner Detection
Shift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Rasterisation
Radiometric Spatial Intensity
Response
Scene Radiance Measured Brightness
state of the art
Studies : marker detection , shifting ) Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18

14. Proposed Method
Key Idea
Light Condition
underexposure : black region %, white region
overexposure : black region , white region %
) error on estimated position of the marker
How we can detect and correct this phenomena ?
Our method
use the marker geometry : inner outer corners
check that white/black region proportion are preserved
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 6 / 18

15. Proposed Method
Corner Detection Shift
Detection Correction
Cross Ratio (CR)
A projective invariant feature dened
for collinear points sets
A B C D
A0
B0
C0
D0
CR =
(AB=AC)
(DB=DC)
=
(A0B0=A0C0)
(D0B0=D0C0)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18

16. Proposed Method
Corner Detection Shift
Detection Correction
Cross Ratio (CR)
A projective invariant feature dened
for collinear points sets
A B C D
A0
B0
C0
D0
CR =
(AB=AC)
(DB=DC)
=
(A0B0=A0C0)
(D0B0=D0C0)
Cross Ratio on markers
A BC D
B A C
D
CR should be constant :
position light
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18

17. Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

18. Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

19. Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

20. Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18

21. Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18

22. Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18

23. Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of the
marker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18

24. Proposed Method
Correction of corner shifts
How cross ratio invariance can be used to x corners shifting ?
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 10 / 18

25. Proposed Method
Correction of corner shifts
Idea
Displace the corners such that CR(displaced) = CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18

26. Proposed Method
Correction of corner shifts
Idea
Displace the corners such that CR(displaced) = CRmodel
Method
H
(+dx;+dy)
(+dx;dy)
p1 p2 p5 p6
p3
p7
p14
p15
p4
p9
p8
p10p13
p12 p11 p16
p0
1
H(p1)
p0
i = H(pi) + (dx;dy)
(dx; dy) = argmin
dx;dy
X
(a;b;c;d)2I
(CR(p0
a; p0
b; p0
c; p0
d) CRmodel)2
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18

27. Proposed Method
Correction of corner shifts
New Corner Positions
H
(+dx;+dy)
(+dx;dy)
p1 p2p5 p6
p3
p7 p14
p15
p8 p10p13
p4 p9
p12 p11p16
p0
1
H(p1)
correct(pi) = H1
H(pi) + (dx;dy) | {z }
p0
i
corner in original image corrected corner
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 12 / 18

28. Proposed Method
Optimization Process
Minimization
argmin
dx;dy
X
(a;b;c;d)2I
(CR(p0
a; p0
b; p0
c; p0
d) CRmodel)2
Levenberg-Marquardt algorithm
not optimal : jacobian not provided, estimated from a
forward-dierence approximation
5 ms for each image : SciPy , (Core i7@2.2GHz), 10 lines for CR
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 13 / 18

29. Validation
Validation
Protocol
Metric
Distance camera $ Marker (before/after correction)
vs a reference distance (chessboard)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

30. Validation
Validation
Protocol
Metric
Distance camera $ Marker (before/after correction)
vs a reference distance (chessboard)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

31. Validation
Validation
Protocol
Metric
Distance camera $ Marker (before/after correction)
vs a reference distance (chessboard)
Pi
O
check
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

32. Validation
Validation
Protocol
Metric
Distance camera $ Marker (before/after correction)
vs a reference distance (chessboard)
Pi
Pproj
i
O
check
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18

33. Validation
Validation
Experiments
Chessboard
distance
cross ratio
Before correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

34. Validation
Validation
Experiments
cross ratio
After correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

35. Validation
Validation
Experiments
distance
After correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

36. Validation
Validation
Experiments
Logitech
c HD Pro
Webcam C910
19201080
Microsoft
c Lifecam 1280720 Logitech
c Procam 640480
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18

37. Validation
Validation
Experiments
3 cameras 5 orientations (range 40-60cm) 8 light conditions
3 groups : illumination condition.
over-exposure generate more errors.
without correction with correction
Under-exposure 1.15 0.22 mm 0.42 0.19 mm
Normal exposure 0.90 0.32 mm 0.67 0.22 mm
Over-exposure 2.60 0.70 mm 0.71 0.25 mm
Table: average errors of the marker depth estimation
Cross Ratio :
Can detect extreme light condition
Can correct depth estimation (1 mm)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 16 / 18

38. Discussion
Discussion Perspectives
Discussion
estimated homography H ! corner correction
assumption detected corners are collinear
assumption of isotropic inuence of light after homographic
rectication
Future Works
use inner corners of markers
minimization : analytical formulation ?
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 17 / 18

39. Q A
Thank you for your attention
vincent.agnus@ircad.fr
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 18 / 18