2015.05.20的seminar報告，就當作提早把碩論要講的內容大致整理一次。

1. 3D Pose Estimation for
Transparent Objects
Presenter: 賴柏任
Advisor:羅仁權教授
05.20.2015

2. Motivation
• Transparent objects are everywhere
• If we know he pose, we can grasp it!
2

3. Problems
3
Color of
transparent
object changes
Hard to locate
transparent
objects
Edge of
transparent objects
are blur
Hard to estimate
pose of
transparent objects

4. Effective cure
4
Color of
transparent
object changes
Edge of
transparent
objects are blur

5. Kinect v.s. Color changes
• Transparent objects produce NaN in
depth map
5
Ref: I. Lysenkov and V. Rabaud, "Pose estimation of rigid transparent objects in
transparent clutter," in Robotics and Automation (ICRA), 2013 IEEE International
Conference on, 2013, pp. 162-169.

6. Graphcut v.s. Blur edge
• Given foreground & background clue
6
Ref: C. Rother, V. Kolmogorov, and A. Blake, "Grabcut: Interactive foreground
extraction using iterated graph cuts," ACM Transactions on Graphics (TOG), vol.
23, pp. 309-314, 2004.

7. Graphcut v.s. Blur edge
• Generate the prob. distribution
7
Ref: C. Rother, V. Kolmogorov, and A. Blake, "Grabcut: Interactive foreground
extraction using iterated graph cuts," ACM Transactions on Graphics (TOG), vol.
23, pp. 309-314, 2004.

8. Graphcut v.s. Blur edge
• Use distance to compensate
8
Ref: C. Rother, V. Kolmogorov, and A. Blake, "Grabcut: Interactive foreground
extraction using iterated graph cuts," ACM Transactions on Graphics (TOG), vol.
23, pp. 309-314, 2004.

9. Graphcut v.s. Blur edge
• OpenCV implementation
9

10. A coarse pipeline
10
Detect NaN
area in
depth map
Feed the
area to
Graphcut
Segment
the edge

11. How to determine pose?
• Model-based matching
• Rotate in x & y axis and store the edge
11
Z-axis Y-axis
The problem becomes a 2D-
2D matching problem

12. Where is the model?
• Kinect Fusion
12

13. Where is the model?
Wrap your
object with
paper
Use Kinect
Fusion to
construct the
model
Store the model
13

14. What if there are some other NaN
objects?
• Some non-transparent objects also
produce NaN in depth map
14

15. What if there are some other NaN
objects?
• Use characteristics of transparent object
to rule out non-transparent objects
15
Transparent
objects produce
highlights
Color of transparent
object is similar to
peripheral area

16. What if there are some other NaN
objects?
• Transparent objects produce highlights
16
Ref: K. McHenry, J. Ponce, and D. Forsyth, "Finding glass," in Computer Vision
and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference
on, 2005, pp. 973-979.

17. What if there are some other NaN
objects?
• Transparent objects produce highlights
17
Ref: K. McHenry, J. Ponce, and D. Forsyth, "Finding glass," in Computer Vision
and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference
on, 2005, pp. 973-979.
Threshold the image from 0-255
Compute the perimeter in each image
Compute the threshold by line fitting (from
255 to 0)

18. What if there are some other NaN
objects?
• Color of transparent object is similar to
peripheral area
18

19. What if there are some other NaN
objects?
• Color of transparent object is similar to
peripheral area
19
Hue histogram

20. A fine pipeline
20

21. Some results
• Pose Matching
21

22. Some results
• Total retrieved candidates are over 200
22
Method Recall Precision
Only NaN 86.11% 38.24%
Characteristics 86.11% 93.93%
Recall = (2/2)*100% =100%
Precision=(2/5)*100% =40%

23. Some other problems
• How to let robot grasp?
• Is there any choice other from Kinect?
23

24. How to let robot grasp?
• Teach and Play
24
Grasp
points

25. Is there any choice other from
Kinect?
• Extract the visual word of transparent
objects
25

26. Is there any choice other from
Kinect?
26
Ref: M. Fritz, G. Bradski, S. Karayev, T. Darrell, and M. J. Black, "An additive latent
feature model for transparent object recognition," in Advances in Neural
Information Processing Systems, 2009, pp. 558-566.

27. Is there any choice other from
Kinect?
• The result can be the input of Graphcut
27
Ref: M. Fritz, G. Bradski, S. Karayev, T. Darrell, and M. J. Black, "An additive latent
feature model for transparent object recognition," in Advances in Neural
Information Processing Systems, 2009, pp. 558-566.

28. 28
Thank you!