SfMLearner + KF selectionを提案した"Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM [ICCV19]"を論文読み会で紹介した時の資料です.
WordPress Websites for Engineers: Elevate Your Brand
Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAMの解説
1. Unsupervised Collaborative Learning of
Keyframe Detection and Visual Odometry
Towards Monocular Deep SLAM [Sheng & Xu+, ICCV’19]
東京大学 相澤研究室
M2 金子 真也
2. 1
本論文
• Unsupervised Collaborative Learning of Keyframe Detection
and Visual Odometry Towards Monocular Deep SLAM
– 著者: L. Sheng, D. Xu, W. Ouyang and X. Wang
– 所属: Beihang University, Oxford, SenseTime
– 採択会議: ICCV2019
3. 2
本論文
• Unsupervised Collaborative Learning of Keyframe Detection
and Visual Odometry Towards Monocular Deep SLAM
– 著者: L. Sheng, D. Xu, W. Ouyang and X. Wang
– 所属: Beihang University, Oxford, SenseTime
– 採択会議: ICCV2019
– Monocular Deep SLAMを実現したいという強い気持ちの論文
– わかりみが深い
4. 3
Introduction
• Visual SLAM
– 3D reconstruction + Camera pose estimation
– 両者の同時最適化 (Bundle Adjustment)
Direct Sparse Odometry [Engel+, TPAMI’18]
5. 4
Introduction
• Deep Learning for Visual SLAM (Deep SLAM)
End-to-end Deep SLAMDL helps SLAM
SfMLearner [Zhou+, CVPR’17]
CNN-SLAM [Tateno+, CVPR’17]
CodeSLAM [Tateno+, CVPR’18]
DeepTAM [Zhou+, ECCV’18]
This figure is from Tombari’s presentation slide @ ICCVW.
6. 5
Introduction
• Deep Learning for Visual SLAM (Deep SLAM)
End-to-end Deep SLAMDL helps SLAM
CNN-SLAM [Tateno+, CVPR’17]
CodeSLAM [Tateno+, CVPR’18]
DeepTAM [Zhou+, ECCV’18]
SfMLearner [Zhou+, CVPR’17]
本論文の目標は,
この領域での最高のDeep SLAMを作ること
This figure is from Tombari’s presentation slide @ ICCVW.
7. 6
Related works
• SfMLearner [Zhou+, CVPR’17]
– 古典的なSfMを応用し, UnsupervisedにDeep SLAMを実現
– Training
• Photometric errorを最小化するように学習
8. 7
Related works
• SfMLearner [Zhou+, CVPR’17]
– 古典的なSfMを応用し, UnsupervisedにDeep SLAMを実現
– Inference
• 入力画像の奥行き画像と, 2視点間のカメラ姿勢をCNNで回帰
9. 8
Related works
• SfMLearner [Zhou+, CVPR’17]
– 古典的なSfMを応用し, UnsupervisedにDeep SLAMを実現
– Inference
• 入力画像の奥行き画像と, 2視点間のカメラ姿勢をCNNで回帰
より従来のVSLAMに近い
Deep SLAMを実現するためには???
29. 28
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(I𝑝𝑝, I𝑡𝑡) > th:
Insert tgt frame I𝑡𝑡 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
Model
30. 29
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {𝓘𝓘𝒔𝒔, 𝐈𝐈𝒑𝒑, 𝐈𝐈𝒏𝒏}
Train all the model
if iteration > 200 &
Similarity(I𝑝𝑝, I𝑡𝑡) > th:
Insert tgt frame I𝑡𝑡 as KF
Merge KF
ℐ𝑠𝑠
{𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
𝐈𝐈𝑡𝑡
Model
KF pool 𝒫𝒫 𝐾𝐾
Dataset
31. 30
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, I𝑝𝑝, I𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(I𝑝𝑝, I𝑡𝑡) > th:
Insert tgt frame I𝑡𝑡 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
ℐ𝑠𝑠
{𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
𝐈𝐈𝑡𝑡
Model Loss
Train
32. 31
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, I𝑝𝑝, I𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(I𝑝𝑝, I𝑡𝑡) > th:
Insert tgt frame I𝑡𝑡 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
ℐ𝑠𝑠
{𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
𝐈𝐈𝑡𝑡
Model Loss
Train
33. 32
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(𝐈𝐈𝑝𝑝, 𝐈𝐈𝑡𝑡) > th:
Insert tgt frame 𝐈𝐈𝒕𝒕 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
ℐ𝑠𝑠
𝐈𝐈𝑝𝑝
𝐈𝐈𝑡𝑡
Model Score
34. 33
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(𝐈𝐈𝑝𝑝, 𝐈𝐈𝑡𝑡) > th:
Insert tgt frame 𝐈𝐈𝒕𝒕 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
ℐ𝑠𝑠
𝐈𝐈𝑝𝑝
𝐈𝐈𝑡𝑡
Model Score
35. 34
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(𝐈𝐈𝑝𝑝, 𝐈𝐈𝑡𝑡) > th:
Insert tgt frame 𝐈𝐈𝒕𝒕 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
ℐ𝑠𝑠
𝐈𝐈𝑝𝑝
𝐈𝐈𝑡𝑡
Model Score
36. 35
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(𝐈𝐈𝑝𝑝, 𝐈𝐈𝑡𝑡) > th:
Insert tgt frame 𝐈𝐈𝒕𝒕 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
ℐ𝑠𝑠
𝐈𝐈𝑝𝑝
𝐈𝐈𝑡𝑡
Model Score
37. 36
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(I𝑝𝑝, I𝑡𝑡) > th:
Insert tgt frame I𝑡𝑡 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
𝐈𝐈𝑛𝑛𝐈𝐈𝑝𝑝
Model
ℐ𝑠𝑠
𝐈𝐈𝑡𝑡
Scores
38. 37
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, 𝐈𝐈𝑝𝑝, 𝐈𝐈𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(I𝑝𝑝, I𝑡𝑡) > th:
Insert tgt frame I𝑡𝑡 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
Model
39. 38
Proposed method
• Training
– Keyframe update & management
Random KF initialization
for epoch:
for iteration:
Choose training pair {ℐ𝑠𝑠, I𝑝𝑝, I𝑛𝑛}
Train all the model
if iteration > 200 &
Similarity(𝐼𝐼𝑝𝑝, 𝐼𝐼𝑡𝑡) > th:
Insert tgt frame 𝐼𝐼𝑡𝑡 as KF
Merge KF
KF pool 𝒫𝒫 𝐾𝐾
Dataset
Model
この操作を繰り返すことで
KF poolの最適化を行う