2017/06/10開催の第40回コンピュータビジョン勉強会＠関東「AR/VRを支える技術」の発表資料です。 https://kantocv.connpass.com/event/56917/

1. 光学シースルーHMDの高性能化
に向けたCV技術活用事例
2017/6/10, 第40回CV勉強会＠関東
伊藤勇太
http://campar.in.tum.de/Main/YutaItoh

2. 光学シースルーHMD
現実世界との整合性
視覚拡張
本日のトピック

3. Non-Optical (Video) See-Through
Head-Mounted Displays (HMD)
Optical See-Through (OST-HMD)

4. VR
HMDs
Video
See-Through
Optical
See-Through
Opaque
display
Scene camera

5. Hugo Gernsback,
TV Glasses
1963

6. Ivan Sutherland, The Sword of Damocles
1968

7. Ivan Sutherland, The Sword of Damocles
1968

8. @Computer History Museum in Mountain View

9. @Computer History Museum in Mountain View

10. HMDs (Non-OST, Virtual Reality)
HTC, HTC Vive,
2016
Oculus, Oculus Rift,
2016
SONY, PSVR, 2016

11. OST-HMDs (光学透過型)
EPSON BT-300,
2016
Microsoft HoloLens,
2016
ODG R-9,
2016
Meta Meta2,
2016

12. Kotaku, Minecraft Hololens demo at E3 2015
https://www.youtube.com/watch?v=xgakdcEzVwg

13. 2016-
https://twitter.com/ksasao/status/715569511684911105
Microsoft, Hololens

14. How to make AR
REAL?
* C. Sandor, M. Fuchs, A. CassinelZli, H. Li, R. Newcombe,
G. Yamamoto, and S. Feiner,“Breaking the Barriers to
True Augmented Reality”, 2015 Photo from Magic leap, 2016

15. Magic Leap, 2016
How to make AR real?

16.  ガチガチの自作ハード
 Computational Photography
OST-HMD研究分野 最近の傾向

17. OST-HMD: Optics Designs (Commercial)
(Free-form)
prism
Point
source
Waveguide
Laser
source
Half-mirror
Eye
Prism-based
Waveguide-based
Half-mirror (or cubic prism)
Classification from: Hainich and Bimber, Displays Fundamentals and Applications, 2011
Retinal Scanning
Vuzix
EPSON
Lumus Microvision
Optivent
NVIS
QD Laser

18. OST-HMD: Optics Designs (Research)
Coherent
light
Eye
Pinhole light array
(Pin-light)
Light field
(Microlens array)
Light field
(Stucked-LCD)
Holography
UNC-CH &
NVIDIA
2013
Moon et al. 2014
Backlight
Multiple
LCDs
Eye
LCD
Pinlight
LCD
Back
light
Takaki & Yamaguchi
2015
Micro
Lens
Array
Spatial light
modulator
UNC-CH
& NVIDIA
2014
Pinhole effect “focus free”
Maimone et
al. 2017

19. Temporal
[Zheng ’14] [Lincoln ’16]
Realism in AR (with OST HMDs)
Perceptual
[Kiyokawa ’01] [Liu ’08]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
[Maimone ’14][Yamaguchi ’16][Fukiage ’14] [Kalkofen ’13]

20. Realism in AR (with OST HMDs)
Don't Be A Glasshole | Mashable
Social?

21. Temporal
[Zheng ’14] [Lincoln ’16]
Realism in AR (with OST HMDs)
Perceptual
[Kiyokawa ’01] [Liu ’08]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
[Maimone ’14][Yamaguchi ’16][Fukiage ’14] [Kalkofen ’13]

22. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-
kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).
Spatial Realism in OST-HMDs

23. Spatial Realism in OST-HMDs

24. Eye-HMD Calibration
3D2D

25. Eye-HMD Calibration
3D

26. Off-axis Pinhole camera
3D2D
~

27. Manual method (De facto standard)
[Tuceryan ’00] [Genc ’02]
SPAAM
Single Point Active
Alignment Method

28. Track the position!
Itoh and Klinker, IEEE 3DUI 2014, IEEE ISMAR 2014
Automated method

29. Automated Spatial Calibration
Plopski, A., Itoh, Y., Nitschke, C., Kiyokawa, K., Klinker, G., & Takemura, H. (2015). Corneal-
imaging calibration for optical see-through head-mounted displays. IEEE transactions on
visualization and computer graphics, 21(4), 481-490.

30. Schematic Eye for Virtual Environments
Jones et al., IEEE 3DUI 2016

31. Eye-tracking OST-HMDs
Hua, Hong, and Chunyu Gao. "A compact eyetracked optical see-through
head-mounted display." IS&T/SPIE Electronic Imaging. International
Society for Optics and Photonics, 2012.

32. VR-HMD with eye-tracking
Stengel, Michael, et al. "An affordable solution for binocular eye
tracking and calibration in head-mounted displays." Proceedings
of the 23rd ACM international conference on Multimedia. ACM,

35. Eye
Environment
Medium
View-Dependent Lens Distortion
Distorted ray
HMD
optics
Image source
Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15

36. Light Field Calibration
World camera
User-view camera
Itoh and Klinker IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15

37. Light Field Calibration
IEEE TVCG‘15 (VR ’15) & IEEE ISMAR‘15

38. View-Dependent Point-Spread Func.
IEEE TVCG‘16 (IEEE ISMAR ’16)

39. Temporal
[Zheng ’14] [Lincoln ’16]
Realism in AR (with OST HMDs)
Perceptual
[Kiyokawa ’01] [Liu ’08]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
[Maimone ’14][Yamaguchi ’16][Fukiage ’14] [Kalkofen ’13]

40. Temporal Realism in OST-HMDs
“How fast is fast enough? : a study of the effects of latency in direct-touch pointing tasks”
Jota et al. CH’13 https://www.youtube.com/watch?v=PCbSTj7LjJg
“latencies down to 2.38 ms are required to alleviate
user perception when dragging”

41. Temporal Realism in OST-HMDs
Digital Light Processing Projector
“Minimizing Latency for Augmented Reality Displays: Frames
Considered Harmful”Zheng et al. ISMAR’14

42. Temporal Realism in OST-HMDs
Lincoln, Peter, et al. "From Motion to Photons in 80 Microseconds: Towards Minimal
Latency for Virtual and Augmented Reality." (2016), IEEE VR 2016 (Best paper award)

43. Temporal
[Zheng ’14] [Lincoln ’16]
Realism in AR (with OST HMDs)
Perceptual
[Kiyokawa ’01] [Liu ’08]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02] [Grubert ‘08]
[Maimone ’14][Yamaguchi ’16][Fukiage ’14] [Kalkofen ’13]

44. Color Consistency
Fukiage, T., Oishi, T., & Ikeuchi, K. (2014, September). Visibility-based blending for real-
time applications. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International
Symposium on (pp. 63-72). IEEE.

45. Color Consistency
IEEE TVCG‘15 (ISMAR‘15)

46. Real time pixel-wise
correction
Color Consistency
Langlotz, T., Cook, M., & Regenbrecht, H. (2016). Real-Time Radiometric Compensation for
Optical See-Through Head-Mounted Displays. IEEE Transactions on Visualization and Computer
Graphics, 22(11), 2385-2394.
Direct views

47. Depth (eye accommodation)
S. Liu, H. Hua, and D. Cheng,“A novel prototype for an optical see-
through head mounted display with addressable focus cues,”IEEE
Trans. Vis. Comput. Graphics, vol. 16, no. 3, pp. 381–393, May 2010.

48. Depth (eye accommodation)
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014,
July). Pinlight displays: wide field of view augmented reality eyeglasses using defocused
point light sources. In ACM SIGGRAPH 2014 Emerging Technologies (p. 20). ACM.
Accommodation
& Occlusion
Light Field Display (Multi-layer LCD)

49. Depth (eye accommodation)
The Light Field Stereoscope,
Huan et al., SIGGRPAH 2015

50. Depth (eye accommodation)
Computational focus-tunable near-eye displays R Konrad, N Padmanaban, E
Cooper, G Wetzstein, ACM SIGGRAPH 2016 Emerging Technologies, 3

51. Depth (eye accommodation)
Wide Field of View Varifocal Near-Eye Display using See-Through Deformable Membrane Mirrors
D. Dunn, C. Tippets, K. Torell, P. Kellnhofer, K. Aksit, P. Didyk, K. Myszkowski, D. Luebke, and H. Fuchs,
TVCG (IEEE VR 2017)
Accommodation
& Wide FoV
Deformable Membrane Mirrors

52. Depth (eye accommodation)
Wide Field of View Varifocal Near-Eye Display using See-Through Deformable Membrane Mirrors
D. Dunn, C. Tippets, K. Torell, P. Kellnhofer, K. Aksit, P. Didyk, K. Myszkowski, D. Luebke, and H. Fuchs,
TVCG (IEEE VR 2017)

53. Depth (eye accommodation)
“Holographic Near-Eye Displays for Virtual and Augmented Reality”
Microsoft Research, Andrew Maimone, Andreas Georgiou, Joel Kollin, Established: May 19, 2017
LCoS Computational Holography

54. Depth (eye accommodation)
Oculus Research ,“Focal Surface Display Discovery”
Nathan Matsuda, Alexander Fix, Douglas Lanman, SIGGRAPH 2017
LCoS“Computational Lens”

55. https://twitter.com/ksasao/status/715569511684911105
Field of View

56. Pinlight Displays
Field of View
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources”Maimone et al., TOG’14
Pinhole effect
 images are always in focus in theory

57. Field of View
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources”Maimone et al., TOG’14
Pinlight Displays

58. Field of View
Augmenting the Field-of-View of Head-Mounted Displays with Sparse Peripheral Displays
Xiao and Benko, CHI 2016

59. Occlusion (physical“Z-buffer”)
Kiyokawa, K., Billinghurst, M., Campbell, B., & Woods, E. (2003, October). An occlusion-
capable optical see-through head mount display for supporting co-located collaboration.
In Proceedings of the 2nd IEEE/ACM ISMAR (p. 133). IEEE Computer Society.

60. If we have a perfect HMD
in the future….
Vision Augmentation

61. 視覚拡張: Vision Augmentation
• Analysis of individual eyes
• Computational photography

62. Color Augmentation
Toshiyuki Amano, Oliver Bimber, and Anselm Grundhöfer. "Appearance Enhancement for Visually Impaired with
Projector Camera Feedback.“, TECHNICAL REPORT, BAUHAUS-UNIVERSITY WEIMAR, JANUARY 2009
E.g., assisting color-blinded people

63. Compensation
layer
Blurred
Enhanced
Defocus Cancelation
Itoh, Y., & Klinker, G. (2015, March). Vision enhancement: defocus correction via
optical see-through head-mounted displays. In Proceedings of the 6th
Augmented Human International Conference (pp. 1-8). ACM.

64. De-blurring OST-HMD screen
Oshima, K., Rompapas, D. C., Moser, K., Swan, E., Ikeda, S., Yamamoto, G., ...
& Kato, H. (2015). Sharpview: Improved legibility of defocussed content on
optical see-through head-mounted displays. In Demo under submission for:
ACM International Symposium on Mixed and Augmented Reality.

65. Predictive vision
Itoh et al. Augmented Human 2016 & SIGGRPAH 2016
OST-HMD + World sensing

66. Tracking system
User
OST-HMD
Virtual
Image
Itoh et al. Augmented Human 2016 & SIGGRPAH 2016

67. Eye adaptation assistance
Hiroi, Y., Itoh, Y., Hamasaki, T., & Sugimoto, M. (2017, March). AdaptiVisor: assisting eye
adaptation via occlusive optical see-through head-mounted displays. In Proceedings of
the 8th Augmented Human International Conference (p. 9). ACM.

68. Eye adaptation assistance
Hiroi, Y., Itoh, Y., Hamasaki, T., & Sugimoto, M. (2017, March). AdaptiVisor: assisting eye
adaptation via occlusive optical see-through head-mounted displays. In Proceedings of
the 8th Augmented Human International Conference (p. 9). ACM.

69. Augmented Vision
Itoh et al. Augmented Human 2016, SIGGRPAH 2016
Predictive vision

70. Tracking system
User
OST-HMD
Virtual
Image
Itoh et al. Augmented Human 2016, SIGGRPAH 2016

71.  現実世界との整合性
 時間的・空間的・視覚的
 視覚拡張
 Computational Photography
x Optical See-Through HMDs
HMD研究最先端つまみ食い
Research: http://campar.in.tum.de/Main/YutaItoh

72.  OST-HMDとAR応用 (MR, VR, CG)
 IEEE TVCG
 IEEE ISMAR, IEEE VR
 ACM ToG
 人間拡張
 Augmented Human
 OST-HMDの光学系
 OSA (Optical Society of America)
 SPIE (国際光工学会)
お勧めの論文誌・会議
Research: http://campar.in.tum.de/Main/YutaItoh