Mobile AR Lecture 8 - AR Panoramas

Researcher: Zi Siang See
Lecture
8:

Augmented
Reality
using
High
Fidelity

Spherical
Panorama
with
HDRI

Research
Presenta,on:
Zi
Siang
See

Human
Interface
Technology
Laboratory
New
Zealand
(HIT
Lab
NZ)

Email
(Industrial
collabora,on):
zisiang@reina.com.my

Zi
Siang
See,
Mark
Billinghurst,
Adrian
David
Cheok
(2015)

Taken from:
SIGGRAPH Asia 2015
Symposium
On
Mobile
Graphics
And
Interac,ve
Applica,ons

Zi
Siang
See,
Mark
Billinghurst,
Adrian
David
Cheok
(2015)
Augmented
Reality
using
High
Fidelity
Spherical
Panorama
with
HDRI.
SIGGRAPH
Asia
2015
Symposium
On
Mobile
Graphics
And
Interac,ve
Applica,ons.

hVp://dx.doi.org/10.1145/2818427.2818445

hVp://www.zisiangsee.com/research/zisiangsee_research_p009.pdf

Mobile-Based Augmented Reality Development

Research
Presenta?on

1)
Introduc?on,
Related
Work

2)
Method
and
Apparatus

3)
Discussion

4)
Prac,cal
Session

Figure:
AR
on
Mobile

•  Mobile
Augmented
Reality
(AR)
mixes
a
live
real-‐world
view
with
virtual

interac,ve
content
on
a
mobile
device.

•  One
of
the
key
enablers
for
this
is
tracking
technology,
such
as
computer

vision
techniques
for
tracking
oﬀ
pre-‐deﬁned
markers
or
markerless

images.

Introduc?on:
Augmented
Reality
(AR)
on
Mobile

Source: google search (keyword “augmented reality”)

Augmented
Reality
using
High
Fidelity
Spherical
Panorama
with
HDRI

High
fidelity
requirements
–
mobile
AR
environment
recogni?on

• Spherical
Panorama

• High
Dynamic
Range
Imaging
(HDRI)

• Op,miza,on
involves
obtaining
accurately
reproduced
spherical
panorama

with
least
acquisi,on
,me
-‐
AR
source
tracking
content
has
sufficient

luminance
by
using
HDRI
having
dynamic
range
extended
from
single

acquired
digital
nega,ve
for
each
angle
in
mul,row
configura,on.

Figure:
(a)
AR
using
marker.
(b)
AR
using
panorama
image,
markerless
experience.

•  There
have
been
previous
studies
on
using
pre-‐produced
panorama

images
for
AR
tracking
(Arth
et
al
2011;
DiVerdi
et
al.
2008;
Langlotz
et
al.

2014;
Ventura
and
Höllerer
2013;
Wagner
et
al.
2010).

•  Most
of
related
studies
describe
how
the
panorama
images
can
be
used

for
AR
tracking.

•  Therefore,
there
is
a
need
for
specifying
the
method
for
high
ﬁdelity

produc,on
of
the
source
panorama
images
for
mobile
AR.

Introduc?on:
AR
using
Panorama
Image

(a)
(b)

Case
Study:
Corporate
Project
Requirements

Townships

Landscape

Augmented
/Virtual

Other
Scope

Construc?on

Panorama
Image

Reproduc?on,
Projects

Heavy
Requirement

&
Workload/Quan?ty

Industrial
Case
Study:
Amount,
Time,
Quality

Figure:
Panorama
image
reproduc,on
process
for
AR
/VR
projects.

Introduc?on:
Spherical
Panorama
Acquisi?on

Figure:
reproduc,on
op,ons
available
for
fast
panorama
image
acquisi,on

Introduc?on:
Spherical
Panorama
Acquisi?on

LadyBug
GoPro
Rig
Mobile
(Handheld)

Figure:
reproduc,on
op,ons
available
for
fast
panorama
image
acquisi,on

Introduc?on:
Spherical
Panorama
Acquisi?on

LadyBug
GoPro
Rig
Mobile
(Handheld)

Parallax
error

Shake
/Blur

Resolu?on
Limited
Dynamic
Range

Inconsistent
ligh?ng
distribu?on

for
mul?ple
angles

Nadir
diﬃculty

Ghos?ng
eﬀect

(HDRI)

Inconsistent
white
balance
for

mul?ple
angles

Time
management

•  Acquire
wider
panoramic
visibility
(Benosman
and
Kang,
2001)

•  Major
diﬃculty
for
spherical
panorama
reproduc,on
may
occur,
this
can

include
limited
dynamic
range
reproduced
and
parallax
error
(Brown
and

Lowe,
2006;
Felinto
et
al,
2012)

Figure:
panoramic
imagery
with
visual
abnormality
-‐
loca,on
in
Sannomiya
Kobe,
Japan.

Introduc?on:
Spherical
Panorama
Acquisi?on

Camera
shake

Limited
dynamic
range
Visual
abnormality

Parallax
error

Figure:
combined
luminance
(EV)
range
being
covered
by
HDRI
outcome.

Darker
images

(luminance)
Brighter
images

(luminance)

Introduc?on:
HDRI
(for
mul?ple
angle
images)

•  High
Dynamic
Range
Imaging
(HDRI)
in
photography
technique
has
a
goal

to
increase
the
dynamic
range
resul,ng
an
improved
luminance

reproduced
in
shadow
and
highlight
areas
(Reinhard
et
al,
2010;
Debevec

and
Malik,
1997;
Fairchild,
2007),
usually
by
combining
mul,ple
exposures

consist
of
darker
and
brighter
images.


•  Obstacles
and
issues
of
HDRI
that
involves
mul,ple
exposures
can
include

lengthy
acquisi,on
(for
each
angle),
ghos,ng
eﬀect
and
misalignment

issue
due
to
inconsistent
acquisi,ons
(Reinhard
et
al,
2010)

Figure:
HDRI
with
visual
abnormali,es,
ghos,ng.

Introduc?on:
HDRI
(for
mul?ple
angle
images)

Mul,ple
exposures
for
HDRI
Single
exposure
(LDR)

Visual
abnormality

•  Spherical
panorama
can
be
used
for
AR
as
source
tracking
image.

•  Various
studies
(Felinto
et
al
2012,
Gawthrop
2007,
Brown
and
Lowe

2006;
Schulz,
2012)
have
shown
that
photographic
combina,on
for
HDRI

and
spherical
panorama
can
be
possible,
however
condi,onally
tolera,ng

with
obstacles
and
issues.

Spherical

Panorama
(Mul?row)
HDRI
requires
mul?ple
exposures

-‐4,
-‐3,
-‐2,
-‐1,
0,
+1,
+2,
+3,
+4
(EV)

Diﬃculty:
Spherical
Panorama
and
HDRI
for
AR?

Figure:
example
of
source
tracking
images
or
scenarios
for
AR

•  For
AR,
Lieberknecht
et
al.
[2009]
has
shown
that
it
can
be
difficult
to

create
source
tracking
images,
or
synthe,c
images
that
reproduce
the
real

effects
of
real-‐world
phenomena
such
as
ligh,ng,
noise,
mo,on
blur,

discre,za,on,
blooming
or
limited
color
depths
during
real
image

acquisi,on.

•  Image
elements
that
influence
real-‐,me
tracking
results
include
texture

richness,
the
texture
repeatability
of
the
objects
to
be
tracked,
the
camera

mo,on
and
speed,
the
changes
of
the
object
scale,
and
varia,ons
of
the

ligh,ng
condi,ons
over
,me.

Difficulty:
Source
Tracking
Images
for
AR

Source:
google
search
(keyword
“augmented
reality”)

Research
Presenta?on

1)
Introduc,on,
Related
Work

2)
Method
and
Apparatus

3)
Discussion

4) Practical Session

..
no
photographic
process
is
complete
un?l
the
ﬁnal

images
can
be
proudly
displayed
(Phase
One,
2015).

Method:
Spherical
Panorama

Addi,onal
3
nadir
acquisi,on

Horizontal
angles

Zenith

Nadir
image
acquired
without

including
the
base
of
the
camera.

Stable
moun,ng
is
required
for

avoiding
driring,
shake
and
parallax

error.

Nadir
without
obstacle,

allow
suﬃcient
space.

Method:
High
Dynamic
Range
Image
(HDRI)

PixelValue
Sequences of extendable dynamic range from single RAW
(photographic recording instrument Nikon D3x)
•  Extending
Dynamic
Range
from
single
acquired
Digital
Nega,ve
(RAW).

•  Consistent
to
the
idea
brought
up
by
Debevec
and
Malik
(1997)
about

extending
dynamic
range
can
be
possible
from
conven,onal
single
ﬁlm

processing
-‐
mul,ple
scanning
method.

Method:
High
Dynamic
Range
Image
(HDRI)

•  -‐2
EV
and
+2
EV,
resul,ng
an
addi,onal
increment
from
8.5EV
to
12.5EV.

• 
The
ﬁnding
in
the
study
about
na,ve
EV
of
dynamic
range
reproduc,on

was
observed
to
have
similar
approximated
usable
exposure
values
with

the
RAW
produced
by
ISO100
according
to
the
test
by
Rehm
(2009).

PixelValue
Sequences of extendable dynamic range from single RAW
(photographic recording instrument Nikon D3x)

Method:
High
Dynamic
Range
Image
(HDRI)

Figure:
an
image
from
mul,ple
angle
before
applying
HDRI.

without

Figure:
proposed
mul,ple
angles
HDRI
for
high
ﬁdelity
AR.

(HDRI
method
that
requires
a
single
acquisi,on
that
extends

the
dynamic
range
from
a
digital
nega,ve)

Method:
High
Dynamic
Range
Image
(HDRI)

with
HDRI

Source
Image
for
AR:
High
Fidelity
Spherical
Panorama
with
HDRI

mul?ple
angles
facilitated
by
HDRI
required
for

producing
the
spherical
panorama

(least
or
near-‐zero
visual
abnormality)

Mul?ple
angles
HDRI
of
mul?row
conﬁgura?on
(from
the
proposed

method)
with
reduced
error
or
mistakes
that
requires
rec?ﬁca?ons,

including
near-‐perfect
nadir
angle
reproduc?on.

Nadir
Angle:

Reproduc?on
from
the
original
loca?on-‐
based
scene,
minimal
post-‐produc?on

correc?on
required

cubic
projec?on

Source
Image
for
AR:
High
Fidelity
Spherical
Panorama
with
HDRI

mul?ple
angles
facilitated
by
HDRI
required
for

producing
the
spherical
panorama

(least
or
zero
visual
abnormality)

Source
Image
for
AR:
High
Fidelity
Spherical
Panorama
with
HDRI


HDRI
ghos?ng
eﬀect

using
mul?ple
exposures

method.

HDRI
reproduc?on
from

single
acquisi?on,
digital

nega?ve
(RAW)

visual
abnormality
is

eliminated
or
avoided

(for
mul?ple
angles

required
for
spherical

panorama)

Source
Image
for
AR:
High
Fidelity
Spherical
Panorama
with
HDRI

(a)
LDR
photo
image
(b)
HDRI
from
single
source
acquisi,on
(c)
HDRI
from
mul,ple
exposures

HDRI
Panorama
with
reduced
overall

photographic
produc?on
processes
and

?me.

Source
Image
for
AR:
High
Fidelity
Spherical
Panorama
with
HDRI

Augmented
Reality
using
High
Fidelity
Spherical
Panorama
with
HDRI

(Authoring
process
for
AR
environment
recogni?on)

Spherical
Panorama
facilitated
with
HDRI

(Cubic
projec,on)

Cropped
/selec,ve
cubic
facade

working
with
image-‐based
AR

Figure:
augmenta,on
can
be
assigned
on
any
cubic
facades,
speculated
to
work
on
most
of
the
AR
browsers
/workﬂow.

AR
for
Mobile
and
Wearable

Figure:
handheld
experience
of
markerless
AR.

AR
for
Mobile
and
Wearable

Figure:
handheld
experience
of
markerless
AR
(video).

Zeiss VR One
•  Carl Zeiss announced VR One, a virtual reality headset for use with a
smartphone. It is a viewer designed to work with phones between 4.7 and 5.2
inches (Zeiss, 2014).
•  Options for HMD using mobile device as computing module are becoming
available.

Samsung Gear VR
•  Samsung has launched Gear VR in October 2014, co-developed with Oculus.
Gear VR can house specific mobile device (Note 4) for providing virtual reality
and augmented user experience (Samsung, 2014)

AR
for
Mobile
and
Wearable

Figure:
stereo
image
of
AR
–
adaptable
for
HMD
such
as
Oculus
or
Google
Cardboard.

Source
Image
for
AR:
High
Fidelity
Spherical
Panorama
with
HDRI

Figure:
HDRI
spherical
panorama
sampled
at
Sannomiya
Sta,on,
Japan
(01
Nov
2015).

Discussion

Poten?al
obstacles
and
issues
(before
op?miza?on):

• 
Parallax
error
(especially
during
hand-‐held)

• 
Unstable
image
acquisi,on,
driring
mo,on
(hand-‐held)

• 
Nadir
angle
diﬃculty

• 
Compromised
geometrical
registra,on
in
image

• 
Low
dynamic
range
image
(LDRI)

• 
High
dynamic
range
requires
mul,ple
exposures

• 
Long
acquisi,on
,me
if
mul,ple
exposures

• 
High
dynamic
range
ghos,ng
with
moving
objects

• 
Mul,ple
exposures
misalignment

• 
Inconsistent
ligh,ng
distribu,on
for
mul,ple
angles

Beneﬁt
of
using
our
method
and
apparatus:

• 
Free
from
parallax
error

• 
Stable
image
acquisi,on

• 
Nadir
angle
with
near-‐error-‐free
authen,city

• 
Accurate
geometrical
registra,on
in
image

• 
High
dynamic
range
image
(HDRI)

• 
High
dynamic
range
from
single
RAW
acquisi,on

• 
High
dynamic
range
with
least
acquisi,on
,me

• 
High
dynamic
range
with
no
ghos,ng

• 
High
dynamic
range
with
perfect
alignment

• 
Consistent
high
dynamic
range
for
mul,ple
angles

Conclusion

•  Markerless
AR.

•  High
ﬁdelity
spherical
panorama
with
HDRI
can
provide
a
near-‐error-‐free

and
dynamic
range
enhanced
source
of
image-‐based
AR
tracking
content.

•  Source
tracking
content
is
reproduced
with
liVle
distor,on,
producing
a

result
very
similar
to
the
original
scene
condi,on.

•  Without
using
extra
sensors
such
as
a
GPS.

•  Allows
an
AR
experience
to
be
delivered
on
a
mobile
device
with
a
lower

processing
requirement
-‐
adaptable
in
other
wearable
system
or
devices.

Implica?ons

•  Android-‐based
mobile
with
RAW
capturing
capability.

•  Approach
adapta,on
in
VR/AR
workﬂow
for
mobile
applica,ons.

Future
Study

•  In
the
future
we
will
conduct
more
extensive
evalua,on
studies
to

compare
the
tracking
accuracy
with
the
systems
using
our
HDRI

panorama
images
to
other
tradi,onal
approaches.

•  We
will
also
explore
other
solu,ons
suitable
for
HDRI
panorama
video

and
hybrid
approaches
that
combine
panorama
image
tracking
and

sensor
input.

Thank you
(Next) Practical and Authoring
Zi
Siang
See,
Mark
Billinghurst,
Adrian
David
Cheok
(2015)
Augmented
Reality
using
High
Fidelity
Spherical
Panorama
with
HDRI.
SIGGRAPH
Asia
2015
Symposium
On
Mobile
Graphics
And
Interac,ve
Applica,ons.

hVp://dx.doi.org/10.1145/2818427.2818445


Research
Presenta?on

1)
Introduc,on,
Related
Work

2)
Method
and
Apparatus

3)
Discussion

4) Practical Session

HDRI
Panorama
reproduc,on

Interac,ve
panorama
output

AR
authoring

Tes,ng
and
simula,on

Real-‐world
tes,ng

Interac?ve
Panorama

Pano2vr
(or
other
tools)

Wikitude
Studio

hcp://studio.wikitude.com

Layar
Creator

hcp://www.layar.com

Load
Panorama
and
Try
AR

for
simula?on

(also
works
on
actual
loca?on)

Thank you
Research Discussion
collaboration /contact
www.zisiangsee.com
Zi
Siang
See,
Mark
Billinghurst,
Adrian
David
Cheok
(2015)
Augmented
Reality
using
High
Fidelity
Spherical
Panorama
with
HDRI.
SIGGRAPH
Asia
2015
Symposium
On
Mobile
Graphics
And
Interac,ve
Applica,ons.

hVp://dx.doi.org/10.1145/2818427.2818445


Additional

Method:
High
Dynamic
Range
Image
(HDRI)

Figure:
early
aVempt
for
HDRI
computa,on
from
the
source
of
single
acquired
digital
nega,ve,
RAW.

HDRI Computation
Input: Source image in digital negative RAW format (e.g. NEF, DNG)
Output: The HDRI is reproduced from a single acquired RAW image having its dynamic
range extended from RAW
1. Three images, I, I+, I- are processed from the single source of RAW.
2. I+ and I- are respectively altered by +2EV and -2EV where dynamic range are extended. That
is, the luminance lx of each pixel x is recalculated as (lx)new = 2m (lx)old, where m is the change in
EV.
3. A new image, I’+, is constructed out of I+, by preserving only the highlight regions of I+.
Similarly, a new image I’- is constructed out of I-, by preserving only the non-highlight regions
of I-.
4. I’+ is then overlaid on top of I’- with 50% opacity. The results of the two are then combined
to form the final image I’.

mul?ple
angles
of
mul?row
conﬁgura?on,
covering
horizontal,

zenith
and
nadir
angles.
(However
having
nadir
angle
diﬃculty,

parallax
error,
low
dynamic
range)

mul?ple
angle
images

Introduc?on:
Spherical
Panorama
Acquisi?on

1/8
sec,
f8

1/15
sec,
f8

1/30
sec,
f8

1/60
sec,
f8

1/125
sec,
f8

1/250
sec,
f8

1/500
sec,
f8

1/1000
sec,
f8

1/2000
sec,
f8

Scenario
1

(high
brightness)

HDRI
Acquisi?on:
Ligh?ng
Scenarios

Mul?ple
exposures
approach
Our
HDRI
approach

(dynamic
range
extended
from
single

acquired
digital
nega?ve,
RAW)

1/8
sec,
f8

1/15
sec,
f8

1/30
sec,
f8

1/60
sec,
f8

1/125
sec,
f8

1/250
sec,
f8

1/500
sec,
f8

1/1000
sec,
f8

1/2000
sec,
f8

Scenario
1

(high
brightness)

HDRI
Acquisi?on:
Ligh?ng
Scenarios

Mul?ple
exposures
approach
Our
HDRI
approach

(dynamic
range
extended
from
single

acquired
digital
nega?ve,
RAW)

Single

Acquisi,on

(RAW)

Process:
Extending

Dynamic
Range

Extended

+2EV

Original

0EV

Extended

-‐2EV

HDRI
from

single
source

HDRI
that
requires
single
acquisi,on
HDRI
that
requires
mul,ple
exposures

-‐4,
-‐3,
-‐2,
-‐1,
0,
+1,
+2,
+3,
+4
(EV)

1/2
sec,
f8

1/4
sec,
f8

1/8
sec,
f8

1/15
sec,
f8

1/30
sec,
f8

1/60
sec,
f8

1/125
sec,
f8

1/250
sec,
f8

1/500
sec,
f8

Scenario
2

(moderate
brightness)

Mul?ple
exposures
approach
Our
HDRI
approach

(dynamic
range
extended
from
single

acquired
digital
nega?ve,
RAW)

HDRI
Acquisi?on:
Ligh?ng
Scenarios

4
sec,
f8

2
sec,
f8

1
sec,
f8

1/2
sec,
f8

1/4
sec,
f8

1/8
sec,
f8

1/15
sec,
f8

1/30
sec,
f8

1/60
sec,
f8

Scenario
3

(low
light)

Mul?ple
exposures
approach
Our
HDRI
approach

(dynamic
range
extended
from
single

acquired
digital
nega?ve,
RAW)

HDRI
Acquisi?on:
Ligh?ng
Scenarios

Additional (extended)

Mobile AR Lecture 8 - AR Panoramas

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Mobile AR Lecture 8 - AR Panoramas

Similar to Mobile AR Lecture 8 - AR Panoramas (20)

More from Mark Billinghurst

More from Mark Billinghurst (20)

Recently uploaded

Recently uploaded (20)

Mobile AR Lecture 8 - AR Panoramas