Mobile Graphics (part2)

www.crs4.it/vic/
Visual Computing Group
Visual Computing – UniCa, June 17th, 2015
Part 2 – Mobile graphics trends
• Marco Agus
• Marcos Balsa
June 2015
1

Marco Agus & Marcos Balsa
Applications
• Wide range of applications
– Cultural Heritage
– Medical Image
– 3D object registration
– GIS
– Gaming
– VR & AR
- Building reconstruction
- Virtual HCI
3D representation
+ additional information
2

Mobile 3D interactive graphics
• General pipeline similar to standard interactive
applications
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
3

MOBILE
DEVICE
SERVER
Remote rendering
• General solution since first
PDAs
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
NETWORK
4

Remote rendering
• 3D graphics applications require intensive
computation and network bandwidth
– electronic games
– visualization of very complex 3D scenes
• Remote rendering has long history and it is
successfully applied for gaming services
[OnLive]
– Limitation: interaction latency in cellular networks
6

MOBILE DEVICE
SERVER
Mixed Mobile/Remote rendering
• As mobile GPUs progress...
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
NETWORK
7

• Model based versus Image based methods
• Model based methods
– Original models
– Partial models
– Simplified models
• Couple of lines
• Point clouds
Eisert and Fechteler. Low delay streaming of computer
graphics (ICIP 2008)
Gobbetti et al. Adaptive Quad Patches: an Adaptive Regular
Structure for Web Distribution and Adaptive Rendering of
3D Models. (Web3D 2012)
Balsa et al.,. Compression-domain Seamless Multiresolution
Visualization of Gigantic Meshes on Mobile Devices (Web3D
2013)
Diepstraten et al., 2004. Remote Line Rendering for Mobile
Devices (CGI 2004)
Duguet and Drettakis. Flexible point-based rendering on
mobile devices (IEEE Trans. on CG & Appl, 2004)
9

• Model based versus Image based methods
• Model based methods
– Original models
– Partial models
– Simplified models
• Couple of lines
• Point clouds
Eisert and Fechteler. Low delay streaming of computer
graphics (ICIP 2008)
Gobbetti et al. Adaptive Quad Patches: an Adaptive Regular
Structure for Web Distribution and Adaptive Rendering of
3D Models. (Web3D 2012)
Balsa et al.,. Compression-domain Seamless Multiresolution
Visualization of Gigantic Meshes on Mobile Devices (Web3D
2013)
Diepstraten et al., 2004. Remote Line Rendering for Mobile
Devices (CGI 2004)
Duguet and Drettakis. Flexible point-based rendering on
mobile devices (IEEE Trans. on CG & Appl, 2004)
10

• Image based methods
– Image impostors
– Environment maps
– Depth images
Noimark and Cohen-Or. Streaming scenes to mpeg-4 video-enabled devices (IEEE,
CG&A 2003)
Lamberti and Sanna. A streaming-based solution for remote visualization of 3D
graphics on mobile devices (IEEE, Trans. VCG, 2007)
Bouquerche and Pazzi. Remote rendering and streaming of progressive panoramas
for mobile devices (ACM Multimedia 2006)
Zhu et al. Towards peer-assisted rendering in networked virtual environments (ACM
Multimedia 2011)
Shi et al. A Real-Time Remote Rendering System for Interactive Mobile Graphics
(ACM Trans. On Multimedia, 2012)
Doellner et al. Server-based rendering of large 3D scenes for mobile devices using G-
buffer cube maps ( ACM Web3D, 2012)
11

Mobile visualization systems
• Volume rendering
• Point cloud rendering
Moser and Weiskopf. Interactive volume rendering on mobile devices. Vision,
Modeling, and Visualization VMV. Vol. 8. 2008.
Noguerat al. Volume Rendering Strategies on Mobile Devices. GRAPP/IVAPP. 2012.
Campoalegre, Brunet, and Navazo. Interactive visualization of medical volume models
in mobile devices. Personal and ubiquitous computing 17.7 (2013): 1503-1514.
Balsa et al. Interactive exploration of gigantic point clouds on mobile devices. ( VAST
2012)
He et al. A multiresolution object space point-based rendering approach for mobile
devices (AFRIGRAPH, 2007)
12
Rodríguez, Marcos Balsa, and Pere Pau Vázquez Alcocer. Practical Volume Rendering
in Mobile Devices. Advances in Visual Computing. Springer, 2012. 708-718.

Mobile visualization systems
• Volume rendering
• Point cloud rendering
Moser and Weiskopf. Interactive volume rendering on mobile devices. Vision,
Modeling, and Visualization VMV. Vol. 8. 2008.
Noguerat al. Volume Rendering Strategies on Mobile Devices. GRAPP/IVAPP. 2012.
Campoalegre, Brunet, and Navazo. Interactive visualization of medical volume models
in mobile devices. Personal and ubiquitous computing 17.7 (2013): 1503-1514.
Balsa et al. Interactive exploration of gigantic point clouds on mobile devices. ( VAST
2012)
He et al. A multiresolution object space point-based rendering approach for mobile
devices (AFRIGRAPH, 2007)
13
Rodríguez, Marcos Balsa, and Pere Pau Vázquez Alcocer. Practical Volume Rendering
in Mobile Devices. Advances in Visual Computing. Springer, 2012. 708-718.

Trends in mobile graphics
• Hardware acceleration for improving
frame rates, resolutions and rendering
quality
– Parallel pipelines
– Real-time ray tracing
– Multi-rate approaches
18

SGRT: Real-time ray tracing
• Samsung
reconfigurable GPU
based on Ray Tracing
• Main key features:
– an area-efficient parallel
pipelined traversal unit
– flexible and high-
performance kernels for
shading and ray
generation
Lee, Won-Jong, et al. SGRT: A mobile GPU architecture for real-time ray
tracing. Proceedings of the 5th High-Performance Graphics Conference, 2013.
Shin et al., Full-stream architecture for ray tracing with efficient data
transmission, 2014 IEEE ISCAS
19

Adaptive multi-rate shading
• The multi-rate shading pipeline
rasterizes triangles into coarse
fragments that correspond to
multiple pixels of coverage
• Coarse fragments are shaded, then
partitioned into fine fragments for
subsequent per-pixel shading
• If the coarse fragment shader
determines an effect should not be
evaluated at low sampling rates,
these computations are performed
in the fine shading stage
• Complex pipeline scheduling reduce
shading costs to more than three
and sometimes up to a factor of
five.
He et al. Extending the graphics pipeline with adaptive, multi-rate
shading. ACM Transactions on Graphics (TOG) 33.4 , 2014.
20

Adaptive multi-frequency shading
• Real-time rendering of tessellated geometry is still fairly expensive,
as current pixel shading methods, e.g. MSAA), do not scale well
with geometric complexity.
• Pixel shading is tied to the coarse input patches and reused
between triangles, effectively decoupling the shading cost from the
tessellation level.
• AMFS can evaluate different parts of shaders at different
frequencies, allowing very efficient shading.
Clarberg, Petrik, et al. AMFS: adaptive multi-frequency shading for future
graphics processors. ACM Transactions on Graphics (TOG) 33.4 , 2014.
21

Coarse pixel shading
• Architecture for varying
shading rates in a
rasterization pipeline,
while keeping the
visibility sampling rate
constant
• A single rendering pass
executes shading code
at one or more different
rates: per group of
pixels, per pixel, and per
sample
Vaidyanathan, Karthik, et al. Coarse Pixel Shading. Eurographics/ACM
SIGGRAPH Symposium on High Performance Graphics, 2014.
22

MOBILE DEVICE
Mobile rendering with capture
• Exploiting mobile device sensors...
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
CAPTUREEnvironment
23

MOBILE DEVICE
Mobile rendering with capture
• Exploiting mobile device sensors...
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
CAPTUREEnvironment
Example:
Google Tango
https://www.google.com/ata
p/projecttango/#project
24

3D acquisition and processing
• Live 3D reconstruction on
mobile phones
• The system allows to obtain
3D models of pleasing quality
interactively and entirely on-
device
• Efficient and accurate
scheme for integrating
multiple stereo-based depth
hypotheses into a compact
and consistent 3D model
Kolev et al. Turning Mobile Phones into 3D Scanners (CVPR 2014)
Tanskanen et al. Live Metric 3D Reconstruction on Mobile Phones (ICCV
2013)
25

Physical simulations
• Framework for physically and chemically-based
simulations of analog alternative photographic
processes
• Efficient fluid simulation and manual process
running on iPad
Echevarria et al. Computational simulation of alternative photographic
processes. Computer Graphics Forum. Vol. 32. 2013.
26

Correcting visual aberrations
• Computational display
technology that
predistorts the
presented content for
an observer, so that
the target image is
perceived without the
need for eyewear
• Demonstrated in low-
cost prototype mobile
devices
Huang, Fu-Chung, et al. Eyeglasses-free display: towards correcting visual
aberrations with computational light field displays.ACM Transactions on Graphics
(TOG) 33.4, 2014.
27

Mobile Graphics
Next:
Real time visualization of massive models
on mobile systems
28

www.crs4.it/vic/
Visual Computing Group
Mobile Graphics
• Marco Agus
• Marcos Balsa
June 2015
Real time visualization of massive models on
mobile systems
29

Visualization techniques
Problems
– Large meshes
– Scenes with complex lighting
– Volume rendering
30

Our Contributions: chunked
multiresolution structures
• Two approaches
– Fixed coarse subdivision
• Adaptive QuadPatches
– Multiresolution inside patch
– Adaptive coarse subdivision
• Compact Adaptive TetraPuzzles
– Global multiresolution
31

Adaptive Quad Patches
Simplified Streaming and Rendering for the Web
• Represent models as fixed number of multiresolution quad
patches
– Image representation allows component reuse!
– Natural multiresolution model inside each patch
– Adaptive rendering handled totally within shaders!
• Works with topologically simple models
Best paper, WEB3D2012
Javascript!
32

Visual Computing – UniCa, June 17th, 2015 33
• Generate clean manifold triangle mesh
• Poisson reconstruction [Kazhdan et al. 2006]
• Remove topological noise
• Discard connected components with too few
triangles
• Parameterize the mesh on a quad-based domain
• Isometric triangle mesh parameterization
• Abstract domains [Pietroni et al. 2010]
• Remap into a collection of 2D square regions
• +Vertex in tri. Barycenter / +Quad at each
edge
• Resample each quad from original geometry
• Associates to each quad a regular grid of samples
(position, color and normal) -> image
• Multiresolution inside quad (image mip pyramid)
Pre-processing (Reparameterization)
Quad-based mesh
Base coarse mesh

Pre-processing (Multiresolution)
• Collection of variable resolution quad patches
– Coarse representation of the original model
• Multiresolution pyramids
– Detail geometry, Color, Normals
• Shared border information
– Ensure connectivity
34

Adaptive rendering
• 1. CPU Lod Selection
– Different quad LOD, but must agree on edges
– Quad LOD = max edge LOD (available)
– Use finest LOD available
– Send VBO with regular grid (1 for each LOD)
• 2. GPU: Vertex Shader
– Snap vertices on edges (match neighbors)
– Base position = corner interpolation (u,v)
– Displace VBO vertices
– normal + displacement (dequantized)
• 3. GPU: Fragment Shader
– Texturing & Shading
35
0,0 u,v
1,1

Rendering example
Patches Levels Shading
36

Results
• Adaptive rendering
– 1 pixel accuracy – 37
fps (min 13 fps)
• Network streaming
– Required 312Kbps (max
2.8Mbps) for no delay
– ADSL 8Mbps – 2 s fully
refined model from scratch
37

Conclusions: Adaptive Quad Patches
• Effective creation and distribution system
– Fully automatic
– Compact, streamable and renderable 3D model representations
– Low CPU overhead  GPU adaptive rendering
– WebGL
• Desktop
• Mobile
• Limitations
– Closed objects with large components (i.e,3D scanned objs)
– Visual approximation (lossy)
– Explore more aggressive compression techniques
38

Compact Adaptive Tetra Puzzles
Efficient distribution and rendering for mobile
39
• Built on Adaptive TetraPuzzles [CRS4+ISTI CNR, SIGGRAPH’04]
– Regular conformal hierarchy of tetrahedra
– Spatially partition of input mesh
• Mesh fragments at different resolutions
• Associated to implicit diamonds
• Objective
– Mobile
• Limited resources (512MB-3GB ram CPU/GPU)
• Limited performance (CPU/GPU)
– Compact GPU representation
• Good compression ratio (maximize resource usage)
• Low decoding complexity (maximize decoding/rendering performance)

Our approach
• Our contribution
– Geometry clipped against containing tetrahedra
– Barycentric coordinates used for local tetrahedra geometry
reparameterization
– Fully adaptive and seamless 3D mesh structure with local quantization
– GPU friendly compact data representation
• 64bit = position (3 bytes) + color (3 bytes)+ normal(2 bytes)
• Normals encoded using the octahedron approach by [Meyer et al. 2012]
– Further compression for web distribution exploiting local data coherence for
entropy coding
Pbarycentric = λ1*P1+λ2*P2+λ3*P3+λ4*P4
40
P2
P1
P3
P4

Overview
• Construction
– Start with hires triangle soup
– Partition model
– Construct non-leaf cells by bottom-up
recombination and simplification of
lower level cells
– Assign model space errors to cells
• Rendering
– Refine graph, render selected
precomputed cells
– Project errors to screen
– Dual queue
Adaptive
rendering
On-line
41
GPU
Cache
Ensure continuity  Shared information on borders

Pre-processing (Barycentric parametrization)
• Tetrahedron geometry is reparameterized into
barycentric coordinates
– Triangles clipped on tetrahedron faces
– Inner vertices (I)
• 4 tetrahedron corners
– Vertices lying on tetrahedron faces
• 3 corners of the triangle defining the face
• Ensure continuity between neighbors
– Snap vertices on boundaries (<threshold)
• Simplification (per diamond)
– Inner versus all
– Inner boundary versus same class
– Outer boundary is fixed
1) Snap to corner 2) Snap to edge 3) Snap to face
42

Rendering process
• Extract view dependent diamond cut (CPU)
• Required patches requested to server
– Asynchronous multithread client
– Apache 2 based server (data repository, no
processing)
• For each node (GPU Vertex Shader):
– VBO containing barycentric coordinates , normals
and colors (64 bits per vertex)
– Decode position : P = MV * [C0 C1 C2 C3] * [Vb]
• Vb is the vector with the 4 barycentric coords
• C0..C3 are tetrahedra corners
– Decode normal from 2 bytes encoding [Meyers et
al. 2010]
– Use color coded in RGB24
48

Results Preprocessing (on 8 cores)
– 24k triangles/s
– 40 to 50 bits/vertex
Rendering (on iPad, tolerance
of 3 pixels)
– Average 30 Mtris/s
– Average 37 fps
– 15 fps for refined views
– (>2M triangles budget)
Rendering (on iPhone,
tolerance of 3 pixels)
– Average 2.8 Mtris/s
– Average 10 fps
– 2.8 fps for refined views (>1M
triangles budget)
Streaming Full screen view
– 30s on wireless,
– 45s on 3G
– David 14.5MB (1.1 Mtri)
– St. Matthew 19.9MB (1.8 Mtri)
49

Conclusions: Compact ATP
• Effective distribution and rendering of gigantic 3D
triangle meshes on common handheld devices
– Compact, GPU friendly, adaptive data structure
• Exploiting the properties of conformal hierarchies of tetrahedra
• Seamless local quantization using barycentric coordinates
– Two-stage CPU and GPU compression
• Integrated into a multiresolution data representation
• Limitations
– Requires coding non-trivial data structures, hard to implement on scripting
environments
50

Applications
– Large meshes
51

Explore Maps
Ubiquitous exploration of scenes with complex illumination
• Real-time limitations ~30Hz
• Difficulties handling complex illumination on mobile/web platforms
with current methods
• Image-based techniques
• Constraining camera movement to a set of fixed camera positions
• Enable pre-computed photorealistic visualization
52

Scene Discovery
• Goal
• Set of probes that provides full coverage of the scene
• Probe = 360° panoramic point of view
• Set of arcs connecting probes that enable full scene
navigation
55
Explore Map

Scene Discovery
• Objective: Full scene coverage !!!
• 1. Add new probe in unseen area
• 2. Optimize view  move towards seen area barycenter
• Repeat until convergence (no improvement | no movement) < threshold
• 3. Goto 1 while coverage < threshold
56
Probe v0 Probe v1 Joint coverage

Probe optimization
• Clustering  Cluster representative [Markov Cluster
Algorithm, MCL]
• Find natural clusters in graph – random walk
• Visit probability encoded in arcs = %coverage overlap between nodes
• Equivalent coverage, but better browsing experience
• Probe synthesis
• Optimization using a simulated annealing approach of:
• coverage & perceptual metrics[Secord et al. 2011]
57

Connecting Probes
• Search for a common visible region and choose the
closest point in this region
• A mass-spring system is used for optimizing and
smoothing this path

Dataset Creation (rendering)
• Input: Explore Map
• Probes with full scene coverage
• Transitions between “reachable” probes
• Pre-processing
• Photorealistic rendering (using Blender 2.68a)
• panoramic views both for probes and transition arcs
• We used 32 8-core PCs, for rendering times ranging from 40
minutes to 7 hours/model
• 1024^2 probe panoramas
• 256^2 transition video panoramas
59

Explore Maps - Results
60

Interactive Exploration
• UI for Explore Maps
• WebGL implementation + JPEG + MP4
• Panoramic images: probes + transition path
• Closest probe selection
• Path alignment with current view
• Thumbnail goto
• Non-fixed orientation
61

Conclusion: Interactive Exploration
• Interactive exploration of complex scenes
– Web/mobile enabled
– Pre-computed rendering
• state-of-the-art Global Illumination
– Graph-based navigation  guided exploration
• Limitations
– Constrained navigation
• Fixed set of camera positions
– Limited interaction
• Exploit panoramic views on paths  less constrained navigation
62

Applications
– Large meshes
69

Practical Volume Rendering in Mobile Devices
70

Our Goal
71

Video
78
• Low resolution
when interacting
• Refine when still
Samsung Galaxy S (512MB)

Conclusions
• 2012 – Volume rendering on mobile devices
– Possible but limited
• adaptive rendering (half resolution when interacting)
– 512MB RAM + 3x slice sets ~ 50-100MB was kind of limit for GPU
resources
– 3D textures as GLES extension – only on Qualcomm GPUs
• Nowadays –
– 1-4 GB of RAM
– 3D textures in core GLES 3.0 – 1024^3 typically supported
– GPU’s evolved  -- complex shaders ??
– ~2x-3x performance (~4fps… )
79

Summary
• Interactive exploration of large mesh models
– Fixed coarse subdivision -- Adaptive QuadPatches
• Fixed number of patches, multiresolution inside patches
– Simple, but limited by topology issues
– Adaptive coarse subdivision -- Compact Adaptive TetraPuzzles
• Multiresolution by combining a variable number of fixed-size patches
– More general, but more complex
• Interactive exploration of complex scenes
– Guided navigation + Global Illumination, but limited freedom
• Practical volume rendering on mobile devices
– Almost there ? – too much fragment load  mobile GPU 
90

SERVER
MOBILE DEVICE
Networked capture/rendering/interaction
• In the future...
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
CAPTURE
Environment
NETWORK
91

SERVER
MOBILE DEVICE
Networked capture/rendering/interaction
• In the future...
DATA
ACCESS
RENDERING
DISPLAY
INTERACTION
Scene
CAPTURE
Environment
NETWORK
92

Opportunities
• Killer applications will be
integration of
– Augmented reality / Mixed reality
technology
– Big Data operations
– Real-time constraints
– Parallelization on a massive scale
93

Challenges
• Input/Output:
– Augmented Reality headsets (Google glasses, pico projectors?)
– Environment imaging (acquisition and reconstruction)
• Computational:
– API improvements (HW progress faster than SW)
• Vulkan? Metal?
– Cloud-device integration
• Ultimate challenge:
– The “network GPU”
– Extend the GPU to network scale
– 109 GPUs for 1021 FLOPs ?
• Exciting era…
94

Questions and contacts
• CRS4 – Visual Computing Group
http://vic.crs4.it
• Speakers:
Marco Agus magus@crs4.it
Marcos Balsa mbalsa@crs4.it
• The DIVA project
http://diva-itn.ifi.uzh.ch
This work is partially supported by the EU FP7 Program under the DIVA project (REA Agreement 290277).
95

Questions and contacts
• CRS4 – Visual Computing Group
http://vic.crs4.it
• Speakers:
Marco Agus magus@crs4.it
Marcos Balsa mbalsa@crs4.it
• The DIVA project
http://diva-itn.ifi.uzh.ch
This work is partially supported by the EU FP7 Program under the DIVA project (REA Agreement 290277).
96

Mobile Graphics (part2)

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Mobile Graphics (part2)

Similar to Mobile Graphics (part2) (20)

More from CRS4 Research Center in Sardinia

More from CRS4 Research Center in Sardinia (20)

Recently uploaded

Recently uploaded (20)

Mobile Graphics (part2)