2. Physically Based Sky, Atmosphere and Cloud Rendering in Frostbite
SEBASTIEN HILLAIRE - ELECTRONIC ARTS / FROSTBITE
3. 3SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
An EA wide collaboration
within Frostbite
Sky and atmosphere systems
Master’s thesis: Gustav Bodare & Edvard Sandberg (Ghost)
Bioware (Mass Effect Andromeda), DICE (Mirror’s Edge Catalyst),
Ghost (Need for Speed)
Volumetric cloud systems
Master’s thesis: Rurik Högfeldt
Bioware: Marc-Andre Loyer (Programmer), Don Arceta (Lead Environment Art) and
Soren Hesse (Tech Environment Art)
The rendering research community
4. 4SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Context
Physically based rendering in Frostbite
Transition done in 2014 [Lagarde&deRousiers14]
Huge increase in visual quality
5. 5SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Context
Previous techniques in Frostbite
Sky
HDRI captures, color gradients
Atmosphere
Depth, height fog, screen space light shafts
Clouds
Panoramic texture + Visual flow [Guerrette14]
Translated planar cloud layers
Cons
Lots of manual work to get components to match
Not dynamic, difficult to achieve time of day
6. 6SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Real-life sky, atmosphere and clouds
With dynamic time of day and evolving weather
Photo by DAVID ILIFF. License: CC-BY-SA 3.0
7. 7SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Scope and motivation
Dynamic time of day and evolving weather at 60 FPS
Physically based rendering
Meaningful material parameters
Decouple material from lighting
Unified interactions
Sky, atmosphere and cloud interact together
Interaction with opaque and transparent objects, global illumination, etc.
8. 8SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Mirror’s Edge Catalyst - DICE
[Christin16]
10. 10SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Outline
Sky and atmosphere rendering
Sun rendering
Volumetric cloud rendering
Photo by DAVID ILIFF. License: CC-BY-SA 3.0
11. 11SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Outline
Sky and atmosphere rendering
Photo by DAVID ILIFF. License: CC-BY-SA 3.0
12. 12SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sky and atmosphere
Atmosphere
Rayleigh scattering
Mie scattering
Components
13. 13SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Previous work
Analytical models [CIESky][Hosek12][Preetham99]
Simpler but limited to ground view
Spectral rendering [Elek09][Preetham99]
Sky atmosphere simulation [Bruneton08]
Pre integrated multi-scattering/absorption in 4D LUTs
Scattering occlusion from terrain
Ignore earth shadow 3D LUTs [Elek09]
Improve LUT parameterization [Yusov13]
[Bruneton08]
[Hosek12]
14. 14SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sky – algorithm overview
Transmittance LUT
(h, θv)
Single scattering LUT
(h, θv, θl)
L
θl
V
θv
H
h
Final multi ScatteringLUT
(Rayleigh, Mie)
Accumulate
scattering
order
Scattered light
integration
N-order scattering
[Bruneton08]
[Bruneton08] [Elek09] [Yusov13]
15. 15SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Atmosphere simulation coefficients
[Nishita93][Riley04][Bruneton08][Adams74][Kutz12]
Type Scattering (𝒎−𝟏
) Extinction (𝒎−𝟏
) distribution
Rayleigh
(standard air) σ 𝒔
𝒓𝒂𝒚
= 𝟓. 𝟖𝑒−6, 𝟏. 𝟑𝟓𝑒−5, 𝟑. 𝟑𝟏𝑒−5 σ 𝒔
𝒓𝒂𝒚
𝒆
−𝒉
𝟖.𝟎 𝒌𝒎
Mie
(pollution, water,
dust, etc.)
σ 𝒔
𝒎𝒊𝒆 >= 3𝑒−6 a × σ 𝒔
𝒎𝒊𝒆
𝒆
−𝒉
𝟏.𝟐 𝒌𝒎
Ozone
--- σ 𝒂
𝒐𝒛𝒐
𝒆
−𝒉
𝟖.𝟎 𝒌𝒎
a = 1.11
σ 𝒂
𝒐𝒛𝒐 = 𝟑. 𝟒𝟐𝟔, 𝟖. 𝟐𝟗𝟖, 𝟎. 𝟑𝟓𝟔 𝒎−𝟏 × 𝟔𝒆−𝟓% [Kutz12]
[Bruneton08]
18. 18SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Ozone absorption
“Essential […] to reproduce the blue of the zenith sky” [Adams74][Kutz12]
Ozone absorption: “Essential […] to reproduce the blue of the zenith sky”[Adams74]
19. 19SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sky and aerial perspective rendering
Render the sky using ScatteringLUTs(h,θv,θl)
Camera aerial perspective volumes
32x32 x 16 depth slices
Integrate scattering/transmittance for each slices
Easy to apply on transparents (per vertex evaluation)
Scattered
luminance
Transmittance
20. 20SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Time of day - Moving sun
LUTs + moving sun automatically
updates:
Sky
Aerial perspective
Triggers updates of
Sky environment map
Local reflection volumes
Global illumination
Light probes
Sky environment lighting debug
21. 21SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sky rendering performance
LUT updates
Needed when changing atmospheric properties
Distribute evaluation over several frames
Each scattering order, sub table parts
While updating, interpolate last results
Complete update
frame count
Mean frame
GPU cost
1 3.50 ms
19 0.22 ms
Component GPU cost
AP camera volume 32x32x16 0.05 ms
Sky/AP rendering 0.42 ms
LUT updates with 4th order scattering
measured on XB1
Sky/Atmosphere rendering cost on XB1 720p
22. 22SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Outline
Sun rendering
Photo by DAVID ILIFF. License: CC-BY-SA 3.0
23. 23SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sun disk luminance and ground illuminance
In Frostbite, artists specify illuminance E for sun at zenith for sun solid angle 𝜴 [Lagarde&deRousiers14]
E = 100000 Lux
(default sun)
L outer space luminance
L outer space luminance
E
current
sun position
24. 24SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Atmosphere transmittance on sun
luminance = apply transmittance LUT per pixel to outer space luminance
25. 25SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Outline
Volumetric cloud rendering
Photo by DAVID ILIFF. License: CC-BY-SA 3.0
26. 26SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Previous work
Mesh + HyperTexture [Bouthors08]
High quality but overall expensive
Splatting based rendering [Yusov14]
Particle based, depth aware blending
Specific look did not match our target
Cloud layer [Schneider15]
Completely dynamic solution
Lighting, shadowing, weather
[Yusov14]
[Bouthors08]
[Schneider15]
27. 27SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Volumetric cloud rendering
Similar to [Schneider15]
Cloud layer with weather animation
Dynamic lighting and shadowing
Perlin-Worley noise for realistic cloud details
Beer-Powder
Improvements
Scattering energy conservation
2-lobe phase functions
Time of day and weather interaction with sky and aerial perspective
[Schneider15]
28. 28SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Volumetric cloud lighting components
Transmittance
Ambient
Sun scattering
Cloud
self shadow
Phase
function
Reality
29. 29SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Scattering integration improvement
Ray marching: improved integration from [Hillaire15]
float3 scattering = float3(0.0, 0.0, 0.0);
float transmittance = 1.0;
for (s= 0; s< StepCount; ++s)
{
𝑆 = sample scattered luminance
sampleTransmittance = 𝑒−𝝈𝒕
𝐷 = sample transmittance
scattering += 𝑆 * transmittance
transmittance *= sampleTransmittance
}
Wrong?
Analytically integrate lighting w.r.t. transmittance over depth D (ShaderToy demo)
0
𝐷
𝑒−𝝈𝒕
𝑥
× 𝑆 𝑑𝑥 =
𝑆−𝑆×𝑒−𝝈𝒕
𝐷
𝝈𝒕
𝝈𝒕 : extinction over range
𝑆 : scattered luminance
𝐷 : integration distance
30. 30SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
scattering += 𝑆 * transmittance
transmittance *= sampleTransmittance
Non energy conservative
31. 31SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
transmittance *= sampleTransmittance
scattering += 𝑆 * transmittance
Over attenuation dark media
32. 32SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Analytic integration
𝑆−𝑆×𝑒−𝝈𝒕
𝐷
𝝈𝒕
Energy conservative
33. 33SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Default scattering integration, 21 samples
Default scattering integration, 512 samples
Improved scattering integration, 21 samples
Improved scattering integration, 512 samples
34. 34SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Phase function
Two-lobes HG phase function
𝒑𝒉𝒂𝒔𝒆 𝑔0, 𝑔1, α, 𝜇 = 𝑙𝑒𝑟𝑝(𝒉𝒈𝑷𝒉𝒂𝒔𝒆 𝑔0, 𝜇 , 𝒉𝒈𝑷𝒉𝒂𝒔𝒆 𝑔1, 𝜇 , α)
Tried best fit
Back-scattering still not very visible
Would require multi-scattering…
Two-lobes HG phase + artist control
More freedom to control cloud style
Default: 𝑔0 = 0.8, 𝑔1 = −0.5, α = 0.5
[Bouthors08]
Cloud
Henyey/Greenstein
Rayleigh
0.0 0.5 1.0 1.5 2.0 2.5 3.0
10 4
0.01
1
100 Two-lobe HG
fitted to cloud phase
Angle
Phase
35. 35SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Phase function0.5 1.0 1.5 2.0 2.5 3.0 3.5
0.3
0.2
0.1
0.1
0.2
0.3
HG phase with g=0.8
36. 36SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Phase function0.5 0.5 1.0 1.5 2.0
0.3
0.2
0.1
0.1
0.2
0.3
2-lobe HG phase
with g0=0.8, g1=-0,5 and lerp=0.5
37. 37SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Phase function0.5 1.0 1.5 2.0 2.5 3.0 3.5
0.3
0.2
0.1
0.1
0.2
0.3
HG phase with g=0.8
38. 38SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Phase function0.5 0.5 1.0 1.5 2.0
0.3
0.2
0.1
0.1
0.2
0.3
2-lobe HG phase
with g0=0.8, g1=-0,5 and lerp=0.5
39. 39SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Aerial perspective and clouds
Must look consistent under all time of day and weather conditions
Visual mismatch (missing fog on clouds)
Clouds without aerial perspective
40. 40SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Aerial perspective and clouds
Per sample: expensive to evaluate
Instead sampled once on cloud front interface
Compute mean depth on cloud interface weighted by transmittance
41. 41SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Aerial perspective on cloud
Clouds with aerial perspective
42. 42SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Cloud coverage and scattering influence
Cloud hemisphere sampling around
camera
• Integrate cloud luminance
• Mean cloud transmittance
New AP scattered luminance =
oldAP x transmittance + luminance
43. 43SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Cloud coverage and scattering influence
RealityClouds with aerial perspective interactions
44. 44SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Volumetric clouds performance
720p 900p
Hemisphere sampling: scattering & coverage 0.090 ms 0.090 ms
Main view (with temporal re-projection) 0.720 ms 1.090 ms
Planar reflection (optional) 0.035 ms 0.048 ms
Total 0.835 ms 1.228 ms
Measured on XB1
Cloud main view resolution = 640x360 (720p/2)
Planar reflection resolution scale = 128x72 (720p/10)
Default quality seen in this presentation
Looking toward horizon
45. 45SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Final result
Sun rise with animated clouds
46. 46SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Final result
Mars blue sunset (eye-balled atmosphere)
[NASA]
47. 47SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Conclusion
Physically based sky, atmosphere and cloud rendering solution
Participating media material definition
Physically based and energy conservative scattering formulation
Supports dynamic time of day and evolving weather
Simulation of interactions between sky, sun, aerial perspective and clouds
Default: earth/sun/moon physical values
Simulation parameters can all be changed to simulate extra-terrestrial worlds
Practical use in Mirror’s Edge Catalyst: [Christin16]
48. 48SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Used widely in production by EA Games
Need for Speed, Ghost
Mass Effect Andromeda, BiowareFIFA 2017
49. 49SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Future work
Aerial perspective scattering shadowing by opaque and translucent clouds
Can’t use epipolar sampling min/max optimisation due to translucent clouds
Directional sky ambient by using cloud local gradient as main direction?
Convolution of sky envmap with phase function for cloud ambient lighting
Dynamic weather texture for cloud animation
GI and reflection volumes update
Still a problem when it comes to delay and performance
A complex problem with different priorities for each games
50. 50SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
References
[Adams74] The influence of Ozone and Aerosol on the Brightness and Color of the twilight Sky, JAS 1974.
[Bouthors08] Interactive Multiple Anisotropic Scattering in Clouds, I3D08.
[Bruneton08] Precomputed Atmospheric scattering, EGSR 2008.
[Christin16] Lighting the City of Glass, GDC 2016.
[CIESky] Spatial distribution of Daylight
[Elek09] Rendering Parametrizable Planetary Atmospheres with Multiple Scattering in Real-time, CESCG 2009.
[Guerrette14] Moving the heavens, GDC 2014.
[Harris02] Real-Time Cloud Rendering for Games, GDC 2002.
[Hestroffer98] Wavelength dependency of the Solar limb darkening, Astronomy and Physics 1998.
[Hillaire15] Physically Based and Unifier Volumetric Rendering in Frostbite, SIGGRAPH 2015.
[Hosek12] An Analytic Model for Full Spectral Sky-Dome Radiance, SIGGRAPH 2012
[Kutz12] Blog post, 2012.
[Lagarde&deRousiers14] Moving Frostbite to PBR, SIGGRAPH 2014.
[NASA] http://www.nasa.gov/multimedia/imagegallery/image_feature_347.html
[Nishita93] Display of the Earth Taking into Account Atmospheric Scattering, SIGGRAPH 1993.
[PBR] Physically Based Rendering book, www.pbrt.org.
[Preetham99] A Practical Analytic Model for Daylight, SIGGRAPH 99.
[Riley04] Efficient Rendering of Atmospheric Phenomena, EGSR 2004.
[Schneider15] The Real-time Volumetric Cloudscapes of Horizon: Zero Dawn, SIGGRAPH 2015.
[Yusov13] Outdoor Light Scattering, GDC 2013.
[Yusov14] High-Performance Rendering of realistic Cumulus Clouds using Pre-Computed Lighting, HPG14.
[Wenzel07] Real time atmospheric effects in game revisited, GDC 2007.
[Wrenninge10] Oz: The Great and Volumetric, SIGGRAPH 2010.
51. 51SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Battlefield, Mass Effect, Need for Speed, FIFA,
Plant vs Zombies, and many more.
We are hiring!
52. 52SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Questions?
Thanks!
The Frostbite rendering Team and game teams
Stephen Hill and Stephen McAuley for the course organisation
All the reviewers
Eric Bruneton and Fabrice Neyret
For further discussions, get in touch
sebastien.hillaire@frostbite.com
https://twitter.com/SebHillaire
Additional notes and Mathematica notebook to follow soon…
54. 54SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sky – how many scattering orders?
Single Scattering 2nd scattering order 4th scattering order
55. 55SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Scattering texture packing
ScatteringLUTs(h, θv, θl) is not storing (RayScat 𝒓𝒈𝒃, MieScat 𝒓𝒈𝒃)
But (RayScat 𝒓𝒈𝒃, MieScat)
MieScat 𝒓𝒈𝒃 = RayScat 𝒓𝒈𝒃 ×
MieScat
RayScat 𝒓
×
σ 𝒔
𝒓𝒂𝒚
𝒓
σ 𝒔
𝒓𝒂𝒚
(RayScat 𝒓𝒈𝒃, MieScat 𝒓𝒈𝒃) (RayScat 𝒓𝒈𝒃, MieScat)
56. 56SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Height fog
Uses scattered luminance at horizon
Seamless sky/fog transition
Per pixel coverage
Per pixel phase function
Approximations
Assumes low altitude fog
Ignores opaque shadows and self shadowing
Horizon
P
ScatteredLightP ≈ ScatteredLightHorizon
57. 57SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sun disk limb darkening
Attenuate luminance near edge of sun disk (Default = sun model [Hestroffer98])
(Very low exposure)
Nolimbdarkening
Sunmodel
Stronger
58. 58SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Night time
Moon and stars rendered as sprite using celestial system
Moon
Angular radius ≈ 0,24 to 0,28 degree
Luminance ≈ 2500 cd/m2
Secondary sky/atmosphere scattering source
Stars
Luminance? Angular radius?
59. 59SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Night time
Sun set
60. 60SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Ambient contribution
Ambient component
Capture the sky environment without sun disk
Integrate luminance to scatter
Approximations
Phase function as uniform
Single environment color ignoring cloud self occlusion
61. 61SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Multi-scattering approximation
Use [Wrenninge10]: multi-octaves single scattering / extinction / phase
Without
multi-scattering
62. 62SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Multi-scattering approximation
Use [Wrenninge10]: multi-octaves single scattering / extinction / phase
With
multi-scattering
63. 63SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Cloud coverage and scattering influence
Thick cloud
+ aerial perspective
+ hemisphere coverage/scattering
Thick cloud only
Clouds do not fit
Thick cloud
+ aerial perspective
Aerial perspective too bright
64. 64SIGGRAPH 2016 Course - Physically Based Shading in Theory and Practice
Sunny 16 validation Sun illuminance at ground level: 100000
Camera: ISO 100, aperture f/16 and shutter speed 1/125
Editor's Notes
Go full screen please: slides have lots of animations
Do not hesitate to contact me:
sebastien.hillaire@frostbite.com
https://twitter.com/SebHillaire
Hi and welcome to this course.
I am Sebastien Hillaire from Frostbite and I will present to you our work on physically based sky, atmosphere and cloud rendering.
But before we start, I would like to mention that this work has been conducted with several engineers, artists and game teams. So thank you very much!
I also would like to thanks the rendering research community for the useful results and also the source code which really saves our implementation/investigation times. If you want your tech to have more chance of being used, you know what to do.
So first a bit of context.
Two years ago, we moved Frostbite to be a physically based rendering engine. This resulted in a huge raise in quality as you can see in this NFS picture. It is hard to tell which one is from Frostbite and which one is real.
Despite all that, environment lighting was still achieved using mostly static procedural composition:
Sky could only be an HDRI capture or a procedural composition of panoramic color gradients.
Atmosphere was achieved using depth and height fog with screen space light shafts as an option.
Clouds could also be achieved with high resolution planar or panoramic textures with motion flow as an option.
All these techniques are great. But, for instance, it is hard to achieve dynamic time of day. Lots of work needs to be put into them to make sure all the different static elements match and interpolate well.
These technologies are all good but both we and game teams wanted to be able to achieve dynamic time of day and evolving weather.
This picture shows what we wanted to be able to achieve: a realistic physically based sky simulation that would support dynamic time of day, dynamic lighting according to sun position on our meshes, and clouds with matching atmospheric lighting.
The picture is from https://en.wikipedia.org/wiki/Aerial_perspective.
So we wanted all that, but of course this should be working for 60 frame per second titles and (as always) run as fast a possible.
We wanted it to be physically based: this means that participating media materials are decoupled from the light sources and parameters are meaningful and represent real physical properties.
Also, we wanted all these visual features to interact together coherently, and be as realistic and unified as possible by default. For instance, the cloud becoming larger and thicker should affect the light scattered through the atmosphere and in turn also affect global illumination.
This video shows the result that can be achieved with the physically based sky as used in Mirror’s Edge Catalyst. You can see the sun sets and everything is affected by the change of in lighting: sky, sun light, global illumination, lights shaft, particles, etc.
This video shows the early results that can be achieved using the physically based sky and dynamic clouds within an unannounced Bioware title.
This is the outline of my presentation.
I will first describe how we render physically based sky and atmosphere.
I will then quickly describe how we render our sun disk
Finally I will present how we render volumetric clouds and how they interact with other components.
3:00
To be able to render sky and atmosphere we need to take into account several components.
We must first consider the atmosphere as a slab around the earth with a exponential distribution of air molecules in it.
Light interacting with air particles that are much smaller than the light’s wavelength results in the highly wavelength dependent Rayleigh scattering. For the earth’s atmosphere, wavelengths of blue light are scattered more and that is why the sky appears blue during the day. However when the sun is at the horizon, light will have to travel a longer distance in the atmosphere and most blue light will be scattered away. Blue light will not travel as far as the green and red light in the atmosphere. That is why sunset and sunrise appear reddish.
Another important component of the atmosphere is the large particles concentrated near the ground. The concentration of these particles depends a lot on weather conditions, or pollution for instance. These particles cause the mostly wavelength independent Mie scattering. So the phase function describing how light will scatter is usually not uniform but biased toward the direction of the light. This phenomenon will cause the bright halo we usually see around the sun.
Several sky rendering models exist.
A few analytical models have been proposed. These models are however limited to ground view or atmosphere parameters that can’t be changed freely to simulate extra terrestrial planets.
The spectral rendering of the sky has also been implemented for an improved accuracy but we cannot afford that for games on today’s platforms.
Bruneton proposed a model to render sky and atmosphere by pre-integrating transmittance and scattering in look up tables. Using these tables, it is also possible to cheaply approximate multi-scattering.
Some nice extensions to that work have been proposed: Elek suggests ignoring earth shadow and thus reducing the LUT dimensionality to 3 instead of 4. Yusov proposed an improved parameterisation that increases the scattering accuracy.
In these models, the participating media material parameters are nicely decoupled from lighting.
In our case, we have chosen the following options from the framework proposed by Bruneton with the highlighted improvements:
RGB rendering, not spectral
3D LUT simplification from Elek
Improved parameterization from Yusov
I will only quickly expose how the LUTs are computed. Please refer to [Bruneton08] and [Elek09] for implementation details.
If you consider a perfectly spherical earth and atmosphere as well as a viewer position. The first parameter is the viewer height in the atmosphere. The second parameter is based on the sun / zenith angle. The third and last parameter is the view direction / zenith angle.
From these inputs, transmittance and single scattering tables can be built. Then as proposed by [Bruneton08], multi-scattering can be integrated by iterating over all these inputs to create a final multi=scattering table.
What coefficients should we use? We have gathered them all here from these reference.
The scattering coefficient depends on temperature and pressure but these are often ignored and given by their mean value.
You can see here the wavelength dependent Rayleigh scattering as well as the wavelength independent Mie scattering coefficients describing how light bounces off particles in the atmosphere. We also need extinction describing how much light will pass through after it travels through a certain path according to absorption and scattering. You can also see the distribution we use.
You can also notice that we use ozone which is not used traditionally. It helps in maintaining the blueness of the sky at sunset for instance. The coefficients have been recovered from measured data.
After discussing with Bruneton, we have learned that these coefficients do not represent scattering gathered from all wavelengths. Instead the scattering coefficients for R, G and B were only taken for the corresponding wavelengths 680, 550 and 440nm.
Light Rayleigh scattering
Default sky Rayleigh scattering
Heavy Rayleigh scattering
No Mie scattering
Default sky Mie scattering
Very dusty atmosphere
Ozone is particularly important to keep the zenith sky blue during sunset and sun rise. You can see the difference in these pictures.
Ozone should be concentrated 32km up in the sky. But this was giving us unexpected results (probably due to using RGB instead of a more complete spectrum). Instead, in our simulation, ozone follows the same atmosphere distribution as the particle distribution for Rayleigh scattering. For more details please refer to the course notes.
So how do we render the final frame?
First we render opaque objects.
Then we render the sky using the scattering look up table.
We also prepare an aerial perspective volume fitted on the camera frustum. Aerial perspective is the effect of the atmosphere on objects (== atmosphere fog). Two textures contain the scattered light as well as the transmittance. Please note that this does not support scattering occlusion so we will not get any volumetric light shafts. Getting these sort of light shafts to very a good level of quality would have added a cost which our 60fps titles were not willing to pay.
The aerial perspective is applied during the sky pass on opaque objects but also on all the transparent objects per vertex. This is because it is important for everything to fit into the scene. This is a low resolution texture but since we do not bake the phase function into it (applied per pixel later), we do not suffer from the low froxel resolution.
The advantage of this approach is that this texture can cheaply be sampled per vertex on transparent objects to make sure these type of entities also fit into the world.
Since we have the look up table ready, the sun can be freely moved around and as a result the sky and aerial perspective lighting will automatically be updated and match.
We do have to update the sky environment map, local reflection volumes as well as our dynamic GI solution, as you can see in this video.
We also take into account atmosphere transmittance on the sun illuminance and light shafts. I will tell you more about that later.
A good example if this was also visible in the introductory videos at the beginning of this presentation.
About performance:
Once the LUTs have been computed, the sun can be moved freely. But changing an atmosphere parameter will trigger a LUT update that can be costly.
To reduce this cost, we evaluate the integration over several frames. We even update only parts of LUTs during a frame. While we’re doing this, we can interpolate the latest result and catch up this way until done.
Doing this, the GPU cost of evaluating multi-scattering is greatly reduced to an affordable cost. You can see here the cost for different distributions and the final sky and aerial perspective rendering cost. The distributed rendering is very valuable in order to reduce the cost.
11:15
As mentioned previously, artists specify sun illuminance at ground level (easier for artists to predict the final look). From that, transmittance at zenith and sun solid angle, we can recover the sun outer space luminance.
Once we know the sun outer space luminance, we can recover the final illuminance at ground level that is used to light the scene for the current sun direction.
---
In course notes:
We suppose the sun is usually very small in the sky so we can approximate the integral over the solid angle by a simple product with the solid angle.
We apply the following process to recover sun luminance and illuminance:
Evaluate atmosphere transmittance at zenith
Use equation 1 to approximate outer space luminance according to this transmittance and sun solid angle
Evaluate transmittance for the sun centre direction
Use equation 2 to approximate sun illuminance at ground level.
Code:
solidAngle = FB_TWO_PI * (1 -cosf(angularRadius))
OuterSpaceSunLuminance = sunIlluminance / (sunSolidAngleInRad * zenithTransmittance)
Now that we have the sun outer space luminance we can use it to render the sun disk. But without taking into account the atmospheric transmittance, the sun disk would look to bright. To solve this we simply apply the atmospheric transmittance per pixel to get the final luminance to store in our light buffer. As you can see in this picture, it is mandatory to get the correct look for the sun.
12:45
Many cloud rendering techniques have been proposed.
Harris proposed using impostors composed of particles. This is very performant thanks to the angle based update of impostors but visual fidelity is not where we would expect nowadays.
Bouthors render clouds with a mix of meshes, hyper textures and ray marching. The final result is of very high visual quality. But overall it remains expensive and the data is not straightforward to author or edit.
Yusov proposed a splat based approach using pre integrated scattering on particles elements. However the final look is very specific and we did not want to follow that visual target.
Finally last year Schneider proposed a layer based ray-marched approach that is giving detailed results. It was also completely dynamic with spatial control and weather and applicable to real time games.
We have decided to base our approach on this technique.
We basically use the same approach to the one described by Schneider.
I am simply going to describe extra interesting details about lighting, scattering energy conservation as well as dynamic weather and time of day interactions.
The clouds are authored using regular physically based scattering, absorption and phase function material parameters.
The resulting visual components are the following:
- Transmittance represents the occlusion of what is behind the clouds
- Ambient lighting (for example, from the sky) will be scattered by the cloud.
- Light scattering from the sun also needs to be taken into account. However, by itself, it does not look good: one needs to evaluate the clouds self-shadowing according to extinction [PBR] [Hillaire15].
- Finally, as for the atmosphere, light can bounce non uniformly and usually, clouds have a very strong forward scattering resulting in the so-called silver lining.
Compared to reality, what is missing here is high resolution and multi-scattering simulation.
To be able to render clouds, you need to ray march them to evaluate transmittance and the scattered light toward the camera. We use the same approach I have described last year to improve the scattering integration.
When we ray march, we have to integrate scattering and also transmittance and both are done in two operations. It is not correct to do one or the other first. So I proposed integrating scattering according to the extinction coefficients over the current range D. There is a small ShaderToy demo you can use as reference (https://www.shadertoy.com/view/XlBSRz).
The issue is very visible on dense cloud, especially with ambient lighting.
This is what clouds look like when updating scattering before transmittance. You can see that it is definitely not energy conserving especially when it come to the scattering of the ambient term and the end result is too bright.
This is what clouds look like when updating transmittance before scattering. It is overall too dark and clouds look a bit like dirt. Instead, when very dense, they should converge to what an opaque diffuse surface would look like.
And now the more correct result when using the improved integration. Dense participating media can now look more consistent and be energy conserving.
It is also important to get correct results when using less samples.
At the top left is the default incorrect scattering integration with 21 samples.
If you use more samples, it will converge to the appropriate solution.
With the improved scattering integration, it looks correct even with a low number of samples.
The phase function is a very important component of the cloud lighting. It will give you all the complex lighting effects that clouds have such as the silver lining, pseudo-specularity and glory haloes. You can see a typical cloud phase function in blue here.
We wanted to approximate it with a 2-lobe Henyey-Grenstein phase function, but in this case the backward scattering was still visually very low (as you can see on this figure). We think that this is because we are missing a proper multi-scattering simulation.
In the end, we gave artists the possibility to tweak the 2-lobe phase functions, that are then combined and normalised for the purpose of energy conservation. This way artists can control the amount of forward and backward scattering they want. This is also coherent with real clouds since depending on the radius distribution of water particles, the shape of the phase function can change a lot.
This is how clouds look like with a single forward phase function.
This is how they look with a 2-lobe phase function. They look a bit less intense since some of the light will be sent backward and that is expected in order to be energy conservative.
This is how clouds look when the sun is behind the camera and only forward scattering is used. There is only the ambient contribution which makes them look flat.
When using a bit of backward scattering, you can now see that details are more visible and clouds look closer to what one would expect in reality.
Now that we can render and animate clouds, we need to make sure they look consistent under all conditions.
Especially when dynamic weather is used by making the cloud layer thicker.
You can see here where we started: without aerial perspective applied on clouds during heavy weather condition changes, it looks like they do not fit in the environment.
However, evaluating the AP for each ray marched samples would raise the rendering cost. Instead we propose to compute a weighted average of the cloud front interface evaluated as a function of the sampled density and depth. It is weighted by the transmittance to make sure the samples stay at the front of the cloud interface as we are raymarching deeper into it.
This final depth can be used to sample the AP once.
On this video you can now see the aerial perspective being applied on the clouds. Much better! This could be usable but something is still off.
When cloud coverage increases, the aerial perspective should be affected and no longer scatter the sun/sky color but only the light scattering through the clouds.
So we need to take into account the cloud coverage when evaluating the AP. But the clouds are also scattering light from the sun and the sky above. So we also need to take into account cloud scattered luminance as another light source lighting the aerial perspective.
To this aim we sample the hemisphere around the camera position on the ground and integrate scattered luminance and cloud coverage. Assuming a uniform phase function, we will occlude previous AP scattering resulting from the sun by the cloud coverage and add the cloud scattered luminance back.
The AP transmittance does not change however since this only depends on the particle density in the air which has not changed.
This is considered as a global effect and a very cheap way to get it. A more involved way to achieve this would be to evaluate cloud shadow and simulate volumetric occlusion in the atmosphere. That would be more consistent for very spatially varying cloud coverage.
And this is the result: you can notice the bright yellow Mie scattering getting progressively occluded and replaced with a dark blue color coming from the integrated cloud scattered luminance. It has a string blue tint due to the sky’s blue ambient color. You can see a real example in this picture.
You could also get a more white atmosphere depending on the density of your clouds and how the sun light penetrates through them.
Cloud pictures:
http://cloud-maven.com/wp-content/uploads/2013/09/DSC05456.jpg
http://www.wolken-online.de/images/wolken/nimbostratus/nimbostratus_pannus_2.jpg
https://upload.wikimedia.org/wikipedia/commons/4/4f/Ns1.jpg
This is the final performance we get on Xbox One for clouds rendered at half resolution in the main view while looking at the horizon with the quality seen in this presentation.
21:15
This is a video showing the final sky, atmosphere and clouds all integrated during a sun rise.
Just for fun, I tried to simulate a Martian sunset using our sky model. I have been able to reproduce the red day color and the blue sunset due to Mars’ atmosphere scattering the red wavelengths more.
See the following reference for Mars’ blue sunset/rise:
http://www.nasa.gov/multimedia/imagegallery/image_feature_347.html
http://www.cbc.ca/news/technology/blue-mars-sunset-captured-in-nasa-video-1.2970932
http://www.npr.org/sections/krulwich/2012/04/25/151376553/a-most-peculiar-sunset
Curiosity: http://www.nasa.gov/sites/default/files/thumbnails/image/pia19142_malhi-mojave.jpg
I have presented to you the current solution we have in Frostbite to render sky, atmospheric effect and clouds.
We follow a physically based approach by being energy conservative and using physically based participating media material definitions. We also simulate many interactions between all these components under dynamic time of day.
And by default we provide our artists with earth-like setup.
The physically based sky and atmosphere simulation has already used in production by several game teams.
The volumetric clouds technique is currently being used in production by unannounced titles.
There is lots of room to improve this technology. Some of the games benefiting from it were under a 60fps constraint. These are examples of improvement we would like to investigate to improve the visual fidelity for our 30fps titles.
Some references used in that presentation and this work.
We are hiring so do not hesitate to get in touch if you are interested
Thank you for your attention! Now it is time for questions. And if we do not have time do not hesitate to contact me online or get in touch after the course.
So how many scattering orders do you need?
It will depend on the performance and update reactivity you need (I’ll tell you more on that later).
You can see the different results here for single scattering, 2nd order and 4th order multi-scattering. This can be selected as a setting by each game. Frostbite games usually go with the default 4th order scattering.
Rayleigh and Mie scattering could be straightforwardly stored into two RGB result textures. However, this increases the memory bandwidth requirements when rendering the sky pass, integrating multiple scattering or applying the aerial perspective. Mie scattering is non wavelength dependent. But the scattered light will still be colored due to Rayleigh scattering and extinction, etc.
To solve this issue we pack Mie scattering as single value representing its R coordinate. And recover its color right at the end when evaluating the final scattered luminance using a bunch of ALUs.
This approximation works well when scattering coefficients are close to extinction (no absorption). but it is still a rough approximation for the following cases:
scattering << extinction
coefficients must not be == 0
We can also apply height fog.
To achieve this aim we sample the scattered luminance once at the horizon as our artists did not want seams at the horizon by default. We apply it as height fog scattered luminance taking into account the height fog coverage and phase function per pixel.
From this diagram, we are considering the scattering light for both paths to be the same - only the coverage and phase function vary (with the coverage influencing how much luminance is sent back towards the camera).
If the sun is simply rendered using transmittance, it will look like a perfectly solid disk when. One interesting visual feature of the sun is limb darkening. It makes sun edges appear less bright because the luminance coming from its centre has first to travel through a layer of media absorbing some of it. So we apply a wavelength dependent luminance reduction based on an astronomical model. We will provide the function in the course notes.
The default sun limb darkening is not very strong but artists can change the coefficients to make it stronger if they need. Note that we ignore the luminance change when we evaluate luminance/illuminance mentioned in the previous slide.
NOTE:
The picture looks different from the previous slide because the view angle, exposure and sun elevation are different.
Models:
http://www.physics.hmc.edu/faculty/esin/a101/limbdarkening.pdf
http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1994SoPh..153...91N&defaultprint=YES&filetype=.pdf
During the night, we can render the moon as a sprite using our celestial system. We use the following moon luminance. The moon is used as a secondary light source for the sky and atmosphere.
And of course we can sprinkle stars in the sky (but we have never found stars mean angular radius and luminance).
You can see here a sunset in progress progressively revealing the moon and then the stars. The moon and stars are rendered using our celestial system and the moon also acts as a very dim second scattering light source
Cloud ambient lighting is very important.
We compute cloud ambient lighting from the sky contribution only (without the sun disk). We render it in a small 64x64 buffer with uniform sphere sampling. This is used to compute cloud ambient lighting component using an average of all the incoming lighting direction(assuming a uniform phase function of 1/4PI ). This approximates ambient lighting using a uniform environment.
Under investigation: we can approximate cloud self occlusion using the Beer-Powder function if necessary.
We do have a multi-scattering approximation implemented as proposed by Wrenninge for film production. Several octaves of scattering are evaluated and each octave features a more diffuse phase function and less dense media.
This picture shows clouds without multi-scattering
This picture shows the result when the optional multi-scattering model is enabled. You can see that the light interaction is more spread out and can travel deeper in the cloud while still maintaining most of the sharp silver lining effect.
This is a summary of the three conditions.
No aerial perspective on the left.
With AP applied on cloud in the middle.
And finally with the AP being affected by cloud coverage and especially the cloud scattered color which is often overlooked.
You can see here a shot following the Sunny 16 rule with the sun at the zenith showing the result on a variety of opaque materials and translucent particles with volumetric shadowing.