GAUDIO LAB develops audio technologies, one of which was adopted as the MPEG-H 3D Audio international standard. Based on these, we provide end-to-end solution for whole VR ecosystem as producer intended. In this workshop, audience will be guided the basics of VR audio solution when creating, delivering, and playing back 360/VR contents. This covers: - Immersive and interactive binaural rendering: a key technology for real immersive VR audio - Object-based audio vs. scene-based audio - VR audio distribution: app-based or codec-based - Considerations on VR audio creation workflow - VR sound recording: mono, Ambisonics and binaural - Mixing and mastering with GAUDIO Works, AAX plugin for Pro Tools HD - Quality matters. But how to guarantee the same sound quality at the end-user stage? - GAUDIO Sol, a reference quality 360/VR audio player (API)

1. End-to-End VR Audio Solution for
Fully Immersive/Interactive Experience
Henney Oh, Ph. D.
2016 AES Int’l Conference on AVAR. Sep 30 - Oct 1, 2016. @ Los Angeles Convention Center

2. |
This workshop covers
2
• Backgrounds
• Immersive and interactive
• Binaural rendering
• Formats: object-based audio vs. scene-based audio
• End-to-end workflow for VR audio: codec-based and app-based
• Solutions
• Mixing and mastering with GAUDIO Works
• Sol Player, a reference quality VR/360 audio player (API)
• GAO, a VR audio format for Ambisonics + object + channel
• GAUDIO Craft, a VR/3D audio plug-in for gaming VR
• Hands-on and practices

3. |
GAUDIO LAB is
3
• Founded by guys from MPEG Audio standardization community
• Proudly, inventors of the binaural rendering technology for MPEG-H
3D Audio (ISO/IEC 23003-8)
• Strong audio engineering forces of 5 Ph.D. and 2 Masters
• Now, focus on providing end-to-end solution for whole VR audio
ecosystem

5. |
What is VR Audio ?
5
Technology or content that enables
“being there” experience in respect of
auditory perception
Immersive & Interactive

6. |
What is immersive ?
6
videolink: https://youtu.be/X1tsJtcXllg

7. |
What is interactive ?
7
• 3DOF (Yaw-Pitch-Roll)
‣ Head orientation only
‣ 360 Video
‣ Ambisonics
• 6DOF (3DOF + X-Y-Z)
‣ Head orientation + Walk around
‣ Full VR
‣ Object-based audio
DOF (Degree Of Freedom)
Yaw
Roll
Pitch
Yaw
Roll
Pitch
Y
Z X
Listener’s freedom of movements for interactivity in VR space

8. |
Loudspeaker or Headphone ?
8
• Hard to remove local room presence
(you are “here”)
• Virtual space limited by room size
• 1 scene / room
• No local room artifact
(block the “here” environment)
• Unlimited virtual space
• 1 scene / person
Vs.
Loudspeaker Headphone (earphone)

9. |
Headphone. then, “binaural”
9
= Having or relating to two ears
widely used to mean “headphone virtualization”

10. |
Binaural Rendering
10
HRTF (HRIR)
Listen at anechoic room
BRIR = HRTF + RIR
Listen at real room
Signal processing technology to make sound virtualise in 3D space with
headphone representation using HRTF or BRIR

11. |
Binaural Rendering
11
Engine
=
Binaural rendering
VR audio Automobile
Core technology to enable VR audio
Compared to the engine in automobile
Core: Core:

12. |
VR Audio Format
12
• Channel
• Ambisonics
• Object
Concern all the ecosystem from content creation, delivery and consumption

13. |
An exemplary VR sound scene
13
Guitar
Bell
Bird
Fountain
You
BGM
Reflections
Diffusion
Direct
Diegetic Sound
Non-diegetic Sound
Scene

14. |
Channel-based
14
• A traditional scene representation
• Pre-rendered in studio to target channel configuration
• Each channel directly maps to each loudspeaker
• More channel, more immersive

15. |
Channel to Binaural Rendering
15
Set loudspeaker layout as the source positions and binaural rendering
45 degree315 degree
135 degree225 degree

16. |
Ambisonics (Scene-based)
16
• Loudspeaker-independent sound-field
representation on spherical harmonics
• Captured by B-format microphone
• Easy for 3DOF,
NOT suitable for 6DOF
Spherical Harmonics
B-format FOA microphone
FOA (First Order Ambisonics)
HOA (Higher Order Ambisonics, 3rd)

17. |
Ambisonics
17
Non-diegetic stereo (BGM) CANNOT be implemented

18. |
Ambisonics to Binaural Rendering
18
Ambisonics to Channel Channel to BinauralRotation (3DOF)
Two-step rendering required after rotating sphere to the head orientation

19. |
Object-based
Object 1
Object 2
Object 3
Object 4
Object 5 (Stereo)
Send every sound component constructing sound-scene to playback-side
Metadata
To Playback

20. |
Object to Binaural Rendering
20
Object to Binaural
Straightforward and easy. Directly from object position to the ears
Complexity linearly increases as the number of objects increases

21. |
Comparative benefit of object-based
21
Directly Binaural Rendering
to Headphone
Rotation
(3DOF)
Sound
Sources
(Tracks)
…
Binaural
Rendering
Binaural
Rendering
Channel
Mapping
Channel mapped scene
PlayerProduction
ChannelRendering
to
Channel
AmbisonicsSound
Sources
(Tracks)
Sound
Sources
(Tracks)
Object
……
…
Rendering
to
Ambisonics

22. |
VR Audio Format Transformation
22
Channel
Object
Ambisonics
O
O
O
O

23. |
Mixed format representation
23
Channel
Ambisonics
Object
Possible and recommended to represent sound scene with the
combinations of the 3 formats

24. |
End-to-end workflow in VR audio
24
Sound acquisition Post-production Transmission Play/Rendering
Cinematic
VR
Gaming
VR
Codec-based distribution
App-based distribution
Player
+
Content + EngineAuthoring tool
(Game Engine)
Sound Source
Player + EngineContent
(format)
Authoring tool
(DAW)
Compatibility
Sound source

25. |
Codec-based vs. App-based
25
Codec-based App-based
Typically in Cinematic VR Gaming VR
What have to deliver Content only Content + Playback engine
Live streaming capability Yes No
Upgradability Difficult Easy
Content distribution
platform
VR Players App market per OS

26. Solution for VR Audio

27. |
GAUDIO Core - the binaural rendering engine
27
• High quality - low complexity
• Pinpoint spatial resolution
• All 3 formats (Object, Ambisonics, Channel)
• Advanced distance rendering
• Source directivity, occulusion, reverberations
• Diegetic & non-diegetic selection per each source
• HRTF personalisation ready
Key Features

28. |
GAUDIO Works
28
Cinematic VR
content
(360 video + VR audio)
Channel
Object
Ambisonics
VR/360 content creation tool for cinematic VR
Sound acquisition Post-production Transmission Play/Rendering
360 Video

29. |
GAUDIO Works
29
• Sound powered by GAUDIO Core
• Support all formats for In-Out
• Object, Ambisonics, Channel
• Intuitive user interfaces
• Realtime automation R/W on 360 Video
• 2 independent 3D map-view
• Automated listener’s head orientation
• Easy to export
• Direct export to 360 video (mp4, mov)
• AV lipsyncPro Tools HD
AAX Plug-in
Key Features

30. |
GAUDIO Works hands-on
30
videolink: https://youtu.be/gB0ZlAzAf94

31. |
GAUDIO Works: Release
31
• Beta release today (Oct. 1, 2016)
• Download at our website
http://www.gaudiolab.com/
• It’s free! (forever)

32. |
GAUDIO Sol Player / Sol API
32
Enjoy VR
Sound acquisition Post-production Transmission Play/Rendering
VR/360 audio player / player api
Cinematic VR
content
(360 video + VR audio)
L
R

33. |
GAUDIO Sol Player / Sol API
33
• Sound powered by GAUDIO Core
• Support any combination of object, Ambisonics, and/or channel
• Low complexity and low latency
• Support multiple platforms
C D E F G
do re mi fa Sol
Why ‘Sol’ ?
Sol Player = G-Player
Key Features
< Reference Sol Player >

34. |
GAUDIO Sol API
34
OS
Device
Android Mac Windows iOS (coming soon)
Cardboard Gear VR Oculus Rift HTC Vive
Now available in

35. |
GAUDIO Sol Player Hands-on (Android version)
35
videolink: https://youtu.be/JDr8uKv3hN0

36. |
Sol Reference Player: Release
36
• It’s also free! (forever)
• Available at
• Android (Google Play) -
already
• Gear VR (Oculus store) -
under certification
• Oculus Rift (in a month)
• HTC Vive (in a month)< Reference Sol Player >
• Stand-alone 360 video with Sol
• Monitoring purpose
• Demo for customer or end-user

37. |
GAO
37
• Dedicated and optimised format for VR audio content delivery
• Gaudio Ambisonics + Objects
• Interface between Works and Sol
• Deliver all VR audio formats including object, Ambisonics, and channel
• Independent from any compression codecs
Ambisonics Objects
FOA, HOA up to 64 objects
Sound acquisition Post-production Transmission Play/Rendering

38. |
GAUDIO Works, GAO and Sol
38
Other platforms
supporting FOA
FOA
Sound acquisition Post-production Transmission Play/Rendering

39. |
GAUDIO Craft
39
• Sound powered by GAUDIO Core
• Sound can be simulated and monitored with GAUDIO Works
• Easy to migrate from existing projects
• Intuitive interface for developer easily localise the sounds
• Support major platforms:
Sound acquisition Post-production Transmission Play/Rendering
VR/3D game engine plug-in
Unity3D Unreal
Key Features

40. |
Conclusions
40
• Covered technical backgrounds and solutions in end-to-end VR
audio ecosystem
• Introduced GAUDIO’s proposed solutions for the ecosystem: Works,
Sol, GAO, and Craft
• Please let us know your problems, then, we keep supplying
technologies aiming at ultimate VR audio experience
• Hands-on demo on the booth !

41. Sound for VR
www.gaudiolab.com | info@gaudiolab.com | 501 475 1974
7083 Hollywood Boulevard, Los Angeles, CA
You’re brilliant, We’re hiring !
(for LA office)
THANK YOU