Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spatial Audio Lessons Learned from Unscramble the Oracle BOSE AR

264 views

Published on

3 Lessons from our BOSE AR Audio-First game. While photorealistic graphics grab the lion share of attention, deeply immersive experiences must address all of the senses. This talk covers a post mortem of an audio-only Unscramble the Oracle (BOSE AR) game using fully spatialized interactive audio scapes that require no vision. Come learn about the challenges we faced creating this first of a kind interactive walkable audiobook. Learn how storytelling, navigation, character interaction change when the player can only can hear. Together we look at how we succeeded and stumbled in creating an audio world to explore with native head, body, and interactions How we leveraged characters, specialize sound, and audio UX. What audio attributes increased spatial positioning, what reduced it, and how create audio highlight states. Additionally, it offers lessons from designing to the other senses hearing and touch including several techniques to cross talk and fool the senses.

Pick up any smart phone and gestures to operate it are nearly universal. Put on a set of smart glasses or play a mobile AR game and they are hobbled by out dated UX/UI standards that interfere with the new opportunities on the emerging AR platforms for spacial computing. In contrast to the UX/UI guidelines for WIMP interfaces; precious little has been written about best practices for AR. With screenless information, spatial computing, floating menus, mid air gestures, restyled 3DOF and 6DOF controllers, new graphics, audio, and haptics AR offers novel interaction patterns that demand new interface language and standards. Come join us for a dive into the emerging UX / UI standards for AR and spatial computing with inspiring examples from Leap Motion, North, mixed reality, Magic Leap, as well as some from XEO's own laboratory and latest experiments. Takeaway
Attendees will walk away with concrete examples and techniques for spatial audio design for AR as well as traditional games. Position is the power of immersive audio. Attendees will learn how precise location separates one audio source from another and the motion of these audio sources unlocks new story telling possibilities. Additionally how to create flexible layouts for the Real World in AR. The lessons in immersive audio also offers lessons for highly dynamic flat screen game worlds that are more than a backdrop for gameplay. Additionally, attendees will learn how to enhance button and object interaction in an audio only world.
Intended Audience
Audio engineers, game designers, and developers of all game platforms will walk away with insights in how leverage psychological properties of sound and how to use use spacialized audio to tell stories, crate interactive audio objects and menus, and interact with a game world that they can hear but not see. These lessons are also suitable for developers of flat screen games wishing to create more immersive experiences

Published in: Design
  • Be the first to comment

Spatial Audio Lessons Learned from Unscramble the Oracle BOSE AR

  1. 1. Nicole Lazzaro Hear Me Feel Me Let Me Play: Lessons from Unscramble the Oracle a BOSE AR Audio-First AR Game GDC Summer SF/Online August 4, 2020 @NicoleLazzaro
  2. 2. HearMeFeelMeLetMePlay: Lessonsfrom UnscrambletheOracle aBOSEARAudio-FirstARGame
  3. 3. SpatialAudioDownsampledtoStereo Bestwith Headphones
  4. 4. StanfordPsychology 4KeystoFun o/ Fiero! TiltWorld thefirstiPhonegame TiltWorld.com TiltWorld Plant16Ktreesin Madagascar UnscrambletheOracle FollowtheWhiteRabbit @NicoleLazzaro FreeDownload 4K2F.com
  5. 5. ThankYou!UnscrambletheOracleTEAM! • DavidFox writing&Sfx ElectricEggplant.com • PatrickO’ShaughnesseyNarr.Engine PatchedReality.com • WonkubusWonkWalk,VO.&Art (MichaelCrabbs) MichaelCrabbs.com • Nicole Lazzaro GameDesign& UX XEODesign.com
  6. 6. AudioFirstChooseYourOwnAdventureYouPlayWalkingAroundtheBlock How We Designed an Audio First GameImage: Wonkubus
  7. 7. LessonsfromBeatles,SaturdayNightFever,AndStonehenge How We Designed an Audio First GameImage: Wonkubus
  8. 8. BLUETOOTHSPEAKERS&IMU BOSEAR SmartPhone
  9. 9. AudioFirstARHeadPoseRobust BOSEARGameJam PlayCrafting July26,2019 Image: Wonkubus
  10. 10. Designis Constraint
  11. 11. PushLimitsLearnAudioSpatializationForAllgames • AudioFirst • 1stperson • “PhoneinPocket” • FullyPlayable • Walk&LookAround • ExplorableAudioWorld • InteractiveAudioGameplay • Onlyplayableinspatialaudio • CoreLoopRequiredAudioSpatialization AudioGoals Game TechnologyDidWell 1. Headpose 2. Tapheadsettoselect 3. Positionaudio(ResonancePlugin) Challenge ✓Responsive ✓Accurate X 6DoF(only3DoF)
  12. 12. 3ChallengesforAudio-FirstGames Stand Walk ChooseSpatialAudio> Image: Wonkubus
  13. 13. Stand 1
  14. 14. MaximizeAudioPositionDataAroundPlayer • ImmersiveAliveAudioWorld • Sound OutsideHead • NoHeadlock • PositionalClarity AudioGoals Stand
  15. 15. WhyDoesDoingThisNotInverttheSound?
  16. 16. BeatlesAbbeyRoad1st toRecordinStereoandSynthesizerMore: https://www.pbs.org/newshour/arts/the-beatles-abbey-road-revolutionized-how-a-track-could-be-recorded-listen-closely-here Here Comes the Sun ©1969 EMI
  17. 17. WantSpatialAudiotoBeOutThereintheWorldAroundthePlayerMore: https://www.pbs.org/newshour/arts/the-beatles-abbey-road-revolutionized-how-a-track-could-be-recorded-listen-closely-here
  18. 18. SurroundSound≠SpatialAudio ØStereo ØAmbisonic ØBinaural ØASMR
  19. 19. HeadMotionWhileStanding-SpatialAudioField
  20. 20. UnscrambletheOracle
  21. 21. UnscrambletheOracle6DoF (DegreesofFreedomforMovement) “FindtheFrog” WalkAroundRotateLookUp&Down StereoInsidetheHead ZeroDegreesofFreedom
  22. 22. HeadPose=DegreesofFreedom(DoF) NoDoF Headphones NoMotion HeadLock 6DoF Walkforward/backward Sidewaysleft/right Jumpup/down 3DoF Lookup/down Left/right TiltLeft/Right SpatializationRequiresMotion
  23. 23. HeadTiltShowsCuriosity AlsoAlsoIncreasedSpatializedSamplingforElevation AmplifiesthedistortionBetweenTwoEars forallOtherdirectionofHeadMotion containingelevationinformation
  24. 24. Auxel AUXEL“AudioPixel” AudioSourceWithPositionalInformation
  25. 25. HowBigisanAuxel?
  26. 26. Rotation Elevation Distance AudioPosition-Auxel=RED Auxel=R5E3D3 “AudioPixel” 5 Howmanydiscreteaudiopositionscanplayershear? 3 3 ? 45Positions1-3atatime
  27. 27. WhileStandinginPlaceDistanceIsLessImportant 5 UnlessAuxelisMovingRelativetoPlayer Auxel=R5E3D3 “AudioPixel” Rotation Elevation Distance 3 ?
  28. 28. 3 AuxelsNeedSpacing PositionTimeMemory 5 3 TheSoundofSilence
  29. 29. EasiertoLocalizeAuxelswhenHearSpatialDistortion SharpAttack RepeatingPattern TIMETO: MoveHeadLATENCYHEADLOCK&OVERSHOOT
  30. 30. SpatializationTips • RespondtoHeadMovementEsp.Rotation • TIMETODISCOVER • Length orLongEnoughLoops • Identifiable&Separation • MonoSourcesinGameWorldNoReflections • SharpAttack • PitchAboveandBelow AudioGoals Stand
  31. 31. Walk 2
  32. 32. WalkThroughtheSoundscape
  33. 33. MaximizeSenseofForwardMotionThroughScene • AudioOnlyStorytelling • AudioDepthPerception (howFarOutthere) • WalkAnywhereandStaywithinBounds • RespondtoPlayer’sWalkSpeed AudioGOALS Walk 3
  34. 34. HowDidFCGCrosstheRoad?Image: Wonkubus
  35. 35. WalkingFocusesOnDistanceCues Auxel=R5E3D3 “Audio-Pixel” Rotation Elevation Distance 5 3 3 ? Howmanydiscreteaudiopositionscanplayershear?
  36. 36. WalkAudioFirstLampposts 3.5DoF
  37. 37. DetectWalkingfromHeadPose 3.5DoF AtLeast1AuxelApart
  38. 38. FastMotionAuxelsOverlap FastPlayerMotion A1 A0 A2 A0
  39. 39. AlternateAudioLamppostsforMoreSpatializationAR+AD 3.5DoF “onrails” TopDownView AD-Distance AR-Rotation
  40. 40. SpatialAudioWalk–BelievableRealWorldOverlay 3.5DoF AsDesigned AsUsed KeepPlayersIn AudioWorld SeatBounce WalkAround theBlock Walk Wander
  41. 41. Walk-UnscrambletheOracle
  42. 42. SpatializationTips • NarrativeAudioBeats • GroupAudioSourcesSidetoSide&3Depths • AuxelSpaceforSlow&FastMotion • ShowProgressPassByHeadw/in1Meter • FootstepsareRespectiveand SamePosition • ActionsLikePullSlicePizzaOffDoor AudioGOALS Walk 3
  43. 43. CHOICE 3 PuttingtheChoose inChooseYourOwnAdventure Image: Wonkubus
  44. 44. Choice Peck1forAdventure Image: Nicole Lazzaro
  45. 45. Mysterious
  46. 46. MaximizeChoiceFromAudioMenusinWorld • 4StateButtons • ChooseBetweenOptions • OptionsIntegratedintoWorld AudioGOALS Choose
  47. 47. PsychologyConstraintsoftheEar Audiomenusrequiredmoreresolutionthanwalkpaths comparedtovisualmenusAuxelMenus 1.Lowresolution 2.Brain’sshorttermmemorybuffers 3.Hardtorefreshaudioinformation 4.Experience1audiosourceatatime 5.Mentallymapaudiosourcewithanobject,event,or interactionopportunity
  48. 48. SpatialAudioDeadZone “FindtheFrog”
  49. 49. SimpleInteraction&SpatialImage: Nicole Lazzaro
  50. 50. 4StateAudioButtons-NoVisuals PlanB oTapthePhone oTapAudioPrompt oYes/NoNods 4StateAudioButtons I’maButton Highlight Activated UnHighlighted Gaze“beakcursor”tointeract Heardwhatyoulookedat DoubleTapHeadsetToChoose ConstantSoundsAnnoying! AddedIconsforWhoWasSpeaking
  51. 51. MenusFocusesOnRotation&Elevation Auxel=R5E3D3 “Audio-Pixel” Rotation Elevation Distance 5 3 ?
  52. 52. SpatialAudioCuestoFindMenuOption(FindTheFrog) HighPitchWingFlapLoop FairyChickenGodmother Button LowPitchFrogCroakLoop [emptyAudioPosition] On/OffHighlightAudio DoubleTapHeadset ToChoose
  53. 53. StonehengeAudioMenus-ChickenFlapAudioCueMoves TellsstoryofChickenflyingtoanewinteraction
  54. 54. 3DoFAudio-Menus IamThePizzaDoor RibbitRibbit DoYouWantto WadeAcross? Menus3DoF NoDistatnce
  55. 55. SpatializationTips • 4-5 AuxelsHorizontallySpaced • BewareDeadZoneFront&Behind • Latency:Bluetooth,WiFi,HRTF • Frogs>Turnips • Character&StoryBeats=EasiertoRemember • BeginAudioMemorably • ReinforcewithIcons AudioGOALS Choose
  56. 56. Opportunity Knocks4 HandHeld&Haptics
  57. 57. PhoneInHand=DetailedTracking&Haptics ChallengePhoneandFaceLookDifferentPlaces=LessImmersion
  58. 58. MaximizeRotationMotionHandheld • RespondtoAuxels withPhone • MakeHandMotionAsNaturalAsHeadMotion • HapticstoenhanceAudioMenuUX AudioGOALS HoldIn Hand
  59. 59. FeelButtonsAsWellAsHearthem HapticsOffersSurfaceTexturesForAudio
  60. 60. SpatialHapticsAmplifyAudioInteraction&Position WeNeedaPBRforHaptics
  61. 61. SpatializationTips • AudioTracksPlayer’sPhoneHand • MinimizelookingleftHeadand MovingPhoneRight • ButtonStateswithHapticSignatures • HapticSurfaceandmediumMovePhone ThroughLikeWater,ForestBranches,Sand AudioGOALS HoldIn Hand
  62. 62. WrapUp 5
  63. 63. 3+ ChallengesforAudio-FirstGames Stand Walk Choose Hold&Haptics Spatial Audio>
  64. 64. Rotation Elevation Distance AudioPosition-Auxel=RED Auxel=R5E3D3 “AudioPixel” 5 Howmanydiscreteaudiopositionscanplayershear? 3 3 ? 45Positions1-3atatime
  65. 65. 3 AuxelsNeedSpacing PositionTimeMemory 5 3 TheSoundofSilence
  66. 66. AudioGOALS Stand Walk Choose HOLD SpatializationTips 1. RespondtoheadPoseandMovement 2. SpaceatLeast1Auxelapart 3. ContrastAudioPatternToSpatializeWell 4. NarrativeCuesExpandShortTermMemory
  67. 67. ThankYou!UnscrambletheOracleTEAM! • DavidFox writing&Sfx ElectricEggplant.com • PatrickO’ShaughnesseyNarr.Engine PatchedReality.com • WonkubusWonkWalk,VO.&Art (MichaelCrabbs) MichaelCrabbs.com • Nicole Lazzaro GameDesign& UX XEODesign.com
  68. 68. SignUpforUpdatesUnscrambletheOracle surveymonkey.com/r/UTOSignUp
  69. 69. ContactMeForMore • Nicole@xeodesign.com • @NicoleLazzaro • XEODesign.com FreeDownload 4K2F.com Followthe WhiteRabbit TiltWorld.com Unscramble theOracle

×