Slide 1: CEVA and VisiSonics 3D Audio Webinar
CEVA and VisiSonics 3D Audio Webinar November 2020 Elia Shenberger, Director of Business Development, Sound BU, CEVA Prof. Ramani Duraiswami, co-Founder and CEO, VisiSonics Copyright © 2020 CEVA, Inc. CEVA Proprietary Information 1
Slide 2: Agenda
Agenda ► History of 3D Audio ► What is 3D Audio? ► Key to 3D ► Applications and Markets ► Multichannel Audio and Ambisonics ► HRTF and Head tracking for 3D Audio CEVA Proprietary Information 2
Slide 3: CEVA's Sensing and Connectivity for IoT
CEVA’s Sensing and Connectivity for IoT Cellular Camera Vision DSP 5G DSP-based platforms Powered DNN accelerators & for smartphones, RAN and cellular IoT by CDNN SDK for NN inferencing IoT Connectivity IMU Comprehensive platforms for Software & algorithms Bluetooth and Wi-Fi Sensor fusion, manageme
Slide 4: VisiSonics - focused on 3D Audio
VisiSonics - focused on 3D Audio Spatial Audio Capture Gaming VR/AR Patented chain digital microphone Physics based RealSpace™3D engine architecture designed to build large for propagating sound in acoustic digital microphone arrays; proprietary spaces rendered in real-time for all XR processing alg
Slide 5: The History of 3D Audio
The History of 3D Audio The first demonstration of 3D Audio was applied by a French Engineer – Clement Ader in 1881 Ader invented the Theatrophone - a system of telephonic transmission where listeners received a separate channel for each ear, enabling stereophonic perception of the actors on a set.
Slide 6: Audio Reproduction of Reality
Audio Reproduction of Reality ► Current Consumer Electronics Headphones – simplistic, find small sets of filters to convolve with on low MIPS DSP ► Or reproduce sound from surround speakers Sell more hardware! Content has to be mixed for these layouts ► Far from “reality” ► In the real world w
Slide 7: What is 3D Audio?
What is 3D Audio? Natural listening which gives you the same experience as being present in a scene Concert, ball game, performance etc. Traditional 5.1 and 7.1 Surround setups are not able to provide a realistic sensation of audio three-dimensionality. As opposed to Surround sound - 3D Audio lets y
Slide 8: Apple's Spatial Audio
Apple’s Spatial Audio Earlier this year Apple released a Firmware update for AirPods Pro, which enables what Apple calls spatial audio. Similar to Dolby Atmos, it applies directional audio filters, adjusting the frequencies that each ear hears so that sounds can be placed virtually around you in 3D
Slide 9: Spatial Audio in AR/VR, Gaming, Entertainment
Spatial Audio in ARVR, Gaming, Entertainment ► Headphone based ► Propagating audio from given sources or speakers in virtual space to ears ► Using an “audio engine” to construct the binaural room impulse response ► Takes as input Source/listener locations Scene geometry (RIR) Listener Head Tra
Slide 10: Computer graphics rendering engines
Computer graphics rendering engines ► Graphics moved from 2D to physically based rendering in the 1980s ► Lead to rendering from 3D models, light source models, subsurface scattering ► Lead to a hardware revolution – GPUs from NVIDIA, AMD, … ► Photo realistic movies ► Using CEVA DSP move audio rende
Slide 11: The Keys to 3D
The Keys to 3D ► 3D/Spatial audio fixes the soundstage outside the user's head for a more realistic experience ► Built up from two linear parts Accurate head tracking Spatial audio engine which takes advantage of tracking, room dimensions and HRTFs Orientation w, x, y, z Spatial Audio Immersive
Slide 12: Efficient computing
Efficient computing ► Real-time processing requires efficient, yet high performance DSP to deliver 3D Audio capability ► CEVA-X2, CEVA-BX1 and CEVA-BX2 DSPs deliver Ultra-low power and High performance capabilities for a broad range of advanced audio/voice applications, requiring signal processing a
Slide 13: CEVA and VisiSonics Collaboration
CEVA and VisiSonics Collaboration ► RealSpace™ 3D is available on CEVA-X2 and CEVA-BX1/BX2 ► RealSpace™ 3D allows stereo, 5.1, 7.1, object based audio, and high order Ambisonics streams to be spatialized, and played with head tracking ► RealSpace™3D is fully optimized to provide the best experience
Slide 14: VisiSonics RealSpace 3D
VisiSonics RealSpace™3D ► VisiSonics leverages world-class scientific discovery to add a third dimension to the sound you hear over headsets, bringing 3D sound to gaming, virtual reality, and entertainment (movies and music) platforms ► RealSpace™ 3D creates virtual placement of sound anywhere in 3-
Slide 15: Headphone market explosion
Headphone market explosion CEVA Proprietary Information 15
Slide 16: CEVA and VisiSonics Collaboration
CEVA and VisiSonics Collaboration Stereo, 5.1, 7.1 or High Order Ambisonics + 16 objects CEVA FSP200 Head tracking data Binaural CEVA-X2 based SoC Stereo out CEVA Proprietary Information 16
Slide 17: Applications and Markets
Applications and Markets 3D Audio is widely deployed in various applications such as: Amusement parks, where 3D audio sound-track immerses the guests in the ongoing story Live Nation Clubs & Theatres-Captured by Sony YouTube and Oculus VR PC and Game Consoles Gaming Headsets Smart Speakers and Sound
Slide 18: How we perceive the world auditorily
How we perceive the world auditorily CEVA Proprietary Information 18
Slide 19: How do we perceive sound location?
How do we perceive sound location? ► Naïve time and level difference Source at ears are not sufficient to describe our ability to localize sound HEAD ► Other physics mechanisms Left ear Right ear necessary to explain Scattering of sound ► Off our bodies ► Off the environment Interaural time delays
Slide 20: Audible Sound Scattering
Audible Sound Scattering Wavelengths are comparable to our ► Sound wavelengths comparable to human rooms, bodies, and features dimensions and dimensions of spaces we live in. Not an accident but evolutionary selection! ► fλ=c 1 10 ► When λ >> a large rooms wave is unaffected by object rooms/offices
Slide 21: Propagation Engine: Achieving Accurate 3D Audio
Propagation Engine Achieving Accurate 3D Audio CEVA Proprietary Information 21
Slide 22: Approach: Render Sound Correctly
Approach: “Render” Sound Correctly ► Get the sound right at the entrances to the ear canals ► Approximately solve the audio propagation problem from sources in the scene to the ear canal entrance ► Do what graphics and vision did – Move from emulation to approximate simulation Use physics based
Slide 23: Head Related Transfer Function (HRTF)
Head Related Transfer Function(HRTF) ► Scattering causes frequency dependent amplification/attenuation Effects can be of the order of tens of dB Encodes location CEVA Proprietary Information 23
Slide 24: Processing/propagation engine
Processing/propagation engine ► Direct path ► Time domain vs frequency domain ► Early reflections / diffraction/occlusion Processing Latency ► HRTF Individual ► Tracking Personalized 3 Dof/ 6 Dof Generic Piggyback on AR and VR ► Environmental ► Equalization Feedback delay network P
Slide 25: Game and Audio Engines Supported
Game and Audio Engines Supported Standalone Engines Audio Middleware Hardware DSP packages Game Engines CEVA Proprietary Information 25
Slide 26: Current Applications
Current Applications ► Game audio, AR/VR ► Sales support for building materials customer Change materials and demonstrate room change over headphones ► Mobile Engine built in to low-level processing Aircraft avionics, Mobile phones, Speakers ► PC engine APO, VAD for spatializing x.y audio Ob
Slide 27: HRTF Personalization
HRTF Personalization ► Humans have different sizes and shapes ► Ear shapes are very individual as well ► If ear shapes and body sizes are different Properties of scattered wave are different HRTFs will be very individual ► Need individual HRTFs for creating virtual audio CEVA Proprietary Informa
Slide 28: Typically HRTFs are measured
Typically HRTFs are measured Measurements from CIPIC, Sydney and AFRL are shown Many other labs also measure HRTFs directly Takes of the order of an hour per person Not scalable CEVA Proprietary Information 28
Slide 29: Scalable HRTF Personalization
Scalable HRTF Personalization 1. Rapid Direct Measurement 2. Mobile device to the Cloud CEVA Proprietary Information 29
Slide 30: The Ecosystem
Personalization Integration with HRTF Data partner’s marketplace and user Listening Habit Data management systems The Ecosystem Support for OTA App Based Games OTT Headphones Live Events updates for personalized 3D Sound, by webcam or smartphone Engine CEVA Proprietary Information 30
Slide 31: Helmholtz' Reciprocity Principle
Helmholtz’ Reciprocity Principle If sound waves are excited at a point A, the pressure at any other point B coincides, in magnitude and phase, with the pressure that would be observed at A if the sound source were placed at B CEVA Proprietary Information 31
Slide 32: Patented Fast Approach
Patented Fast Approach ► Turned out headphone drivers ► Array of tiny microphones ► Send out a highpass signal and measure received signal ► Have productized this and collected about 800 HRTFs CEVA Proprietary Information 32
Slide 33: Current version
Current version Thousands of HRTFs measured with auxiliary data CEVA Proprietary Information 33
Slide 34: Digital microphone chain architecture
Digital microphone chain architecture CEVA Proprietary Information 34
Slide 35: Remote Approach
Remote Approach ► Instead of an exact HRTF – find an improved HRTF via smart database matching ► Find best match from datasets for left and right ear images, and head size ► Use modern computer vision and AR toolkits on mobile devices ► Use ideas of HRIR scaling, and Kendall shape theory to increase
Slide 36: Landmarks and Shape Capture
Landmarks and Shape Capture Labelled by hand the two ears of all our subjects, and recorded the coordinates Using Kendall shape ideas, non-dimensionalized the measurements Scaled HRIR CEVA Proprietary Information 36
Slide 37: Detecting Landmarks
Detecting Landmarks ► Used our thousands labeled ears to train a deep network to predict landmarks on captured images ► Various augmentations and transformations to robustify process Lighting Pose CEVA Proprietary Information 37
Slide 38: Personalization
“Personalization” Landmark Prediction Ear Matching Template HRTF, Ear Photos Convert L->R Ear Convert back to L Landmarks Ear Scale, HRTF HAT Model Correction Personalized HRTF CEVA Proprietary Information 38
Slide 39: Availability
Availability ► RealSpace™3D embedded is available on CEVA-X2, CEVA-BX1 and CEVA-BX2 DSPs, optimized for best performance and ready for deployment ► RealSpace™3D is available in different variants for maximum flexibility Supports various inputs and can be connected to game engine output ► Software
Slide 40: Thank You Q&A
Contact us: Elia Shenberger, Director of Business Development, Sound BU, CEVA Elia.Shenberger@ceva-dsp.com Prof. Ramani Duraiswami, co-Founder and CEO, VisiSonics Ramani.d@visisonics.com Thank You Q&A Copyright © 2020 CEVA, Inc. and/or its subsidiaries. All rights reserved. CEVA Proprietary Informat