Slide 1: Enhancing the TWS User Experience with Sensor Fusion
Enhancing the TWS User Experience with Sensor Fusion December 2020 Seth Sternberg – Product Manager, Sensor Fusion Ken Walker – Member Technical Staff, Software CEVA Proprietary Information- non NDA Copyright © 2020 CEVA, Inc. CEVA CEVAProprietary ProprietaryInformation Information – non NDA 1
Slide 2: Outline
Outline ► Background ► TWS UI Elements ► Device Context Awareness ► User Context Awareness ► Complexities of Sensor Fusion ► A Look to the Future CEVA CEVAProprietary ProprietaryInformation Information – non NDA 2
Slide 3: Background
Background CEVA CEVAProprietary ProprietaryInformation Information – non NDA 3
Slide 4: Hearables Market Explosion
Hearables Market Explosion ► High Market Growth No more headphone jacks Hearables Shipments (Millions) Lower cost hardware 160 140 Bluetooth Low Energy 120 100 ► High Compound Annual Growth 80 Rate (CAGR) 60 40 24% shipments CAGR 20 0 13.4 M in 2017 2017 2018 2019 2020 2021 2022 2023 2024
Slide 5: Top Customer Purchase Drivers
Top Customer Purchase Drivers ► Context awareness is a big deal: 58% of consumers surveyed were interested in context aware earbuds or headphones Source: Qualcomm State of Play Report 2020 CEVA CEVAProprietary ProprietaryInformation Information – non NDA 5
Slide 6: Supplying Functionality to Meet Demand
Supplying Functionality to Meet Demand ► Ease of Use Natural, reliable controls on the earbud (play/pause, next/prev track, volume) Strong wireless connection w/auto pairing Virtual assistant, wake word, voice commands Acoustic Echo Cancellation and Voice Activity Detection Consistent, rob
Slide 7: Features for TWS Usage Modes
Features for TWS Usage Modes Voice Activity/ Example Active Noise Noise Acoustic Echo Virtual Usage Mode Scenarios Cancellation Reduction Activity Cancellation 3D Audio Fitness Equalizer Detection Tracking Assistant Audio/Video Home, Commute Office, Calls & Meetings Commute, Home Gam
Slide 8: Sensor Types
Sensor Types Measures Benefits Challenges Gravity when still Small & inexpensive Difficult to separate gravity Gravity + linear acceleration Low power from linear acceleration when moving Long-term stable measurement “Double integration” to linear Accelerometer of 2/3 of angular positi
Slide 9: Sensor Types
Sensor Types Measures Benefits Challenges Background noise Air pressure variations Redundancy significantly Wind from sound waves increases accuracy and enables Multiple microphones may be advanced features required Placement considerations Microphone Change in capacitance Gloves can
Slide 10: What is Sensor Fusion?
What is Sensor Fusion? Sensor fusion is using measurements from multiple sensors to give a better estimate of real state (e.g. orientation) Accelerometer and magnetometer provide long- accel Gravity term s table meas urement of orientation Sensor Orientation G yros cope provides s hort- gyro ter
Slide 11: Why is Sensor Fusion Hard?
Why is Sensor Fusion Hard? All Have Exacerbated by Which affect And can sensors anomalies environment operation destroy the UX! gain aging accel offset Angular error for AR/VR display Motion sickness temp gyro noise non-linearity voltage mag skew Inaccurate Activity Inaccurate Caloric camera manufac
Slide 12: TWS UI Elements
TWS UI Elements CEVA CEVAProprietary ProprietaryInformation Information – non NDA 12
Slide 13: Input Gestures
Input Gestures Input Which sensor? Factors Affecting Advantages Challenges • Can’t see button • Too much force – pushes • Location • Tactile feedback into ear Click Physical button on earbud • Size • Resists false negatives • Too little force – false • Force required positives • Need to brace earbud
Slide 14: Input Gestures
Input Gestures Input Which sensor? Factors Affecting Advantages Challenges • Can’t see button • Too much force – pushes • Location • Tactile feedback into ear Click Physical button on earbud • Size • Resists false negatives • Too little force – false • Force required positives • Need to brace earbud
Slide 15: Input Gestures
Input Gestures Input Which sensor? Factors Affecting Advantages Challenges • Can’t see button • Too much force – pushes • Location • Tactile feedback into ear Click Physical button on earbud • Size • Resists false negatives • Too little force – false • Force required positives • Need to brace earbud
Slide 16: Input Gestures
Input Gestures Input Which sensor? Factors Affecting Advantages Challenges • Can’t see button • Too much force – pushes • Location • Tactile feedback into ear Click Physical button on earbud • Size • Resists false negatives • Too little force – false • Force required positives • Need to brace earbud
Slide 17: Input Gestures
Input Gestures Input Which sensor? Factors Affecting Advantages Challenges • Can’t see button • Too much force – pushes • Location • Tactile feedback into ear Click Physical button on earbud • Size • Resists false negatives • Too little force – false • Force required positives • Need to brace earbud
Slide 18: Input Gestures
Input Gestures Input Which sensor? Factors Affecting Advantages Challenges • Can’t see button • Too much force – pushes • Location • Tactile feedback into ear Click Physical button on earbud • Size • Resists false negatives • Too little force – false • Force required positives • Need to brace earbud
Slide 19: Voice Input
Voice Input ► What is it? ► Advantages Single or multiple microphones fuse Natural input input to provide the best quality voice Hands-free interface signal Can support many possible commands ► How can it be used? ► Challenges Wake Words Cost increases for multiple mics Voice Commands
Slide 20: Device Context Awareness
Device Context Awareness CEVA CEVAProprietary ProprietaryInformation Information – non NDA 20
Slide 21: What is Context Awareness?
What is Context Awareness? ► Context Awareness: Understanding the state and environment of a user or device State = action, location, pose, ID, etc. Multiple sensors and devices may be used to determine the context of a single user or device ► Scene Analysis: Understanding the broader context of
Slide 22: Context Awareness - Street Café
Context Awareness - Street Café ► You’re walking down a busy sidewalk on a windy day Activity Classifier recognizes “Walking” Semantic Location recognizes “Street” Music plays automatically Notifications are blocked for safety ► A firetruck rolls past, siren blaring Audio Event “Firetruck”
Slide 23: Basic Fitness Detection
Basic Fitness Detection ► Steps/Step Counting Step counting is the basis of all fitness tracking Features are trending from phone -> wrist -> ear ► Stationary Time ► Distance and Calorie Estimates Calculated with steps and stride length (estimated or user input) ► Heart Rate Monitors (HRMs)
Slide 24: In Ear Detection
In Ear Detection ► The earbuds detect whether or not they are in the user’s ears ► Uses Automatically pause music when an earbud is removed, and play when it is put back in Switch to mono mode when only a single earbud is in Put earbuds into lower-power mode when out CEVA CEVAProprietary Propr
Slide 25: In Ear Detection
In Ear Detection Sensor Factors Affecting Advantages Challenges • False positives when sensor is covered, but not in ear • Geometry of ear/earbud • Simple implementation • False negative if not seated Proximity • Placement of sensor • Detect each earbud’s state properly in ear • Sensitivity and cali
Slide 26: In Ear Detection
In Ear Detection Sensor Factors Affecting Advantages Challenges • False positives when sensor is covered, but not in ear • Geometry of ear/earbud • Simple implementation • False negative if not seated Proximity • Placement of sensor • Detect each earbud’s state properly in ear • Sensitivity and cali
Slide 27: In Ear Detection
In Ear Detection Sensor Factors Affecting Advantages Challenges • False positives when sensor is covered, but not in ear • Geometry of ear/earbud • Simple implementation • False negative if not seated Proximity • Placement of sensor • Detect each earbud’s state properly in ear • Sensitivity and cali
Slide 28: In Ear Detection
In Ear Detection Sensor Factors Affecting Advantages Challenges • False positives when sensor is covered, but not in ear • Geometry of ear/earbud • Simple implementation • False negative if not seated Proximity • Placement of sensor • Detect each earbud’s state properly in ear • Sensitivity and cali
Slide 29: In Ear Detection
In Ear Detection Sensor Factors Affecting Advantages Challenges • False positives when sensor is covered, but not in ear • Geometry of ear/earbud • Simple implementation • False negative if not seated Proximity • Placement of sensor • Detect each earbud’s state properly in ear • Sensitivity and cali
Slide 30: Other Device Contexts
Other Device Contexts In Case Detection Low Battery Ear Tip Fit Detection Bluetooth Connection State CEVA CEVAProprietary ProprietaryInformation Information – non NDA 30
Slide 31: User Context Awareness
User Context Awareness CEVA CEVAProprietary ProprietaryInformation Information – non NDA 31
Slide 32: Movement State
Movement State ► What is it? Walking/Running/In Vehicle/Still Primarily detected by accelerometer today ► How is it used today? Turn on ANC when Still, pass-through mode when Walking or Running for safety When Running, match music to cadence or start virtual coach When Running, start GPS o
Slide 33: Transportation
Transportation ► What is it? Car/Bus/Train/Plane A deeper application of InVehicle, fusing microphone data ► How is it used today? Turn ANC to lesser level so PA announcements can be heard when on public transport Knowing mode of transportation, pull up schedules and maps On Planes, tune A
Slide 34: Fitness Activities
Fitness Activities ► What is it? Sports and fitness like swimming, hiking, gym workouts, cycling, and soccer Usually Machine Learning models based on a fusion of accelerometer, heart rate, and pressure sensors ► How is it used today? Track activity-specific metrics for motivation and improveme
Slide 35: Voice Activity Detection
Voice Activity Detection ► What is it? Fusing microphone data to detect human speech Some OEMs are starting to fuse special-purpose accelerometer for increased accuracy ► How is it used today? Differentiate speech from background noise for call noise reduction Differentiate user from other s
Slide 36: 3D Audio
3D Audio ► What is it? Natural listening which gives you the same experience as being present in a scene ► How is it used today? Increased situational awareness for pilots, driver, soldiers, call centers Improving immersion in XR experiences ► What factors affect it? Cost – requires accelero
Slide 37: The Key to 3D
The Key to 3D ► 3D/Spatial audio fixes the soundstage for a more realistic experience ► Built up from two linear parts Accurate head tracking Spatial audio algorithm Orientation w, x, y, z Spatial Audio Immersive User information Algorithm Experience ► Accurate orientation means fewer glitches a
Slide 38: Context Awareness improves the audio experience
Context Awareness improves the audio experience Activity classifiers set your audio profile Features Head tracking powers immersive 3D audio ► Activity classifiers Walking/Running/Still ► Tap / Double Tap detection InEar and Tap detection control playback ► InEar Detection Advanced Accel-only
Slide 39: Complexities of Sensor Fusion
Complexities of Sensor Fusion CEVA CEVAProprietary ProprietaryInformation Information – non NDA 39
Slide 40: Gyroscope Offset or Bias (ZRO)
Gyroscope Offset or Bias (ZRO) ► Important characteristic of IMU that directly affects Key Performance Indicator “heading drift” measured in degrees/second Gyro ZRO vs. Temp (Multiple Samples of One Sensor Model) > Gyroscope provides angular rate which should be 15 zero if not turning 10 5 ZRO (˚C/m
Slide 41: Accelerometer Offset or Bias (ZGO)
Accelerometer Offset or Bias (ZGO) ► When still, accelerometer measures the force of gravity which enables the calculation of roll and pitch from this three-axis sensor ► Any offset will result in pitch and roll errors ► Small offsets can result in moderately sized errors (see graph) which can rende
Slide 42: Datasheet Deficiencies Lead to Errors
Datasheet Deficiencies Lead to Errors Sensor Variations are not accurately described in manufacturer’s datasheets Nominal ZGO: ► Typical only, no maximum or minimum ① ① ② ② ► “Tested” in production only, not guaranteed ② ① Temperature Effects on ZGO: ► “Typical” only, no max or min Example: Flaws in
Slide 43: Compensation Through Calibration
Compensation Through Calibration Noise ► Sensor calibration is a means of True value (under observation) Sensor output Calibrated output compensating for structural errors in sensor output data ► Structural errors are differences between expected sensor output (true value) and measured sensor output
Slide 44: Types of Calibration
Types of Calibration Static (Nominal) Static (Per-Device) Dynamic (Run-time) ► Sensor data is adjusted by ► Sensor data is adjusted by ► Correction terms continuously a fixed correction term a fixed correction term updated at run-time ► For nominal calibration, ► For per-device calibration, ► Not al
Slide 45: CEVA's Calibration Approach
CEVA’s Calibration Approach ► Accel ZGO and Gyro ZRO calibrated during normal operation Tracking ZRO over temperature is especially important CEVA provides dynamic run-time calibration whenever product demands better performance ► Gyro scale cannot be calibrated dynamically w/o some truth estima
Slide 46: CEVA's Full Suite of Calibration Support
CEVA’s Full Suite of Calibration Support Our sensor analytics data allows us to create calibration records Static (Nominal) that are more accurate than using OEM’s data sheet We provide manufacturing line tools, reference designs, and Static (Per-Device) software to our customers for factory calibra
Slide 47: A Look to the Future
A Look to the Future CEVA CEVAProprietary ProprietaryInformation Information – non NDA 47
Slide 48: Audio Events & States
Audio Events & States ► Neural network based classification of microphone data Potentially fused with accel, gyro, other sensors ► Detect discrete audio events for awareness, action Car honking Baby crying Someone calling your name ► Ambient sounds that characterize a situation Outdoors
Slide 49: PDR + SLAM
PDR + SLAM ► PDR = Pedestrian Dead Reckoning User’s position relative to their start, based on steps and direction Generally IMU based, may be paired with other location service ► SLAM = Simultaneous Location and Mapping Creating a map of “landmarks” and spotting your location based on them
Slide 50: Semantic Location
Semantic Location ► Location as humans understand it, rather than machines ► A robot views its location as X,Y coordinates ► A phone views its location as latitude, longitude coordinates ► A human views location as In the Kitchen At the Office In a Crowd On the Red Line Train ► Semantic loca
Slide 51: Scene Detection
Scene Detection ► Understanding the broader context of several users or devices, or the world around them ► The context of a user is “wet”, while the scene is “in the rain” Semantic location = Outside Ambient sound = Rain Ambient light sensor = Darker Humidity + pressure = Rain Multiple us
Slide 52: The Future – The Big Game
The Future – The Big Game ► You hop on the train, heading to the big game Context - On Train Scene - Packed Train Virtual Assistant checks calendar: Scene - Travelling to Game Pregame show on radio to psych you up ANC on to block the crowd PA announcements prioritized ► You get to your s
Slide 53: Supported Sensors
Supported Contextual Gestures & Activity & Sensors Fusion Events Health accel tap double tap step count energy/distance 6-Axis gyro in ear walking running 3D head tracking proximity shake stability still CEVA CEVAProprietary ProprietaryInformation Information – non NDA 53
Slide 54: CEVA's Complete Hearables Ecosystem
CEVA’s Complete Hearables Ecosystem Sensor Fusion Hillcrest Labs contextual sensor fusion software and software stacks AI Processors AI Processors scaling from IoT to Automotive Sound DSP, smart microphone software and audio libraries CEVA’s broad range and depth of understanding adds innovative fea
Slide 55: Thank You
Contact us: Seth Sternberg, Product Manager for Sensor Fusion Seth.Sternberg@ceva-dsp.com Ken Walker, Member of Technical Staff, Software Ken.Walker@ceva-dsp.com Thank You Q&A Copyright © 2020 CEVA, Inc. and/or its subsidiaries. All rights reserved. CEVA Proprietary Information – non NDA All specifi