Slide 1: Making an on-device personal assistant a reality
June 2018 @qualcomm_tech Making an on-device personal assistant a reality Qualcomm Technologies, Inc.
Slide 2: AI brings human-like understanding and behaviors to the machines
Reasoning Learn, infer context, AI brings human-like and anticipate understanding and behaviors to the machines Perception Action Act intuitively, interact Hear, see, and naturally, and protect observe privacy 2
Slide 3: Advancing AI research to make on-device AI ubiquitous
Advancing AI research to make on-device AI ubiquitous A common platform is fundamental to scaling AI internally and across the industry IoT Mobile Automotive Perception Reasoning Action Object detection, speech Scene understanding, language Reinforcement learning recognition, contextual fusion under
Slide 4: A true personal assistant
A true personal assistant One of many use cases requiring a broad set of AI capabilities IoT Mobile Automotive Perception Reasoning Action Object detection, speech Scene understanding, language Reinforcement learning recognition, contextual fusion understanding, behavior prediction for decision maki
Slide 5: Voice is the transformative user interface we've been waiting for
v Voice is the transformative user interface (UI) we’ve been waiting for Designed to be: Always-on Conversational Personal Private Critical to create a true virtual assistant 5
Slide 6: Voice UI components required for an end-to-end solution
Voice UI components required for an end-to-end solution Machine speech chain: listener and speaker Text-to-speech Signal acquisition and playback Speech Front-end synthesis processing Natural language generation Speech denoising Natural Speech Dialog management language pre-processing processing Ech
Slide 7: Machine learning has ignited the voice UI revolution
Machine learning has ignited the voice UI revolution GMM CNN RNN + CNN 95% Human accuracy 70% 60% 62% 55% 50% Machine automatic speech recognition accuracy 1970 1980 1990 2000 2010 2020 GMM: Gaussian Mixture Model, CNN: Convolutional Neural Network, RNN: Recurrent Neural Network “As speech recogniti
Slide 8: Voice UI is proliferating across product categories
Voice UI is proliferating across product categories IoT XR Smartphones Smart and tablets speakers In-car entertainment systems Headphones TV and headsets Wearables PCs and laptops 8
Slide 9: Moving voice UI functionality to the end device
Moving voice UI functionality to the end device An end-to-end solution powered by machine learning On-device processing Cloud processing (always-on and real-time) (services) Multi-mic echo Voice Automatic speech Maps Wikipedia cancellation, activation recognition beamforming, (ASR) and speech Servic
Slide 10: Moving voice UI functionality to the end device
Moving voice UI functionality to the end device An end-to-end solution powered by machine learning On-device processing (always-on and real-time) Cloud processing (services) Multi-mic echo Voice Automatic speech Maps Wikipedia cancellation, activation recognition beamforming, (ASR) and speech Servic
Slide 11: On-device processing
Machine learning models On-device Cloud tasks processing On-device tasks Complex voice fallback of voice UI Automatic speech recognition Training and model update Provides unique benefits Natural language processing complementing the cloud Knowledge base Always-on audio cognition Services On-device
Slide 12: Speech denoising
4000 Frequency 3000 2000 Noisy speech spectrogram 1000 Speech “If people were more generous, 0 0.5 1 1.5 2 2.5 3 Time there would be no need for welfare” denoising • Single or multiple mics • Applicable for ◦ Two-way conversation DL-based DL-based denoising model denoising trained with extensive spe
Slide 13: Qualcomm VA power consumption
WCD9330 Qualcomm® Voice Activation (VA) Qualcomm Voice High accuracy, robust to background noise, and supports multiple languages Activation - 47% supports: Qualcomm VA power consumption Deep learning is improving performance Among state-of-the-art in terms of performance vs. power consumption Amazo
Slide 14: Automatic speech recognition
Automatic On-device automatic speech recognition (ASR) speech recognition “Turn Æ on Æ the Æ light” Transcribe the Acoustic features Acoustic model Language model audio to text Reduce input audio to Deep learning converts Uses context and language essential information input into linguistic units st
Slide 15: An end-to-end on-device voice UI example for smart homes
An end-to-end on-device voice UI example for smart homes Demo of automatic speech recognition and natural language understanding Large command set Intent understanding Turn on the living room lights Turn on the kitchen light Click the kitchen lights off Click kitchen light on Turn off all lights Swi
Slide 16: A true virtual assistant
A true virtual assistant A “digital me” sitting on the device: context aware and personalized 16
Slide 17: Contextual intelligence is required for personalization
Contextual intelligence is required for personalization The fusion of many types of sensors and personal information Sensor data On-device data Off-device data Environment Iris scan Ambient light Compass Calendar Cloud data Camera Microphone Temperature Humidity Messaging IoT data Sensor fusion Gyro
Slide 18: Creating personalized memories
Creating personalized memories Sound analysis Activity analysis Talking with my son at sunset in Strolling on the beach at sunset La Jolla in La Jolla talking with my son Live sentiment Visual analysis analysis A sunset over the ocean Strolling on the beach at sunset in La Jolla in La Jolla talking
Slide 19: A true personal assistant is responsive and proactive
A true personal assistant is responsive and proactive Responsive Proactive Decision-making and conversation based Decision-making and conversation based on contextual analysis and prompting on contextual analysis without prompting (e.g. finding memories) (e.g. automatically sharing memories) “Rememb
Slide 20: The first step to an on-device virtual assistant
The first step to an on-device virtual assistant Enabling on-device voice UI On-device processing Cloud processing Multi-mic echo Voice Automatic speech Maps Wikipedia cancellation, activation recognition (ASR) beamforming, and speech Service manager denoising SMS Natural language understanding (NLU
Slide 21: Adding an "AI agent" to create a true virtual assistant
Adding an “AI agent” to create a true virtual assistant The on-device AI agent continuously learns personal knowledge and acts intuitively Sensors On-device processing Cloud processing (services) Multi-mic echo Voice Automatic speech cancellation and activation recognition (ASR) beamforming Cloud kn
Slide 22: Adding an "AI agent" to create a true virtual assistant
Adding an “AI agent” to create a true virtual assistant Contextualization allows personalization at acoustic, intent, and behavior levels Sensors On-device processing AI agent Cloud processing (services) Multi-mic echo Voice Automatic speechContextual fusion Speaker identification cancellation and a
Slide 23: Acoustic event detection
Various kitchen noise samples Acoustic event ML-based acoustic event detection classification • ML techniques 50 0.9 Posterior-gram are used to 100 0.8 0.7 ◦ Classify acoustic 150 0.6 signals into a set of 200 0.5 predefined events 250 0.4 ◦ Infer acoustic environment 300 0.3 350 0.2 • Low power, 40
Slide 24: We are advancing AI research to make on-device AI ubiquitous
We are advancing AI research to make on-device AI ubiquitous We are creating AI platform innovations that are fundamental to scaling AI across the industry We provide the low-power end-to-end on-device solution for a true personal assistant 24
Slide 25: Thank you!
Thank you! Follow us on: For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog Nothing in these materials is an offer to sell any of the References in this presentation to “Qualcomm” may mean Qualcomm components or devices referenced herein. Incorporated, Qualcomm Technologies,