Qualcomm Incorporated · NASDAQ:QCOM
Qualcomm — Technical Presentation (November 2014)
40 slides from Qualcomm's technical presentation, Nov 28, 2014
Slide 1: Computational Neuroscience for Technology: Event-based Vision Sensors and Information Processing
Computational Neuroscience for Technology: Event-based Vision Sensors and Information Processing Jörg Conradt Neuroscientific System Theory Center of Competence on Neuro-Engineering Technische Universität München http://www.nst.ei.tum.de 28.11.2014 Qualcomm Vienna
Slide 2: Neuromorphic Engineering
Neuromorphic Engineering Neuro - morphic “to do with neurons”, “structure or form” i.e. neurally inspired Discovering key principles by which brains work and implementing them in technical systems that intelligently interact with their environment. Examples: • analog VLSI Circuits to implement neura
Slide 3: Computation in Brains
Computation in Brains Getting to know your Brain • 1.3 Kg, about 2% of body weight • 1011 neurons • Neuron growth: 250.000 / min (early pregnancy) … but also … loss 1 neuron/second
Slide 4: Computation in Brains
Computation in Brains Getting to know your Brain • 1.3 Kg, about 2% of body weight • 1011 neurons • Neuron growth: 250.000 / min (early pregnancy) … but also … loss 1 neuron/second “Operating Mode” of Neurons • Analog leaky integration in soma • Digital pulses (spikes) along neurites • 1014 stochast
Slide 5: Computation in Brains ... and Machines
Computation in Brains ... and Machines Getting to know your Brain Getting to know your Computer’s CPU: • 1.3 Kg, about 2% of body weight • 50g, irrelevant for most applications • 1011 neurons • 2 • 109 transistors (Intel Itanium) • Neuron growth: • ideally no modification over lifetime 250.000 / min
Slide 6: Computation in Brains ... and Machines
Computation in Brains ... and Machines Getting to know your Brain Getting to know your Computer’s CPU: • 1.3 Kg, about 2% of body weight • 50g, irrelevant for most applications • 1011 neurons • 2 • 109 transistors (Intel Itanium) • Neuron growth: • ideally no modification over lifetime 250.000 / min
Slide 7: Research Area «Neuroscientific System Theory»
Research Area «Neuroscientific System Theory» Neuronal-Style Information Processing in Closed-Control-Loop Systems • Distributed Local Information Processing • Growing and Adaptive Networks of Computational Units On-Board Vision-Based • Neuromorphic Sensor Fusion and Distributed Actuator Networks Co
Slide 8: Retinal Inspired Dynamic Vision Sensor
Retinal Inspired Dynamic Vision Sensor • 128 x 128 pixels, each signals 30mm temporal changes of illumination (“events”) • Asynchronous updates (no image frames) • 15 μs latency, up to 1Mevents/sec 52mm DVS Sensor ARM7 μC UART/SPI Joint work with the Neuromorphic Sensors Group, ETH/University Zurich
Slide 9: Event-Based Vision
Event - Based Vision Frame - based Event - based full frames singular pixels continuous time discrete time
Slide 10: Event-Based Vision
Event - Based Vision Advantages • Sparse data stream: ~0.1 MB/s bandwidth • ~10us response time • Local contrast adjustment per pixel • Automatic preprocessing Challenges • Static scenes • New vision algorithms for tracking, localization, ...
Slide 11: An Asynchronous Temporal Difference Vision Sensor
An Asynchronous Temporal Difference Vision Sensor Conradt et al, ISCAS 2009
Slide 12: An Asynchronous Temporal Difference Vision Sensor
An Asynchronous Temporal Difference Vision Sensor Conradt et al, ISCAS 2009
Slide 13: Application Example: Tracking an Active Object
Application Example: Tracking an Active Object Power Switch Microcontroller Target Marker LED eDVS Event Time: t = 5612μs Previous Event Time for PWM Signal E Event Position: PE = (3 /2) Memory: tM = 4600μs Target LED ΔtE=tE-tM 2215 1650 4132 13 4231 36mm Rechargeable 3254 4132 4600 3432 2144 Batter
Slide 14: Application Example: Tracking an Active Object
Application Example: Tracking an Active Object Marker Laser Diode Tilt Servo eDVS Sensor 240mm Pan Servo Müller et al, ROBIO 2011
Slide 15: Application Example: Tracking an Active Object
Application Example: Tracking an Active Object Georg Müller 10 time / s 8 6 Laser Diode Tilt Servo 4 eDVS Sensor 2 Pan Servo 0 Müller et al, ROBIO 2011
Slide 16: (Fun) Application Example: High Speed Balancing of Short Poles
(Fun) Application Example: High Speed Balancing of Short Poles Conradt et al, ISCAS 2009
Slide 17: High Speed Balancing: Converting Events into Meaning
(Fun) Application Example: High Speed Balancing of Short Poles Converting Events into Meaning Incoming events Current estimate Conradt et al, ECV 2009
Slide 18: Extended Application Example: High Speed Tracking
Extended Application Example: High Speed Tracking Bamford et al, submitted EBCCSP’15
Slide 19: Application Example: SLAM for Event-Based Vision Systems
Application Example: SLAM for Event-Based Vision Systems Framework: Particle Filter Weikersdorfer et al, ROBIO 2012
Slide 20: SLAM: Event Based Localization
Application Example: SLAM for Event-Based Vision Systems Event Based Localization Weikersdorfer et al, ROBIO 2012
Slide 21: SLAM: Event Based Mapping
Application Example: SLAM for Event-Based Vision Systems Event Based Mapping Weikersdorfer et al, ICVS 2013
Slide 22: SLAM: Event Based Mapping
Application Example: SLAM for Event-Based Vision Systems Event Based Mapping Weikersdorfer et al, ICVS 2013
Slide 23: Application Example: SLAM for Event-Based Vision Systems
Application Example: SLAM for Event-Based Vision Systems ? 5x real time Hoffmann et al, ECMR 2013
Slide 24: Event-based 3D SLAM with a depth-augmented Dynamic Vision Sensor
Event-based 3D SLAM with a depth-augmented Dynamic Vision Sensor David Weikersdorfer, David B. Adrian, Daniel Cremers, Jörg Conradt Technische Universität München, Germany • Sensor combination: event-based dynamic vision and PrimeSense object distance • Sparse sensory data allows real-time processin
Slide 25: An Autonomous Indoor Quadrocopter (eb3D SLAM)
An Autonomous Indoor Quadrocopter (eb3D SLAM) Parrot AR.Drone diam. ca. 80 cm Selected hardware components : eDVS (red), Mini-PC (blue), Primesense IR projector (yellow) and IR camera (green), and drone control board (magenta).
Slide 26: An Autonomous Indoor Quadrocopter (eb3D SLAM)
An Autonomous Indoor Quadrocopter (eb3D SLAM) Evaluation against ground truth: eb-3DSLAM desired trajectory ext. tracker (OptiTrack Flex13)
Slide 27: Outlook: An Autonomous Indoor Micro Quadrocopter (3D SLAM)
Outlook: An Autonomous Indoor Micro Quadrocopter (3D SLAM) Key Technical Specs: • Diameter 16cm • Weight 38g • Autonomy 14min 16cm Micro - eDVS Lightweight Lenses Microcontroller UART Port Key Specs: DVS Sensor • 20x20mm, 3g • 75mW power • 64KB SRAM • 32bit, 64MHz
Slide 28: NST
NST
Slide 29: Neuromorphic Computing Landscape
Neuromorphic Computing Landscape
Slide 30: SpiNNaker Project
SpiNNaker Project • A million mobile phone processors in one computer • Able to model about 1% of the human brain… • …or 10 mice!
Slide 31: SpiNNaker Chip
SpiNNaker Chip Mobile DDR SDRAM interface Multi-chip packaging by UNISEM Europe
Slide 32: Distributed Neuronal Computation “on chip”
Distributed Neuronal Computation “on chip” Asynchronous Spiking Neural Computation Hardware for low-power real-time operation in Closed-Loop Systems ... to simulate 1 Billion Spiking Neurons 18 Core Stand-Alone Prototype in real-time 864 Core “Rack Version” 72 Core Evaluation • Multi-channel spiking
Slide 33: The SpiNNaker – Robot Interface Board
The SpiNNaker – Robot Interface Board NST SpOmniBot WLAN Interface Interface Board Embedded 768core SpiNNaker Board OmniBot IO Board SpiNNaker SpiNNaker Sensory/ Retina Actuator Interface Board packets integrates as Balancer Stereo eDVS sensors “additional chip” … Denk et al, ICANN 2013
Slide 34: Interface to Compliant Robotic Arm
Interface to Compliant Robotic Arm
Slide 35: Interface to Compliant Robotic Arm
Interface to Compliant Robotic Arm
Slide 36: Example Projects: SpOmniBee
Example Projects: SpOmniBee • 2 laterally facing retinas • Retina events feed into SpiNNaker • Distributed computation of local optical flow • Global combination of optic flow • Computation of Motor commands on SpiNNaker, send to robot Goal: keep SpOmnibot centered in hallway with marking, similar t
Slide 37: Application Example: Event-Based Optic Flow for Robot Control
Application Example: Event-Based Optic Flow for Robot Control
Slide 38: Example Projects: Ball Balancer
Example Projects: Ball Balancer • One Retina to observe the platform in top-down view • Two motors to control platform’s pan and tilt angles • SpiNNaker Interface Board is connected to retina and motors Goal: Keep ball (one or more!) on a certain trajectory Live Demo at HBP (EU Flagship) Inaugural S
Slide 39: NST
NST
Slide 40: Neural Principles for System Control
Neural Principles for System Control http://www.nst.ei.tum.de