Qualcomm Incorporated · NASDAQ:QCOM
Qualcomm — Technical Presentation (January 2015)
50 slides from Qualcomm's technical presentation, Jan 27, 2015
Slide 1: Visual SLAM with an Event-based Camera
Qualcomm Augmented Reality Lecture Series: Visual SLAM with an Event-based Camera Hanme Kim Supervisor: Prof. Andrew Davison Robot Vision Group Department of Computing Imperial College London January 27, 2015 Hanme Kim Visual SLAM with an Event-based Camera
Slide 2: Robot Vision Group
Robot Vision Group • Principle investigator: Prof. Andrew Davison. MonoSLAM - Davison, ICCV 2003 DTAM - Newcombe et al., ICCV 2011 KinectFusion - Newcombe et al., ISMAR 2011 Dyson Robotics Lab • Closely linked to the Dyson Robotics Laboratory. Hanme Kim Visual SLAM with an Event-based Camera
Slide 3: Self Introduction
Self Introduction • 2nd year PhD student with industrial experience (about 12 years). CCTV A/V Mixer ISP Sensor Guided Robot ANPR SceneLib2 Hanme Kim Visual SLAM with an Event-based Camera
Slide 4: Visual SLAM Applications
Visual SLAM Applications • We are interested in visual SLAM applications. Dyson 360 Eye Google Tango Amazon PrimeAir • They require: ◦ fast control feedback; ◦ high dynamic range; ◦ low power consumption and hardware complexity. Hanme Kim Visual SLAM with an Event-based Camera
Slide 5: Limited by Conventional Imaging Sensors
Limited by Conventional Imaging Sensors • Only work well with controlled camera motion and scene condition. • High power consumption (e.g. Hands-free Google Glass with a battery pack on the hand!). • High hardware complexity (e.g. powerful GPU requirement). www.techradar.com Hanme Kim Visual SLAM wi
Slide 6: Sophisticated State-of-the-Art Algorithms
Sophisticated State-of-the-Art Algorithms • Sophisticated algorithms which mainly reduce their computational burden by selecting and processing only informative data. Semi-Dense VO, Engel, J. et al., ICCV 2013 Semi-Direct VO, Forster, C. et al., ICRA 2014 • They still rely on conventional imaging se
Slide 7: Motivation
Motivation • How to satisfy these requirements of real-time SLAM applications? • Can bio-inspired silicon retinas from Neuromorphics be a solution? DVS128, iniLabs & Background from Nano Retina Inc. Hanme Kim Visual SLAM with an Event-based Camera
Slide 8: Event-based Camera
Event-based Camera Benosman, R. et al., 2014 Hanme Kim Visual SLAM with an Event-based Camera
Slide 9: DVS (Dynamic Vision Sensor) Live Demo
DVS (Dynamic Vision Sensor) Live Demo Hanme Kim Visual SLAM with an Event-based Camera
Slide 10: Event-based Camera
Event-based Camera • Advantages: ◦ Asynchronous and fast visual measurements, low latency. ◦ High dynamic range. ◦ Compressed visual information requires lower transmission bandwidth, storage capacity, processing time, and power consumption. • Has potential to overcome the limitations of conventiona
Slide 11: e.g. How To Calibrate It?
e.g. How To Calibrate It? Hanme Kim Visual SLAM with an Event-based Camera
Slide 12: Related Work: Tracking and Recognition
Related Work: Tracking and Recognition • Tracking algorithms showing its low latency capability. Delbruck, T. and Lichtsteiner, P., 2007 DVS Laser Tracker, 2008 Conradt, J. et al., 2009 • Combined with biologically inspired learning approaches. Pérez-Carrasco, J. A. et al., PAMI 2013 Lee, J. et al.
Slide 13: Related Work: SLAM Applications
Related Work: SLAM Applications • SLAM with limitations or extra sensors. Weikersdorfer, D. et al., ICVS 2013 Mueggler, E. et al., IROS 2014 Weikersdorfer, D. et al., ICRA 2014 Censi, A. and Scaramuzza, D., ICRA 2014 Hanme Kim Visual SLAM with an Event-based Camera
Slide 14: What We Want To Achieve
What We Want To Achieve • 3D Visual SLAM with a single event-based camera: ◦ able to track extremely fast 6 DoF camera motion and reconstruct 3D scenes; ◦ requires low computational cost, hardware complexity and power consumption; ◦ suitable for real world applications. ETAM (Event-based Tracking an
Slide 15: Simultaneous Mosaicing and Tracking with an Event Camera
Simultaneous Mosaicing and Tracking with an Event Camera • Hanme Kim, Ankur Handa, Ryad Benosman, Sio-Hoi Ieng, Andrew J. Davison. • Published at BMVC (British Machine Vision Conference) 2014. • Oral presentation (7.7% acceptance rate). • Best industry paper. Hanme Kim Visual SLAM with an Event-base
Slide 16: Proposed Algorithm
Proposed Algorithm Hanme Kim Visual SLAM with an Event-based Camera
Slide 17: Event-based Tracking
Event-based Tracking (t) (t) (t) (t) (t) (t) (t) {p1 , p2 , p3 , p4 }, pi = {Ri ∈ SO(3), wi } Hanme Kim Visual SLAM with an Event-based Camera
Slide 18: Event-based Tracking
Event-based Tracking (t) (t−τ ) exp( 3k=1 nk Gk ), ni ∼ N (0, σi2 ) P Ri = Ri Hanme Kim Visual SLAM with an Event-based Camera
Slide 19: Event-based Tracking
Event-based Tracking Hanme Kim Visual SLAM with an Event-based Camera
Slide 20: Event-based Tracking
Event-based Tracking (t) (t) (t−τ ) (t) (t−τc ) w1 = P(z|R1 )w1 , z = log(M(pm )) − log(M(pm )) Hanme Kim Visual SLAM with an Event-based Camera
Slide 21: Event-based Tracking
Event-based Tracking (t) (t) (t−τ ) (t) (t−τc ) w1 = P(z|R1 )w1 , z = log(M(pm )) − log(M(pm )) Hanme Kim Visual SLAM with an Event-based Camera
Slide 22: Event-based Tracking
Event-based Tracking (t) (t) (t−τ ) (t) (t−τc ) w2 = P(z|R2 )w2 , z = log(M(pm )) − log(M(pm )) Hanme Kim Visual SLAM with an Event-based Camera
Slide 23: Event-based Tracking
Event-based Tracking (t) (t) (t−τ ) (t) (t−τc ) w3 = P(z|R3 )w3 , z = log(M(pm )) − log(M(pm )) Hanme Kim Visual SLAM with an Event-based Camera
Slide 24: Event-based Tracking
Event-based Tracking (t) (t) (t−τ ) (t) (t−τc ) w4 = P(z|R4 )w4 , z = log(M(pm )) − log(M(pm )) Hanme Kim Visual SLAM with an Event-based Camera
Slide 25: Event-based Tracking
Event-based Tracking Hanme Kim Visual SLAM with an Event-based Camera
Slide 26: Event-based Tracking
Event-based Tracking Hanme Kim Visual SLAM with an Event-based Camera
Slide 27: Event-based Tracking
Event-based Tracking Hanme Kim Visual SLAM with an Event-based Camera
Slide 28: Event-based Tracking
Event-based Tracking Hanme Kim Visual SLAM with an Event-based Camera
Slide 29: Gradient Estimation
Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 30: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 31: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 32: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 33: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 34: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 35: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation • Measurement z and measurement model h. (t) (t) z (t) = τ1c , h(t) = g C·v • Update the gradient vector and its covariance matrix using standard EKF equation. (t) (t−τc ) g(t) = g(t−τc ) + Wν, Pg = Pg − WSW> ν = z (t) − h(t) (t−τc ) ∂h > −1 W = Pg ∂g S (t−τc ) ∂h
Slide 36: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 37: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 38: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 39: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 40: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 41: Pixel-wise EKF Gradient Estimation
Pixel-wise EKF Gradient Estimation Hanme Kim Visual SLAM with an Event-based Camera
Slide 42: Reconstruction from Gradients in 1D
Reconstruction from Gradients in 1D Agrawal, A. and Rasker, R., 2007 Hanme Kim Visual SLAM with an Event-based Camera
Slide 43: Reconstruction from Gradients in 1D
Reconstruction from Gradients in 1D Agrawal, A. and Rasker, R., 2007 Hanme Kim Visual SLAM with an Event-based Camera
Slide 44: Reconstruction from Gradients in 1D
Reconstruction from Gradients in 1D Agrawal, A. and Rasker, R., 2007 Hanme Kim Visual SLAM with an Event-based Camera
Slide 45: Reconstruction from Gradients in 2D
Reconstruction from Gradients in 2D • Reconstruct the log intensity of the image whose gradients Mx and My across the whole image domain are close to the estimated gradient gx and gy in a least squares sense (Tumblin, J. et al., 2005): (Mx − gx )2 + (My − gy )2 dxdy . RR J(M) = The Euler-Lagrange eq
Slide 46: Demo Video
Demo Video Hanme Kim Visual SLAM with an Event-based Camera
Slide 47: Contributions and Limitations
Contributions and Limitations • Contributions ◦ Track camera rotation while building a mosaic of a scene purely based on an event stream with no additional sensing. ◦ Reconstruct high resolution and dynamic range scenes by harnessing the characteristics of event cameras. ◦ Show that all visual infor
Slide 48: Current Extensions
Current Extensions • Real-time operation. ◦ Passion equation → parallelizable primal-dual equation. ◦ Particle filter based tracking → EKF based tracking. ◦ Track against an estimated gradient map directly. • Performance improvement. ◦ Stronger motion model (e.g. constant velocity or acceleration).
Slide 49: Move Towards 3D SLAM
Move Towards 3D SLAM • 3D simulator development. ◦ Generate synthetic 3D scenes (POV-Ray), trajectories and event datasets. ◦ Plan to make it as open source. • Depth estimation. ◦ Events are also related to translating motion and depth. ETAM (Event-based Tracking and Mapping), conceptual drawing Han
Slide 50: Things Worth Restating
Things Worth Restating • Useful pixels are determined in hardware, at no computational cost. • Event camera vs conventional camera. Event Camera Conventional Camera Data Rate 40-180kB/s 10MB/s Latency few µs few ms Dynamic Range about 120dB about 60dB Power Consumption few hundreds mW few W • Much m