Qualcomm Incorporated · NASDAQ:QCOM
Qualcomm — Technical Presentation
37 slides from Qualcomm's technical presentation
Slide 1: Dense 3D scene flow estimation for locally rigid scenes
Dense 3D scene flow estimation for locally rigid scenes Konrad Schindler Photogrammetry and Remote Sensing, ETH Zürich
Slide 2: 3D Scene Flow
3D Scene Flow ! “Joint stereo and optical flow” ! Given !2 video frames from !2 different viewpoints Institute of Geodesy and Photogrammetry 2
Slide 3: 3D Scene Flow
3D Scene Flow ! “Joint stereo and optical flow” ! Given !2 video frames from !2 different viewpoints ! Estimate dense 3D shape and 3D motion field ! Institute of Geodesy and Photogrammetry 3
Slide 4: 3D Scene Flow
3D Scene Flow ! 4 unknowns per point: depth + 3D motion vector ! stereo/flow constraints between any two images ! (for simplicity, will refer to the 2-view case in the talk) Institute of Geodesy and Photogrammetry 4
Slide 5: Applications
Applications ! Entertainment, 3D TV ! Motion capture ! Driver assistance, autonomous robots [Vedula, Baker, Rander, [Wedel, Rabe, Vaudrey, Collins, Kanade 1999] Brox, Franke, Cremers 2008] Institute of Geodesy and Photogrammetry 5
Slide 6: Estimation
Estimation ! First 2D flow, then stereo [Vedula et al. 1999] ! First stereo, then 2D flow [Pons, Keriven, Faugeras 2007] [Wedel et al. 2008] ! Everything jointly [Huguet, Devernay 2007] [Basha, Moses, Kiryati 2010] ! Usually parametrized w.r.t. a reference image |flow| depth Institute of Geodesy and
Slide 7: Scene Flow is ill-posed
Scene Flow is ill-posed ! Available constraint (data term): the flow field should be compatible with all observed images ! e.g. brightness constancy, cross-correlation, Census, ... ! Like for stereo and optical flow, this is insufficient to determine the flow field ! 3D aperture problem ! uninformat
Slide 8: Standard regularization
Standard regularization ! surface / motion field is piecewise smooth û = argmin E(u) , E(u) = Edata (u) + λ · Esmooth (u) ! robust penalty to allow for discontinuities, e.g. total variation (TV-L1) � Edata = ρ(∇u)dx , ρ(∇u) = |∇u| ! widely used in stereo and optical flow estimation Institute of Geo
Slide 9: TV-L1 applied to 3D scene flow
TV-L1 applied to 3D scene flow ! Total variation of depth and motion field ! Does not work well for narrow baselines ! Isotropic smoothing is biased against depth discontinuities Institute of Geodesy and Photogrammetry 9
Slide 10: Rigidity
Rigidity ! Observation: many scenes of practical importance consist of relatively few rigid, independently moving parts ! Develop a regularizer that encourages rigidity rather than smoothness Institute of Geodesy and Photogrammetry 10
Slide 11: Attempt 1 - local rigidity
Attempt 1 - local rigidity ! For each scene point, penalize the deviation from rigidity ! look at a small patch around the point ! find the “non-rigid motion residual”, i.e. the difference between the observed motion and its projection onto the rigid motion subspace ! smoothness term is a robust fun
Slide 12: Local rigidity
Local rigidity ! For small motions this can be done quite efficiently ! linearize rotation, discretize to pixel grid ! non-rigid motion residual has linear closed form ! it is not necessary to explicitly compute the rigid motion ! Pixels in the patch are weighted to avoid fitting across depth and mo
Slide 13: Inference
Inference ! variational energy minimization ! usual tricks from optical flow can (and should) be used ! auxiliary dual variables to decouple data and smoothness terms ! course-to-fine scheme ! careful gradient interpolation ! structure-texture decomposition Institute of Geodesy and Photogrammetry 13
Slide 14: Evaluating scene flow
Evaluating scene flow ! Ground truth ! no easy way to obtain dense ground truth ! no existing benchmark data ! quantitative results only for synthetic scenes ! (note, some public ground truth is wrong) ! Error measures ! angular error of 3D flow component ! RMS of depth, normalized to scene extent !
Slide 15: Results
Rot Txyz Tz Results ! Synthetic scenes with known ground truth Institute of Geodesy and Photogrammetry 15
Slide 16: Results
Rot Txyz Tz Results ! Synthetic scenes with known ground truth Institute of Geodesy and Photogrammetry 16
Slide 17: Results
Results ! Qualitative results Institute of Geodesy and Photogrammetry 17
Slide 18: Results
Results ! Qualitative results Institute of Geodesy and Photogrammetry 18
Slide 19: Results
Results ! Stereo camera on car Institute of Geodesy and Photogrammetry 19
Slide 20: Results
Results ! Maria (three views) Institute of Geodesy and Photogrammetry 20
Slide 21: Attempt 2 - piecewise planarity and rigidity
Attempt 2 - piecewise planarity and rigidity ! Explicitly model scene as a collection of planar patches ! Recent trend in both ! stereo [e.g. Bleyer et al. 2011] and ! optical flow [e.g. Sun, Sudderth, Black 2010] [Bleyer, Gelautz, Rother, Rhemann 2009] Institute of Geodesy and Photogrammetry 21
Slide 22: Piecewise planarity and rigidity
Piecewise planarity and rigidity ! For 3D scene flow, represent scene as a collection of rigidly moving planes ! Why that? ! a good approximation for most object surfaces ! stronger regularization ! large, well-delineated support regions for estimation ! simple mapping with homographies ! (potential
Slide 23: Energy
Energy ! In addition to data fidelity and smoothness, encourage “good” segmentation ! Data term ! two optical flow pairs ! two stereo pairs ! Smoothness at boundary pixels ! 3D distances between patches small ! curvature small (computed from distances of auxiliary points) Institute of Geodesy and Ph
Slide 24: Segmentation regularization
Segmentation regularization ! Segments should be spatially coherent (" compact) ! Suitable models exist in energy-based segmentation ! Potts model to encourage segment boundaries at high gradients ! allow only assignments to nearby segments [Veksler, Boykov, Mehrani 2010] Institute of Geodesy and Ph
Slide 25: Inference
Inference ! Initial segmentation based only on intensity ! Compute multiple scene flow proposals for each segment ! run simpler scene flow, 2D flow + stereo ! fit rigidly moving planes to segments Institute of Geodesy and Photogrammetry 25
Slide 26: Inference
Inference ! Assign each segment to one of the proposals ! minimize energy function over all pixels of all segments ! #-expansion with QPBO Institute of Geodesy and Photogrammetry 26
Slide 27: Occlusions
Occlusions ! Scene flow is 3D, allows for explicit occlusion modeling ! occluded pixels do not pay a data penalty ! instead they are assigned a fixed occlusion penalty ! Integration into inference scheme ! in each binary expansion step, find cases “if super-pixel p is on plane X and super-pixel q is
Slide 28: Inference
Inference ! Estimate segment-level occlusions ! include occlusion potentials ! solve assignment again Institute of Geodesy and Photogrammetry 28
Slide 29: Inference
Inference ! Keep plane parameters fixed, reassign pixels to segments ! simpler now, because only few segments within the allowed distance ! (iterate) Institute of Geodesy and Photogrammetry 29
Slide 30: Results
Results ! Quantitative results ! KITTI flow+stereo benchmark ! errors of reprojected 2D disparity/flow (no other ground truth) Institute of Geodesy and Photogrammetry 30
Slide 31: Results
Results ! Comparison ! other 3D scene flow algorithms ! intermediate stages and variants of the framework Institute of Geodesy and Photogrammetry 31
Slide 32: Results
Results ! Qualitative results Institute of Geodesy and Photogrammetry 32
Slide 33: Results
Results ! Qualitative results ! Institute of Geodesy and Photogrammetry 33
Slide 34: Summary and Outlook
Summary and Outlook ! Does rigidity help? Yes! ! we believe it is the best regularizer for scene flow thus far ! Local or piecewise rigidity? Hard to say; ! obviously depends on the scene - “horses for courses” ! in practice piecewise so far works much better ! Something obvious missing? Yes! ! >2 t
Slide 35: Cast listing
Cast listing Christoph Vogel Stefan Roth Konrad Schindler Institute of Geodesy and Photogrammetry 35
Slide 36: ECCV 2014 - European Conference on Computer Vision
ECCV 2014 - European Conference on Computer Vision Zurich, September 5-12, 2014 HOME IMPORTANT DATES LOCAL ARRANGEMENT PEOPLE Important Dates ¢ ECCV 2014 Submission Deadline: 7 March 2014 - Friday © ECCV 2014 Supplementary Materials Deadline: 14 March 2014 - Friday © ECCV 2014 Announcement of Decisi
Slide 37: !""#$%&'##'()"*+,*-')+".(/"01231"4/@A'=+#" 5B"C$)+"895:
!""#$%&'##'()"*+,*-')+".(/"01231"4)),-#" 56"47/'-"895: """;.$--"7,7+/"7++/</+='+>? !""#$%&'##'()"*+,*-')+".(/"01231"4/@A'=+#" 5B"C$)+"895: """;,%#D/,@D"/+='+>? !!!"#$%&$"'&()%*+,-./