Intel Corporation · NASDAQ:INTC
Intel — Mobileye — Technical Presentation (November 2019)
49 slides from Intel's technical presentation, November 2019
Slide 1: Mobileye Sensing Status and Road Map
November 2019 Mobileye Sensing Status and Road Map Dr. Gaby Hayon, EVP R&D 1 Confidential
Slide 2: The Challenge of Sensing for the automotive market
The Challenge of Sensing for the automotive market ME sensing has three demanding customers Sensing state for ME policy under Smart agent for harvesting, ADAS products working everywhere the strict role of independency and localization and dynamic and at all conditions on millions of redundancy. inf
Slide 3: True redundancy
True redundancy Surround computer vision Radar/Lidar sub-system
Slide 4: ME's AD Perception
ME’s AD Perception ME’s AD Perception surround computer vision comprehensive env. model
Slide 5: Comprehensive CV Environmental Model
Comprehensive CV Environmental Model Full and unified surround coverage of all decision-relevant environment elements. These are generally grouped into 4 categories: Road Geometry (RG) All driving paths, explicitly/partially/implicitly indicated, their surface profile and surface type. Road Boundari
Slide 6: Robust CV Environmental Model
Robust CV Environmental Model Multiple independent visual-processing engines overlap in their coverage of the 4 categories (RG, RB, RU, RS) To satisfy extremely-low nominal failure frequencies of the CV-Sub-system Object detection DNNs RU Texture engine , example Lanes detection DNN RG Semantic Segm
Slide 7: Actionable CV Environmental Model
Actionable CV Environmental Model Support of different driving decisions & planning requires extraction of additional, essential set of contextual cues: ▪ Longitudinal and Lateral Driving plans / decisions • Overtake : Is the vehicle an obstacle? • Lane change: “Give-way“ /“take-way” labeling of obj
Slide 8: Environment Model Elements
Visual perception Environment Model Elements 8 Confidential
Slide 9: Road Users
Road Users
Slide 10: Road Users
Road Users 360 degrees detection and inter-camera tracking of any movable road-user, and actionable semantic-cues these users convey (light indicators, gestures) On top of the standalone Object detection networks running on all cameras, 2 Dedicated 360-stitching engines have been developed to assure
Slide 11: Road Users
Road Users
Slide 12: Road Users
Road Users
Slide 13: Road Users: Metric Physical Dimensions estimation
Road Users Metric Physical Dimensions estimation dramatically improving measurements quality using novelty methods Dimension net output Temporal tracker
Slide 14: Road Users: Wheels
Road Users Wheels- RU-part (relatively regular in shape) which we deliberately detect to affirm vehicle detections, 3D position, and tracking for high-function customers.
Slide 15: Road Users: Semantic segmentation
Road Users ▪ The semantic segmentation is evident of all Road users, redundant to the dedicated networks ▪ It is also evident of extremely-small visible fragments of road users; These may potentially be used as scene-level contextual cues.
Slide 16: Road Users - open door
Road Users – open door Open car door is uniquely classified , as it is both extremely common, critical and of no ground intersection
Slide 17: Road Users - VRU
Road Users - VRU Baby strollers and wheel chairs are detection through a dedicated engine on top of the highly matured pedestrians detection system
Slide 18: Road Users - VRU
Road Users - VRU Baby strollers and wheel chairs are detection through a dedicated engine on top of the highly matured pedestrians detection system
Slide 19: Road Boundaries
Road Boundaries Surround-view stitched SR FS Occupancy Grid: ▪ Fusion of free space signal from 4 parking cameras, and front camera ▪ Main usages: a very accurate signal for handling crowded scenes, and a redundancy layer for objects detection, specifically general objects as containers, cones, cart
Slide 20: Slide 20
Slide 21: Road Users semantics
Road Users Road users semantics ▪ Head/pose orientation Pedestrians posture/gesture. ▪ Vehicle light indicators Emergency vehicle/Personnel classification. Emergency vehicle , light indicators Pedestrian understanding
Slide 22: Pedestrian Gesture Understanding
Road Users Pedestrian Gesture Understanding Come closer Stop! On the phone You can pass
Slide 23: Dense Structure-based Object detection
Road Users Dense Structure-based Object detection • Redundant to the appearance-based engines • Reinforce detection and measurements to support higher level of end-functions • E.g.- dealing with “rear protruding” objects – which hover above the objects ground intersection. Confidential
Slide 24: How do we do this? DNN based multi-view stereo
Road Users How do we do this? DNN based multi-view stereo • Infers depth in "center" view using input from "center" and overlapping "surround" cameras • Flexibility in camera placement and orientation compared to canonical stereo-baseline camera pair setups • Covering blind-regions using e.g. parkin
Slide 25: Road Users
Road Users Front left Front right 500 400 300 200 100 Parking left Parking right 400 Rear right
Slide 26: DNN based multi-view stereo
Road Users DNN based multi-view stereo
Slide 27: DNN based multi-view stereo
Road Users DNN based multi-view stereo
Slide 28: Leveraging Lidar Processing Module for Stereo Camera Sensing
Road Users Leveraging Lidar Processing Module for Stereo Camera Sensing – “Pseudo-Lidar” Dense depth image from stereo High-res Pseudo-Lidar Upright obstacle ‘stick’ Object detection cameras extraction
Slide 29: View Range
Road Users View Range knowing that you don’t know • RSS safety envelope should not be violate even in areas with limited visibility • To ensure that, we must determine whether the reason for not detecting an object is because it doesn't exist or due to an occlusion • The solution- creating a 360 deg
Slide 30: View range applications
Road Users Policy-level applications Z axis view range placing "fake targets" in occluded areas that copping with occlusions deriving from intersect with ego's planned path, assuming road elevation plausible speed and trajectory Ghost target Visible range Occluded Visible range Occluded
Slide 31: View range origin legend
View range origin legend Main Front Narrow Front Front Right Front Left Rear Right Rear Left Rear
Slide 32: Road Boundaries
Road Boundaries Detection of Any delimiter of the road surface- 3D structure and semantics. Both laterally delimiting elements(FS) and longitudinally (GO/debris) The Semantic segmentation engine provides a rich, high resolution pixel-level labeling; The SSN vocabulary is especially enriched to class
Slide 33: Road boundary segmentation examples
Road Ped Curb Edge General object Flat Car GuardRail Snow Bike Concret
Slide 34: Road boundary segmentation examples
Road Ped Curb Edge General object Flat Car GuardRail Snow Bike Concret
Slide 35: Surround Road/nRoad classification
Surround Road/nRoad classification
Slide 36: Road Boundaries
Road Boundaries Detection of Any delimiter of the road surface, it’s 3D structure and semantics. Both laterally delimiting elements(FS) and longitudinally (GO/debris) The Parallax Net engine provides an accurate understanding of structure by assessing residual elevation (flow) from the locally gover
Slide 37: Debris Detection
Debris Detection Debris detection identifies structural deviations from road surface. Structure from Motion approach: geometry-based & appearance-invariant. detects any type of hazard.
Slide 38: Road Geometry - Road3 in production
Road Geometry - Road3 in production Advanced lane applications (VW) Volkswagen Passat Travel Assist 2.0 with Mobileye camera https://www.youtube.com/watch?v=s7HCI33KVHA
Slide 39: Road Geometry
Road Geometry Road4 Technology provides deep lanes understanding rather than “simple” lane-marks detection ▪ Severely occluded lane-marks - Endures gaps of over 20m within marker ▪ Semi/partly/unmarked lane marker ▪ Multi-geometry lane structures – merge, split, HWE, junctions ▪ Stable DP map also p
Slide 40: Road Geometry
Road Geometry Parallax-Net provides a dense understanding of all driving surface elevation model , and local detailed ‘longitudinal profile’ characteristics such as road bumps and ditches
Slide 41: Road Geometry
Road Geometry The Generalized HPP technology (VF) provides ▪ Host Driving Path : Geometry and Center ▪ Any-object (point) driving path ▪ Any-object (point) lane assignment ▪ Road-elevation - accounted-for by inference Does not involve explicit detection and modeling of lane-boundary evidence, but ra
Slide 42: Road Semantics
Road Semantics ▪ Road-side directives (TFL/TSR) ▪ on-road directives (text, arrows, stop-line , crosswalk) ▪ Lane type- HOV, bicycle lane ▪ The DP association ▪ Road Friction ▪ Boundary type ▪ OCR
Slide 43: Road Semantics
Road Semantics Type: ufiknown
Slide 44: Road Semantics
Road Semantics K
Slide 45: Lidar/Radar Sensing Subsystem
Lidar/Radar Sensing Subsystem Confidential
Slide 46: Lidar/Radar-only Subsystem Setup
Lidar/Radar-only Subsystem Setup Confidential
Slide 47: Environment Modeling– Road Users& Free- Space Detection
Environment Modeling– Road Users& Free- Space Detection Free-Space detection via 3D Occupancy Engine Road User detection & tracking Model-based approach Model-based approach
Slide 48: Lidar Semantics - Shape Classification
Lidar Semantics - Shape Classification Data-driven classification approach Dedicated Deep Neural Net fed with Lidar reflections to resolve semantic ambiguities. Key use-case static object near crosswalk - distinguish between: Traffic signs – drive through Pedestrians – give way
Slide 49: Lidar-Localization in Camera-Generated Map
Lidar-Localization in Camera-Generated Map Localization in sparse semantic map is enabled by extracting rich Lidar features Vehicle trajectory Bird’s view display + Semantic map information & Lidar reflections projected onto front map semantics camera