Qualcomm Incorporated · NASDAQ:QCOM
Qualcomm — Technical Presentation (June 2014)
50 slides from Qualcomm's technical presentation, Jun 25, 2014
Slide 1: How Do We Measure Depth Perception in Near-Field Augmented Reality Inspired by Medical Applications?
How Do We Measure Depth Perception in Near-Field Augmented Reality Inspired by Medical Applications? Dr. J. Edward Swan II 25 June 2014 Professor Dept of Computer Science & Engineering Adjunct Professor Dept of Psychology
Slide 2: Augmented Reality (AR)
Augmented Reality (AR) • AR demo from my lab • Virtual objects • At real-world locations • Spatially related to real-world objects
Slide 3: AR Medical Applications and Perceived Depth
AR Medical Applications and Perceived Depth • AR-based training system for planning brain tumor resection – Robarts Research Institute, Western University, Ontario, Canada – Kamyar Abhari, Terry Peters, and many collaborators • What is the perceived depth of the tumor?
Slide 4: How does depth perception operate?
How does depth perception operate? • We see 3D world from ambiguous 2D retinal images • Important scientific question since mid-1800’s • Current theories emphasize importance of depth cues
Slide 5: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 6: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 7: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 8: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 9: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 10: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 11: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 12: Near-Field Depth Cues
Near-Field Depth Cues n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective
Slide 13: How Do We Measure Depth Perception?
How Do We Measure Depth Perception? • Problem: That’s about a meter… perception is invisible • Solution: depth judgment tasks • Based on cognition – Verbal report • Based on perception-action Who cares? – Perceptual matching Let me hit the ball! – Blind reaching
Slide 14: Previous work in near-field AR depth perception
Previous work in near-field AR depth perception • Rolland et al. [1995] • Rolland et al. [2002] • Ellis & Menges [1998] • McCandless, Ellis, Adelstein [2000] • Edwards et al. [2004] • Singh, Ellis, Swan [2010, 2013] • Hua, Ellis, Swan [2014]
Slide 15: Exp I: Matching vs. Reaching
Exp I: Matching vs. Reaching • Used perceptual matching and blind reaching tasks • Extend Ellis & Menges [1998] apparatus and task • Effect of occluding surface (x-ray vision) • Within-subjects design
Slide 16: Exp I: Tasks
Exp I: Tasks occluder = absent, present display calibration judgment = match cross judgment = blind reach hydraulic jack
Slide 17: Exp I: Apparatus
Exp I: Apparatus —t——____ tracker emitter curtain (covers calibration cross) occluder Se closed-loop slider Open-loop slider (under the table)
Slide 18: Exp I: Results
Exp I: Results Experiment I 2 1 Error (cm), +/– 1 SEM occluder 0 -1 -2 occluder -3 -4 -5 -6 -7 Distance (cm) 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 Occluder absent present absent present Judgment match reach
Slide 19: Exp II Real/AR Exp I
Exp II Real/AR Matching > Reaching Matching , Reaching Calibration Similar Biomechanics Between-Subjects Exp I Matching/Reaching X-ray Vision Within-Subjects
Slide 20: Exp II: Real vs. AR
Exp II: Real vs. AR cameras • Compare real and AR targets background curtain • Compare perceptual matching and blind reaching judgments real-world target • Same biomechanical movement for matching and reaching closed-loop slider foam ridge (restricts HMD movement) • Between-subjects design Environme
Slide 21: Exp II: Results
Exp II: Results Experiment II 3 2 1 Error (cm), +/– 1 SEM 0 -1 -2 -3 -4 -5 -6 Distance (cm) 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 Environment real AR real AR Judgment match reach
Slide 22: The Accommodative / Vergence Problem for VR and AR
The Accommodative / Vergence Problem for VR and AR vergence distance vergence distance focal distance focal distance vergence distance focal distance Normal viewing Viewing with vergence Viewing with farther than accommodation accommodation farther than vergence
Slide 23: Exp II: Disparity Changes for AR Matching
Exp II: Disparity Changes for AR Matching Experiment II 3 2 1 Error (cm), +/– 1 SEM β 0 -1 Δv = α – β -2 -3 -4 -5 -6 Distance (cm) 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 Environment real AR real AR Judgment match reach α Experiment II AR Matching 0.4 β Δv = α – β 0.2 Δvergence°
Slide 24: Exp II: Results
Exp II: Results Experiment II 3 2 1 Error (cm), +/– 1 SEM 0 -1 -2 -3 -4 -5 -6 Distance (cm) 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 Environment real AR real AR Judgment match reach
Slide 25: Mon-Williams & Tresilian [1999], fig 2
Mon-Williams & Tresilian [1999], fig 2 Exp II Real/AR Matching > Reaching Matching , Reaching Calibration Similar Biomechanics Between-Subjects Exp I Matching/Reaching X-ray Vision Within-Subjects Matching > Reaching No Calibration Matching > Reaching Exp III Calibration Feedback Use Finger Matching
Slide 26: Exp III: Feedback, Proprioception
Exp III: Feedback, Proprioception • Feedback: – Pretest, Expose, Posttest design – Reach-1, Match-2, Reach-3 – Match-1, Reach-2, Match-3 • Point with Finger: – Reaching and matching both use finger • Compare real and AR targets
Slide 27: Exp III: Tasks
Exp III: Tasks blind reaching cardboard ridge perceptual matching
Slide 28: Comparing Exp II and Exp III Pretest
Comparing Exp II Experiment II 5 and Exp III Pretest 4 3 Error (cm), +/– 1 SEM 2 1 0 -1 -2 -3 -4 -5 -6 Distance (cm) 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 34 38 42 46 50 Environment real AR real AR Judgment match reach Experiment III 5 4 3 Error (cm), +/– 1 SEM 2 1 0 -1 -2 -3 -4 -5 -6 Distanc
Slide 29: Exp III: Feedback (Pretest, Expose, Posttest)
Exp III: Feedback 5 Experiment III Environment = real (Pretest, 4 Error (cm), +/– 1 SEM 3 Expose, 2 1 Posttest) 0 -1 -2 -3 -4 Distance (%) 55 63 71 79 87 55 63 71 79 87 55 63 71 79 87 55 63 71 79 87 55 63 71 79 87 55 63 71 79 87 Judgment Block 1 -2 3 -1 2 -3 h- ch h- ch h- ch e ac at e ac at e ac at
Slide 30: Exp III: Posttest Results (Reach-3, Match-3)
Exp III: Posttest Results (Reach-3, Match-3) Experiment III (Posttest) 5 4 3 Error (cm), +/– 1 SEM 2 1 0 -1 -2 -3 -4 -5 -6 Distance (%) 55 63 71 79 87 55 63 71 79 87 55 63 71 79 87 55 63 71 79 87 Environment real AR real AR Judgment match reach
Slide 31: Exp II Real/AR Exp I
Exp II Real/AR Matching > Reaching Matching , Reaching Calibration Similar Biomechanics Between-Subjects Overestimation AR-Matching Exp IV Exp I Overestimation Accommodation Matching/Reaching AR-Matching Collimated, Consistent, X-ray Vision Midpoint, Real Within-Subjects Younger Matching > Reaching
Slide 32: Haploscopes
Haploscopes • Commercial HMDs don’t allow enough control • Vision scientists have long built haploscopes: Presents separately- controlled image to each eye Berkeley Haploscope HASA Haploscope (HMD) Clemson Haploscope 32
Slide 33: Haploscope
Haploscope Monitor 10 cm Minimization Lens (Concave) Collimation Lens (Convex) Accommodation Lens (Concave) Stimulus Object -10 D +10 D § Adjustable focus display § Can present objects with varying accommodation Image generated by minimization lens 5 cm Image is formed at 33.3 - 50 cm 10 cm and ve
Slide 34: Exp IV: Accommodation
Exp IV: Accommodation • Environment / Accommodation: – Real / Consistent – AR / Collimated – AR / Consistent – AR / Midpoint Haploscope Environment = Real, AR • Perceptual Matching Task = match • 33 to 50 cm
Slide 35: Exp IV: Results
Exp IV: Results Experiment IV 2.5 2.0 Error (cm), +/– 1 SEM 1.5 1.0 0.5 0.0 -0.5 -1.0 Distance (cm) .3 .4 .0 .4 .0 .3 .4 . 0 .4 .0 .3 .4 .0 .4 .0 . 3 .4 .0 .4 .0 33 36 40 44 50 33 36 40 44 50 33 36 40 44 50 33 36 40 44 50 Condition eal ed en t in t R at t o llim nsis id p M C o Co
Slide 36: Exp II Real/AR Exp V
Exp II Exp V Real/AR Brightness Matching > Reaching Matching , Reaching Bright, Dim Calibration Similar Biomechanics Collimated, Consistent, Midpoint Between-Subjects Overestimation Younger AR-Matching Between-Subjects Exp IV Exp I Overestimation Accommodation Matching/Reaching AR-Matching Collimate
Slide 37: Exp V: Brightness
Exp V: Brightness • Exp IV luminance: Real-world AR Exp IV – Real-world = 15.5 cd/m2 – AR = 33.2 cd/m2 • Exp V: – Replicated Exp 4 AR conditions – dimmer object = 6.8 cd/m2 AR Exp V
Slide 38: Exp V: Results
Exp V: Results Experiment V 2.5 2.0 Error (cm), +/– 1 SEM 1.5 1.0 0.5 0.0 -0.5 -1.0 Distance (cm) .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 3.3 6.4 0.0 4.4 0.0 3.3 6.4 0.0 4.4 0.0 3.3 6.4 0.0 4.4 0.0 3.3 6.4 0.0 4.4 0.0 33 36 40 44 50 33 36 40 44 50 33 36 40 44 50 3 3 4 4 5 3 3 4 4 5 3 3 4 4 5 3
Slide 39: Exp II Real/AR Exp V
Exp II Exp V Real/AR Brightness Matching > Reaching Matching , Reaching Bright, Dim Calibration Similar Biomechanics Collimated, Consistent, Midpoint Between-Subjects Overestimation Younger AR-Matching Between-Subjects Exp IV Exp I Overestimation Accommodation Matching/Reaching AR-Matching Collimate
Slide 40: Exp VI: Age
Exp VI: Age • Observers 40+ • Same as Exp IV: – Real / Consistent – AR / Collimated – AR / Consistent – AR / Midpoint • Used bright AR objects Duane [1912], 1500 subjects
Slide 41: Experiment VI: Results
Experiment VI: Results Experiment VI 2.5 Error (cm), +/– 1 SEM 2.0 1.5 1.0 0.5 0.0 -0.5 -1.0 Distance (cm) .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 .3 .4 .0 .4 .0 33 36 40 44 50 33 36 40 44 50 33 36 40 44 50 33 36 40 44 50 33 36 40 44 5
Slide 42: Exp VII: Replicate Edwards et al. (2004)
Exp VII: Replicate Edwards et al. (2004)
Slide 43: Exp VII: Effect of Occluding Surface
Exp VII: Effect of Occluding Surface • Introduce highly salient occluding surface • Densely sample around occluding surface • Use matching judgment
Slide 44: Exp VII: Results
Exp VII: Results occluder = absent 6 4 2 0 -2 Error (mm), ± 1 SEM -4 -6 occluder = present 6 4 2 0 -2 -4 -6 -80 -60 -40 -20 0 20 Surface Relative Depth (mm)
Slide 45: Exp VII: Results
Exp VII: Results occluder = absent 6 4 y = –0.06x – 2.35 r 2 = 0.98 2 0 -2 Error (mm), ± 1 SEM -4 -6 occluder = present 6 4 2 y = –0.07x – 3.33 r = 0.70 2 0 -2 -4 -6 -80 -60 -40 -20 0 20 Surface Relative Depth (mm)
Slide 46: Exp VII: Linear Underestimation and IPD Change
Exp VII: Linear Underestimation and IPD Change occluder = absent 6 4 y = –0.06x – 2.35 r 2 = 0.98 2 0 -2 Error (mm), ± 1 SEM -4 -6 occluder = present 6 4 y = –0.07x – 3.33 r 2 = 0.70 2 0 -2 -4 -6 -80 -60 -40 -20 0 20 Surface Relative Depth (mm)
Slide 47: Exp VII: Effect of Introducing Occluder
Exp VII: Effect of Introducing Occluder occluder = absent 6 4 2 (occluder = absent) – (occluder = present) 6 0 4 ΔError (mm), ± 1 SEM -2 Error (mm), ± 1 SEM 2 -4 0 -6 occluder = present 6 -2 4 -4 2 -6 0 -80 -60 -40 -20 0 20 Surface Relative Depth (mm) -2 -4 -6 -80 -60 -40 -20 0 20 Surface Relative D
Slide 48: Summary of Findings
Summary of Findings • Accommodative / vergence mismatch matters; but sufficient to focus at midpoint of workspace • Brightness of virtual object matters; needs to be similar to surrounding real objects • Older observers as accurate as younger observers • Proper modeling of IPD (likely) matters • Sal
Slide 49: The AR Depth Perception Team
The AR Depth Perception Team • Students: – Kenny Moser (Computer Science PhD) – Nate Phillips (Computer Science PhD) • Alumni: – Chunya Hua (Computer Science MS) – Gurjot Singh (Computer Science PhD) – Adam Jones (University of Southern California) • Collaborators: – Stephen R. Ellis (NASA Ames) – C
Slide 50: How Do We Measure Depth Perception in Near-Field Augmented Reality Inspired by Medical Applications?
How Do We Measure Depth Perception in Near-Field Augmented Reality Inspired by Medical Applications? Dr. J. Edward Swan II 25 June 2014 Professor Dept of Computer Science & Engineering Adjunct Professor Dept of Psychology