Slide 1: OpenXR State of the Union
Unifying Reality OpenXR State of the Union Brent E. Insko, PhD Lead VR Architect at Intel & OpenXR Working Group Chair GDC, March 2019 © Khronos® Group Inc. 2019 - Page 1
Slide 2: OpenXR Provisional 0.90 Release is Here!
OpenXR Provisional 0.90 Release is Here! Press announcement Updated landing page © Khronos® Group Inc. 2019 - Page 2
Slide 3: OpenXR Overview Guide
OpenXR Overview Guide For video viewers download from: https://www.khronos.org/openxr “Overview Guide” © Khronos® Group Inc. 2019 - Page 3
Slide 4: Agenda
Agenda • What is OpenXR? • A Brief History of the Standard • What are the Problems we are trying to Solve • OpenXR Timeline of Development • Technical Walkthrough • Provisional Release • What’s Next? • Recap • Call To Action • Questions © The Khronos® Group Inc. 2018 - Page 4
Slide 5: A Brief Aside…
A Brief Aside… • Note: This is a provisional release - Things are likely to change between now and final release of the 1.0 specification • Talk assume that you know: - A little bit about VR and AR - A little bit about programming and very basic real-time rendering - Nothing about the specification
Slide 6: What is OpenXR?
What is OpenXR? OpenXR is a royalty-free, open standard that provides high-performance access to Augmented Reality (AR) and Virtual Reality (VR)—collectively known as XR—platforms and devices. © The Khronos® Group Inc. 2018 - Page 6
Slide 7: A Brief History of OpenXR
A Brief History of OpenXR • Among the first VR hardware available 2016 • Need applications… - Each platform provided an SDK to interface with the hardware - Each was different from the other © The Khronos® Group Inc. 2018 - Page 7
Slide 8: XR Ecosystem Fragmentation
XR Ecosystem Fragmentation •Increased development time and therefore cost. •Increased validation overhead and therefore cost. •Time and resources spent developing one title, impacts developers’ ability to create more titles. © The Khronos® Group Inc. 2018 - Page 8
Slide 9: Major XR Runtimes
Major XR Runtimes © The Khronos® Group Inc. 2018 - Page 9
Slide 10: OpenXR
OpenXR • Recognizing the problem, several companies got together in late 2016 / early 2017 and formed the OpenXR working group in Khronos. Virtual Reality V A Augmented Reality © The Khronos® Group Inc. 2018 - Page 10
Slide 11: OpenXR – Solving XR Fragmentation
OpenXR – Solving XR Fragmentation © The Khronos® Group Inc. 2018 - Page 11
Slide 12: OpenXR – Solving XR Fragmentation
OpenXR – Solving XR Fragmentation © The Khronos® Group Inc. 2018 - Page 12
Slide 13: VR Software Stack (Example)
VR Software Stack (Example) Game VR Application Engine Browser WebXR VR APIs • Application interface • Driver(HW) interface Server Compositor VR Runtime Vulkan | Direct3D | OpenGL (+ other system APIs) VR Hardware Windows | Linux | Android © The Khronos® Group Inc. 2018 - Page 13
Slide 14: OpenXR Architecture
OpenXR Architecture OpenXR does not replace XR Runtime Systems! It enables any XR Runtime to expose CROSS-VENDOR APIs to access their functionality XR Apps and Engines Current Device State Outgoing Requests Controller/Peripheral State OpenXR Pre-distortion image to display Normalized Poses API Hapti
Slide 15: The Structure
The Structure Application OpenXR Application Layer Runtime B Runtime A OpenXR Device Plugin Extension VR / AR Hardware VR / AR VR / AR Hardware Hardware © The Khronos® Group Inc. 2018 - Page 15
Slide 16: OpenXR Philosophies
OpenXR Philosophies 1 Enable both VR and AR applications The OpenXR standard will unify common VR and AR functionality to streamline software and hardware development for a wide variety of products and platforms Be future-proof 2 While OpenXR 1.0 is focused on enabling the current state-of-the-art,
Slide 17: Where are we on the timeline?
Where are we on the timeline? Call for Participation / Exploratory Group Formation Fall F2F, October 2016: Korea Statement of Work / Working Group Formation Winter F2F, January 2017: Vancouver Specification Work Spring F2F, April 2017: Amsterdam Interim F2F, July 2017: Washington Defining the MVP Fa
Slide 18: OpenXR Application Development Walkthrough
OpenXR Application Development Walkthrough © The Khronos® Group Inc. 2018 - Page 18
Slide 19: The OpenXR Loader
The OpenXR Loader • A separate component for supporting multiple runtimes on a single system • Similar mechanism to other Khronos APIs • Loader determines the runtime to use for the requesting application • Complexity can vary from “just pick one” to more intelligent decisions based on user factors,
Slide 20: Time to use the Overview Guide (back)
Time to use the Overview Guide (back) Thanks to Yin Li, Microsoft © The Khronos® Group Inc. 2018 - Page 20
Slide 21: Initial steps: API Layers & Extensions
Initial steps: API Layers & Extensions •API Layers xrDoSomething() Application OpenXR Application Layer1::xrDoSomething() Layer(s) LayerN::xrDoSomething() Runtime MyRuntime::xrDoSomething() - xrEnumerateApiLayerProperties() - Query the API layers available - Validation, debug, tracing, profiling lay
Slide 22: The Instance
The Instance XrInstance: • The XrInstance is basically the application’s representation of the OpenXR runtime • Can create multiple XrInstances, if supported by the runtime OpenXR Runtime A • xrCreateInstance specifies app info, KHR_Extension_2 layers, and extensions KHR_Extension_3 Application xrCr
Slide 23: The System
The System XrSystem: • OpenXR groups physical devices into logical systems of related devices • XrSystem represents a grouping of devices that the application chooses to use (e.g. HMD and controllers) • XrSystems may have display, input, tracking, etc. • xrGetSystem also returns the type of form fac
Slide 24: Form Factors
Form Factors • Currently two XrFormFactors in the specification: - XR_FORM_FACTOR_HANDHELD_DISPLAY - XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY Camera Passthrough AR Stereoscopic VR / AR Projection CAVE-like Photo Credit: Dave Pape One View Two View (one per eye) Twelve Views (six per eye) XR_FORM_FACTOR_H
Slide 25: View Configurations
View Configurations • Currently two XrViewConfigurations in the specification: - XR_VIEW_CONFIGURATION_TYPE_PRIMARY_MONO - XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO Camera Passthrough AR Stereoscopic VR / AR Projection CAVE-like Photo Credit: Dave Pape One View Two View (one per eye) Twelve Views (s
Slide 26: View Configurations
View Configurations Session: • After getting the system, must determine what view configuration(s) are supported • xrEnumerateViewConfigurations() • xrGetViewConfigurationProperties() • xrEnumerateViewConfigurationViews() • Returns the recommended and max widths and heights of the views • xrEnumerat
Slide 27: The Session
The Session Session: • A session is how an application indicates it wants to render and output VR / AR frames • An app tells the runtime it wants to enter an interactive state by beginning a session with xrBeginSession and ending it with xrEndSession • No session, no frames. Application xrCreateInst
Slide 28: Session Lifecycle
Session Lifecycle Spec page 118 © The Khronos® Group Inc. 2018 - Page 28
Slide 29: Events
Events Events are messages sent from the runtime to the application. They’re put into a queue by the runtime, and read from that queue by the application using xrPollEvent © The Khronos® Group Inc. 2018 - Page 29
Slide 30: Back to the walkthrough
Back to the walkthrough © The Khronos® Group Inc. 2018 - Page 30
Slide 31: Frame Timing
Frame Timing Let’s examine frame timing first in the simplest case of a single-threaded render loop xrWaitFrame: Called before we begin simulation of the next frame. This is responsible for throttling. Also returns when frame will be displayed. xrBeginFrame: Signals that we’re ready to begin renderi
Slide 32: Frame Timing
Frame Timing Simple Multithreaded Example (DX11, OpenGL) Frame 100 Frame 101 Frame 102 Frame 103 Simulation Thread Render Thread GPU xrWaitFrame • Frame 100: Late, so we hold Frame 101 until xrBeginFrame xrBeginFrame can kick off right after the Compositor Frame Hook xrEndFrame • Frame 101: Ideally
Slide 33: Frame Timing
Frame Timing Deeply Pipelined Multithreaded Example (Unreal Engine 4 with Vulkan, DX12, Metal) Frame 100 Frame 101 Frame 102 Frame 103 Simulation Thread Render Thread Render Workers RHI Thread Render Workers GPU xrWaitFrame xrEndFrame Render Thread Fence xrBeginFrame Compositor Frame Hook © The Khro
Slide 34: Swap Chains and Rendering
Swap Chains and Rendering xrCreateSwapchain XrSwapchains: XrSwapchains are limited by the xrGetSwapchainImages capabilities of the XrSystem that they are being created for, and can xrAcquireSwapchainImage be customized on creation based on application needs xrWaitSwapchainImage Render Loop • Usage F
Slide 35: Compositor Layers
Compositor Layers The Compositor is responsible for taking all the Layers, reprojecting and distorting them, and sending them to the display • Layers are aggregated by the Compositor in xrEndFrame for display Compositor • You can use multiple layers, up to the limit of the runtime © The Khronos® Gro
Slide 36: Compositor Layers
Compositor Layers XR_TYPE_COMPOSITION_LAYER_PROJECTION: XR_TYPE_COMPOSITION_LAYER_QUAD: Most common type of Layer. This is the Quad layers are common for UI elements, or classic “eye” layer, with each eye videos or images represented in the virtual represented by a standard perspective world on a qu
Slide 37: Spaces
Spaces XrSpace +y XrSpace is one of the fundamental concepts used throughout the API to map virtual objects to real-world locations • XrSpaces define meaningful spaces within the +x environment, which can be related to one another, and used as a basis for functions that return spatial values +z • Th
Slide 38: Reference & Action Spaces
Reference & Action Spaces • Reference Spaces - 3 reference spaces predefined by the API: VIEW, +y LOCAL, and STAGE - VIEW: Head-locked space, for view rays, reticle rendering, etc. - LOCAL: world-locked origin, gravity-aligned, +Y up, +x +X right, -Z forward (useful for seated experiences) - STAGE:
Slide 39: Input and Haptics
Input and Haptics © The Khronos® Group Inc. 2018 - Page 39
Slide 40: Input and Haptics
Input and Haptics When user clicks button “a” it results in the user teleporting © The Khronos® Group Inc. 2018 - Page 40
Slide 41: Input and Haptics
Input and Haptics Input in OpenXR goes through a layer of /user/hand/left/input/a/click abstraction built around Input Actions (/interaction_profile/ControllerCorp/fancy_controller /input/a/click) (XrActions). These allow application developers to define input based on resulting action (e.g. “Grab”,
Slide 42: Input and Haptics – Simple Example
Input and Haptics – Simple Example •Example Walkthrough: - Application has sets of inputs valid for different parts of the game, for example: - Game Actions: - Explode - Teleport - Menu Actions: - Select - Exit © The Khronos® Group Inc. 2018 - Page 42
Slide 43: Input and Haptics – Create Action Sets
Input and Haptics – Create Action Sets Action Sets – collections of actions the application can quickly switch between during its lifetime xrCreateActionSet() Game Action Set Menu Action Set © The Khronos® Group Inc. 2018 - Page 43
Slide 44: Input and Haptics – Create Actions
Input and Haptics – Create Actions Actions are created and associated with an action set xrCreateAction() Game Action Set Menu Action Set Explode Action Select Action Teleport Action Exit Action In this example: xrCreateAction( <Explode Action> , <Game Action Set>) xrCreateAction(<Teleport Action> ,
Slide 45: Input and Haptics – Bind Actions to Inputs
Input and Haptics – Bind Actions to Inputs • Application suggests a set of bindings between actions and inputs • XrActionSuggestedBinding structure Action Binding Explode Action ? Teleport Action ? • How does the application know what bindings to use? © The Khronos® Group Inc. 2018 - Page 45
Slide 46: Enter Interaction Profiles…
Enter Interaction Profiles… © The Khronos® Group Inc. 2018 - Page 46
Slide 47: Input and Haptics - Interaction Profiles
Input and Haptics - Interaction Profiles • Collections of input and output sources on physical devices • Runtimes can support multiple interaction profiles ControllerCorp’s Fancy_Controller: - /user/hand/left - /user/hand/right - /input/a/click - /input/b/click - /input/c/click - /input/d/click - /i
Slide 48: Input and Haptics - Predefined Interaction Profiles
Input and Haptics - Predefined Interaction Profiles •Interaction profiles for many current products are predefined in the OpenXR specification including: - Google Daydream* controller - HTC Vive and Vive Pro* controllers - Microsoft* Mixed reality motion controllers - Microsoft* Xbox controller - Oc
Slide 49: Input and Haptics – Bind Actions to Inputs
Input and Haptics – Bind Actions to Inputs • Application suggests a set of bindings between actions and inputs • XrActionSuggestedBinding structure Action Binding Explode Action ? Teleport Action ? • How does the application know what bindings to use? © The Khronos® Group Inc. 2018 - Page 49
Slide 50: Input and Haptics – Bind Actions to Inputs
Input and Haptics – Bind Actions to Inputs • Application suggests a set of bindings between actions and inputs • XrActionSuggestedBinding structure Action Binding Explode Action ? Teleport Action ? • How does the application know what bindings to use? - The application is built around a particular i
Slide 51: Input and Haptics – Bind Actions to Inputs
Input and Haptics – Bind Actions to Inputs • Application suggests a set of bindings between actions and inputs • XrActionSuggestedBinding structure Action Binding Explode Action Trigger Click Teleport Action Button A Click • How does the application know what bindings to use? - The application choos
Slide 52: Input and Haptics – Suggested Bindings
Input and Haptics – Suggested Bindings • Application submits its suggested bindings along with the interaction profile it is using to the runtime • xrSetInteractionProfileSuggestedBinding() Action Binding ControllerCorp Explode Action Trigger Click Fancy_Controller Teleport Action Button A Click Sug
Slide 53: Input and Haptics - Runtime Binding Decision
Input and Haptics - Runtime Binding Decision • Runtime ultimately decides the bindings - “dev teams are ephermal, games last forever” - More likely the runtime is updated than individual games • Reasons for selecting different bindings: - 1. this runtime does not have ControllerCorp’s fancy_controll
Slide 54: Input and Haptics
Input and Haptics /user/hand/left/input/a/click (/interaction_profile/ControllerCorp/fancy_controller /input/a/click) OpenXR Runtime .../input/trigger/click Explode .../input/a/click Teleport XrAction: “Teleport” © The Khronos® Group Inc. 2018 - Page 54
Slide 55: Input and Haptics – Input Action Types
Input and Haptics – Input Action Types If path is a scalar value, a threshold must be applied. If not a value, needs to be bound to XR_INPUT_ACTION_TYPE_BOOLEAN …/click If path is a scalar value, then input is directly bound. If the bound value is boolean, the runtime XR_INPUT_ACTION_TYPE_VECTOR1F m
Slide 56: Input and Haptics
Input and Haptics Haptics build upon the same XrAction system, and have their own Action Type: XR_HAPTIC_VIBRATION. Just like other XrActions, they can be used with XrActionSets, but unlike inputs, they are activated with xrApplyHapticFeedback Currently, only XrHapticVibration is supported: • Start
Slide 57: That’s it for Core API coverage today
That’s it for Core API coverage today © The Khronos® Group Inc. 2018 - Page 57
Slide 58: Extensions
Extensions Core concepts that are fundamental to the Core Standard specification for all use cases Examples: Instance management, tracking, frame timing Functionality that a large class of runtimes will likely implement KHR Extensions Examples: Platform support , Graphic API Extensions, Device Plugi
Slide 59: Current Provisional Extensions
Current Provisional Extensions • Platform-specific support: • Support for specific XR layer types: - KHR_android_create_instance - KHR_composition_layer_cube - KHR_android_surface_swapchain - KHR_composition_layer_depth - KHR_android_thread_settings - KHR_composition_layer_equirect • Performance imp
Slide 60: What hasn’t made it in?
What hasn’t made it in? © The Khronos® Group Inc. 2018 - Page 60
Slide 61: What hasn’t made it in?
What hasn’t made it in? •Top priority: solve application fragmentation © The Khronos® Group Inc. 2018 - Page 61
Slide 62: Device Plugin Extension
Device Plugin Extension Runtime xrCreateInstance() xrDevicePluginConnectKHR() xrDevicePluginAddDeviceKHR() MyDevice1 (HMD) xrDevicePluginPollEventKHR() /devices/MyVendor/MyDevice Display Display Timing Info Trackability Update Pose Audio Out … xrDevicePluginSetDeviceStatusKHR() © The Khronos® Group
Slide 63: Things to consider for after 1.0 or for extensions
There are a list of things to consider for after 1.0 or for extensions • Many of the items on the list are obvious next steps in the progress of AR and VR development • Won’t list them here • But feel free to send us you list of requests in via the feedback channels we’ll provide in a sec © The Kh
Slide 64: OpenXR Provisional 0.90 Release is Here!
OpenXR Provisional 0.90 Release is Here! https://www.roadtovr.com/openxr-0-9-provisional-release-microsoft-oculus-collabora-implementations/ https://uploadvr.com/openxr-provisional-release/ https://www.phoronix.com/scan.php?page=news_item&px=Khronos-OpenXR-0.90 https://venturebeat.com/2019/03/18/khr
Slide 65: What Resources Are Available?
What Resources Are Available? © The Khronos® Group Inc. 2018 - Page 65
Slide 66: What Resources Are Available?
What Resources Are Available? • Fair warning: - This is a provisional release - This is our working group’s first ever public release - Much of the effort and time leading to the provisional release was completing the release, the supporting software and infrastructure is lagging - Some things may b
Slide 67: What Resources Are Available?
What Resources Are Available? 200+ page specification © The Khronos® Group Inc. 2018 - Page 67
Slide 68: What Resources Are Available?
What Resources Are Available? 200+ page specification Reference Pages © The Khronos® Group Inc. 2018 - Page 68
Slide 69: What Resources Are Available?
What Resources Are Available? 200+ page specification Reference Pages Overview Guide © The Khronos® Group Inc. 2018 - Page 69
Slide 70: What Resources Are Available?
What Resources Are Available? • https://github.com/KhronosGroup/OpenXR-Docs • Contains the source for generating the specification document and reference pages, scripts to be added soon • Contains the openxr header files © The Khronos® Group Inc. 2018 - Page 70
Slide 71: What Resources Are Available?
What Resources Are Available? • https://github.com/KhronosGroup/OpenXR-Registry • Contains the specification, reference pages, and overview guide © The Khronos® Group Inc. 2018 - Page 71
Slide 72: What Resources Are Available?
What Resources Are Available? • https://github.com/KhronosGroup/OpenXR-SDK • Contains the source for: - Loader - Some basic API layers - Test sample • For the current best example code, see: src/tests/hello_xr © The Khronos® Group Inc. 2018 - Page 72
Slide 73: Additional Resources
Additional Resources • OpenXR Landing Page - Specification, Reference Pages, Sample Code, Overview - https://www.khronos.org/openxr • OpenXR Forum and Slack Channel - Forum: https://khr.io/openxrfeedback - Discussion: https://khr.io/slack • Detailed specification overview and SIGGRAPH session videos
Slide 74: Engine and Platform Support
Engine and Platform Support Vinay Narayan, vice president, platform strategy, HTC “HTC VIVE is committed to creating a viable ecosystem for the XR industry which is why we are proud to support OpenXR. Bringing the community together to help define standards and best practices, allows all of us to mo
Slide 75: What’s Next?
What’s Next? © The Khronos® Group Inc. 2018 - Page 75
Slide 76: What’s Next?
What’s Next? Call for Participation / Exploratory Group Formation Fall F2F, October 2016: Korea Statement of Work / Working Group Formation Winter F2F, January 2017: Vancouver Specification Work Spring F2F, April 2017: Amsterdam Interim F2F, July 2017: Washington Defining the MVP Fall F2F, September
Slide 77: What’s Next?
What’s Next? • Continue Refining the Specification - Very unlikely any new functionality between now and 1.0 • Incorporate Community Feedback - Bug fixes, things missing, etc. • Finalize Implementations • Establish an Adopter’s Program • Develop conformance tests! - Major next hurdle - Conformance t
Slide 78: Thanks!
Thanks! • To these companies for enabling their engineers to dedicate time to OpenXR! © The Khronos® Group Inc. 2018 - Page 78
Slide 79: Thanks to the Engineers!
Thanks to the Engineers! Adam Gousetis, Google | Alex Turner, Microsoft | Andreas Loeve Selvik, Arm | Andres Rodriguez, Valve Software | Armelle Laine, Qualcomm Technologies, Inc | Blake Taylor, Magic Leap | Brad Grantham, Google | Brandon Jones, Google | Brent E. Insko, Intel | Brent Wilson, Micros
Slide 80: Recap
Recap • What is OpenXR? • A Brief History of the Standard • What are the Problems we are trying to Solve • OpenXR Timeline of Development • Technical Walkthrough • Provisional Release • What’s Next? © The Khronos® Group Inc. 2018 - Page 80
Slide 81: OpenXR Win-Win-Win
OpenXR Win-Win-Win XR End-Users Can run the apps they want on their system – reducing market confusion and increasing consumer confidence XR Vendors Can bring more applications onto their platform by leveraging the OpenXR content ecosystem XR ISVs Can easily ship on more platforms for increased mark
Slide 82: Before we go… the Call for Action…
Before we go… the Call for Action… © The Khronos® Group Inc. 2018 - Page 82
Slide 83: Before we go… the Call for Action… Feedback!
Before we go… the Call for Action… Feedback! •Tell us: - What’s wrong with the spec - What should be in the spec - What shouldn’t be in the spec - How this won’t work for your application/runtime/hardware/OS/… - How purple is a great color choice! © The Khronos® Group Inc. 2018 - Page 83
Slide 84: Before we go… the Call for Action… Feedback!
Before we go… the Call for Action… Feedback! •Tell us: - What’s wrong with the spec - What should be in the spec - What shouldn’t be in the spec Better yet, send us examples! - How this won’t work for your application/runtime/hardware/OS/… - How purple is a great color choice! © The Khronos® Group I
Slide 85: Join Khronos!
Join Khronos! •Get more involved •Have direct impact on the direction of the API •Be part of the effort to deliver OpenXR 1.0! © The Khronos® Group Inc. 2018 - Page 85
Slide 86: Thank You!
Thank You! © The Khronos® Group Inc. 2018 - Page 86
Slide 87: Questions…
Questions… Tweet #KhronosDevDay to win! © The Khronos® Group Inc. 2018 - Page 87