VEC: Medium: The 3D Memex for Virtual Teleportation

University Of Washington, Seattle WA

Investigators

Abstract

Although virtual and augmented reality headsets are finally on the brink of mass consumer availability, very little is known about how to capture the world with the fidelity needed to enable the most exciting applications of VR and AR: designing a home remodeling plan by adding fixtures or removing walls with a flick of a finger, teleporting to Hawaii to "walk through" a vacation rental, or having a conversation with a close relative on their couch, as if you're literally in the same room. Achieving such teleportation scenarios will require 3D scene modeling and rendering that go beyond the current state of the art. The primary factor holding back progress on the 3D front is lack of data. There are billions of photos available online, and this has revolutionized image understanding. Since publicly available 3D data is comparably scarce, this project takes a data-driven approach to the research problem of virtual teleportation. Its centerpiece is the creation of a 3D Memex: a vast, evolving, freely available, online repository of 3D geometry, segmentations, and semantic labels. The 3D Memex will capture a broad range of household objects and scenes. Beyond geometry and labels, the 3D Memex will describe articulations and degrees of freedom. This research will leverage this new database, producing 3D-Memex-aware scanning and modeling algorithms to bring the state of the art closer to high quality immersive VR experiences. Memex-aware reconstruction algorithms will infer complete scene models, extrapolating far beyond what the camera sees, by matching to similar scenes in the 3D Memex and completing hidden geometry and texture. Instead of reconstructing static point clouds, it will build complete models and kinematic degrees of freedom (i.e., doors will open and close, chairs move, etc.) These breakthrough capabilities will enable important teleportation scenarios to be prototyped in the project. The teleportation capabilities will enable new and better applications for all, including real estate, tourism, and better interpersonal communication through live telepresence. All data, along with visualization and labeling tools, will be released freely online for the broader community, along with newly developed, next-generation 3D scanning tools. Finally, the project will advance educational objectives by engaging undergraduates directly in the ongoing research, and through a capstone course for undergraduates and graduates that will help seed the 3D Memex and expose the students to the rapidly growing technologies and algorithms for virtual reality.