CAREER: RUI: CasualVR: Casual Capture of Dynamic Virtual Reality Environments

California Polytechnic State University Foundation, San Luis Obispo CA

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). The need for compelling new ways to connect, learn, and share remotely has never been more immediate and vital for society. While most digital media today consist of flat, two-dimensional images and video, virtual reality technology will allow for sharing a complete three-dimensional representation of a scene, for a truly collaborative experience. With virtual reality technology, schoolchildren could virtually stand at the entrance of an ancient pyramid, look at paintings in a renowned museum, or explore a rainforest. Towards this high-level objective, the aim of this project is to enable anyone with a smartphone to easily capture and share a highly immersive virtual reality experience of their surroundings. The development of widespread and consumer-accessible technology for sharing high-quality virtual reality experiences would enable the creation of more impactful and lasting learning experiences, support the development of empathy by enabling people to experience the lives of others more convincingly, and allow for more faithfully recordings of important experiences that might otherwise be lost. The research objectives of this project will be integrated with several education and outreach activities. The investigator will work with an interdisciplinary team of students and faculty in biology and geography to develop immersive educational content about biodiversity in California and will implement an outreach workshop for California students to introduce them to the concepts behind three-dimensional reconstruction and view synthesis and have them create their own virtual reality experiences using the tools developed. Datasets generated as part of the research activities will be made public, serving as a great resource for other researchers and application developers. Despite a groundswell of recent developments, critical issues remain in terms of reconstructing virtual reality experiences from casually captured video, handling dynamic scenes, and rendering speed. This project aims to close these gaps by exploring novel techniques for capture and reconstruction of dynamic physical environments. To address challenges in content capture and reconstruction, the project will explore new methods for automatic self-calibration and dynamic scene reconstruction from casually captured video using a neural implicit scene representation. This becomes a challenging task for more complex and dynamic scenes, where it might be tough to find a single transition (from scene to scene) which appears seamless across the entire volume. For faster capture and better handling of dynamic scenes, the project will develop deep learning techniques to automatically convert consumer panoramic video into a dynamic and immersive multi-layer representation. Addressing storage efficiency and fast rendering, the project will investigate image-based capture and reconstruction of neural light fields. The project will also produce new datasets for training and evaluating immersive view synthesis methods in dynamic scenes and develop new measures/metrics for the evaluation of the generated immersive content. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.