CAREER: Cameras and Algorithms that turn Rays Efficiently into Everyday Reconstructions

Brown University, Providence RI

Investigators

Abstract

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2). This project aims to enable people to capture digital versions of real-world scenes more accurately, efficiently, and flexibly than is currently possible such that it can be a simple everyday task. In computer vision, capture of the real world is called reconstruction, and it is a core challenge that requires estimating the 3D shape, motion, object materials, and lighting within a scene. Successful reconstruction can provide spatial and geometric information of objects independent from lighting conditions to intelligent systems so that they can use the information for reasoning or creating applications of these objects from different viewing angles, e.g., in virtual and augmented reality. This project will scientifically investigate how to overcome the challenges of reconstruction by combining signals from different types of cameras in a way that is consistent with the physics of image formation. The project will integrate research and education by creating new interdisciplinary courses and outreach activities (e.g., supporting our K-12 AI4ALL local effort), as well as attending teaching workshops at Brown’s Sheridan Center for Teaching and Learning. To help overcome the ill-posed problem of scene reconstruction from passive RGB cameras, this project has three areas of focus: 1) Investigate new camera systems that integrate multiple kinds of signals via physically based image formation models. Existing platforms handle typically one modality and frequency (visible light), but the project aims to combine visible light, time of flight, and event cameras to balance the negative effects of each camera and produce a signal of a quality that no individual camera could produce: high spatio-temporal resolution 3D video. 2) Investigate lighting and material decomposition via better capture, sampling, and reconstruction from heterogeneous omnidirectional cameras via new fast view synthesis methods adapted to represent incident illumination. This will use learned material priors from factorizations of physically based reflectance models that can exploit captured full and partial omnidirectional samples. 3) Investigate hybrid representations, optimization, and machine learning methods, including initialization based on reliable sparse sampling from depth sensors, via physically based self-supervised transforms to constrain optimization, and via residual error channels to allow the model to explain all that it can in a physically meaningful way and still train on real-world data. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.