RI: Small: Generative Priors for 3D Reconstruction of Objects and Scenes

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

We live in a 3D world. While we only observe it via 2D retinal percepts, we have the remarkable ability to understand the underlying 3D structure from such 2D observations. Indeed, this ability is central to accomplishing a plethora of tasks in our day-to-day lives, e.g., we can understand the shape of the cup we are about to grasp or the width of the partially visible chair behind the table when planning to sit. The goal of this project is to develop perception systems that can similarly infer the 3D structure of generic objects or scenes from 2D images. While current computer vision methods can reconstruct accurate 3D from 2D images, they cannot do so given partial observations and typically require dense multi-view captures. In contrast, this project will enable reconstruction given a few (or just one) casually captured 2D images of objects and scenes. Such a system can immediately unlock applications across robotics and graphics, e.g., allowing robots to reason with complete 3D representations of objects they are interacting with, or enabling designers to easily virtualize and manipulate scenes from just a few images. This project will also contribute to the development of graduate and undergraduate students via research involvement and specialized courses and benefit the community at large through dissemination of research and organization of tutorials. This project develops a generic framework that allows leveraging expressive generative priors for high-fidelity 3D inference. The key insight is that 3D inference can be cast as a mode seeking optimization under the likelihood induced by generative priors in conjunction with the available image observations. As a first contribution, this project will formulate mechanisms for learning novel 3D generative priors that can be leveraged in this framework. Specifically, by learning diffusions models that can co-generate multi-view images or directly synthesize 3D without requiring 3D training data, this project will capture the distribution over plausible 3D structures. Building on these, the project will develop inference mechanisms to tackle different 3D reconstruction tasks such as: (a) reconstructing generic objects from single or few views, (b) obtaining relightable 3D representations from casual captures by disentangling the materials from environmental illumination, and (c) inferring editable 3D representations of complex scenes by factoring them into independent components. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.