CAREER: Physically-Motivated Learning of 3D Shape and Semantics

University Of California-San Diego, La Jolla CA

Investigators

Abstract

A system that navigates or interacts with the real world must reason about 3D geometric properties such as distances or orientations of objects, as well as semantic properties such as part locations or object types. This project combines physics-based modeling of images and shapes, with the versatility of robust optimization and deep learning, to recover 3D shape and semantic information. The work establishes connections between computer vision, machine learning, computer graphics and perception. Vision is a powerful sensing modality since images encode rich information about shape and semantics. However, image formation is a physical phenomenon that often includes complex factors like shape deformations, occlusions, material properties and participating media. Consequently, practical deployment of autonomous or intelligent vision-based systems requires robustness to the effects of diverse physical factors. Such effects may be inverted by modeling the image formation process, but hand-crafted features and hard-coded rules face limitations for data inconsistent with the model. Recent advances in deep learning have led to impressive performances, but generalization of a purely data-driven approach to handle such complex effects is expensive. To address these challenges, this project develops technologies of handling the diversity of real-world images through incorporation of physical models of image formation within deep learning frameworks. The project creates a cross-disciplinary educational program in vision, graphics, learning and perception through coursework that draws connections across wide areas such as physically-based modeling, deep learning, 3D reconstruction and semantic understanding. The program also develops K-12 educative modules that provide experiential insight into novel technologies such as virtual reality or self-driving, with a focus on outreach to students from under-represented backgrounds. This research lays the foundations for physically-motivated learning of 3D shape and semantics, with benefits such as higher accuracy, better generalization or greater ease of training. It develops theoretical frameworks that relate unknown material behavior to 3D shape, which allows robust optimization frameworks and convolutional neural network architectures for material-invariant shape estimation. It designs novel network structures that model complex transformations, to generalize recovery of shape or semantics across non-rigid and articulated deformations, or distortions due to refraction and participating media. Further, it uses physical models of appearance or motion to bridge the domain gap between simulations and real images, leading to weakly supervised frameworks that mitigate the expense of data annotation. These advances enable novel applications for light field imaging, augmented reality, self-driving in challenging weather, or underwater robotic exploration. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.