RI: Small: 3D Reconstruction via Differential Rendering and Deep Learning

University Of Maryland, College Park, College Park MD

Investigators

Abstract

Digitally reconstructing the 3D shapes of real-world objects is a core technology that enables a very wide range of applications, such as autonomous robot navigation; 3D printing for personal purposes or reverse engineering; archiving and virtual heritage; creating assets for movies, games, and augmented and virtual reality; or large-scale reconstruction for geographical information systems. This project develops novel computer algorithms to reconstruct the 3D shapes of objects using digital images as inputs. It addresses significant limitations of current techniques that often lead to inaccurate results in real-life applications. To achieve this, the project follows an innovative approach leveraging artificial intelligence techniques to understand 3D shapes based on digital images. The formulation of 3D shape reconstruction using artificial intelligence methods represents an important scientific advancement that promises further advances in the research field. A student-led augmented reality (AR) and virtual reality (VR) club gains first-hand experience with state of the art research and experiments with artificial intelligence-based 3D reconstruction to design innovative AR and VR applications. This research develops algorithms building on two key techniques, differentiable rendering and deep learning. Combining these two methods leads to synergies that can overcome the limitations of current algorithms. Rendering is the process of algorithmically evaluating an image formation model, which may include sophisticated light transport effects such as non-diffuse surfaces, shadows, and indirect illumination, to compute an image of a virtual 3D object or environment. Using automatic differentiation (AD), a differentiable renderer calculates the partial derivatives of pixel values of rendered images with respect to all unknown model parameters of the virtual 3D model. Leveraging the power and generality of AD and differentiable rendering allows to overcome the overly simplistic image formation models common in previous work. In addition, multi-view reconstruction is often ill-posed because of the large number of unknown parameters and the limited information present in a set of views. Therefore, strong priors and robust error metrics are required. This work obtains these error metrics and priors using large-scale shape and image databases and deep learning techniques, to capture the full complexity of real-world objects. Crucially, it connects deep learning to the unknown 3D model parameters through differentiable rendering, which makes it possible to leverage gradient-based optimization techniques to solve for the desired 3D shapes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.