NRI: INT: A New Paradigm for Geometric Reasoning through Structure from Category

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

The task of a robot determining the 3D shape and pose of an object is critical to the advancement of generally deployed collaborative robotics. Current artificial intelligence (AI) still struggles with such tasks. Until now the problem of inferring an object's 3D pose and shape (i.e., geometric reasoning) through AI has largely been treated as a 3D supervised learning problem. That is, the AI is given 2D images with corresponding 3D labels to learn. These 3D labels are costly and error prone to obtain at a large scale, acting as an intrinsic barrier to the advancement of geometric reasoning within robotics. In this project, the research team advocates a new paradigm for geometric reasoning that requires no 3D supervision - using a mathematical framework called "structure from category". Success will result in autonomous systems such as vehicles, robots, and drones with dramatically enhanced perception and planning abilities to navigate their way in 3D world. Deep neural networks (DNNs) - the heart of most AI systems in robotics - are currently used like black boxes. It is hard to instill them explicitly with real-world knowledge; instead they learn implicitly through labelled examples. This becomes a problem when such examples are not available in abundance. This research aims to shift this paradigm for AI from an opaque black-box to a transparent "glass-box", to facilitate the principled inclusion of explicit constraints and priors such as geometry. In this project, the research team has a novel way around this impasse, by reinterpreting the nonlinearities within a DNN as hierarchical sparsity constraints. Using this approach, the research team advocates for a method of inferring 3D shape and pose solely from 2D labels, dramatically improving the ability of modern DNNs to perform effective geometric reasoning at scale across large amounts of data. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.