CAREER: Learning to Perceive the Interactive 3D World from an Image

New York University, New York NY

Investigators

Abstract

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2). The human-built world is filled with interactive objects that have parts that can be manipulated by humans, ranging from cabinets with doors to dressers with drawers. In order for intelligent machines to be able to understand and assist humans in realistic settings, they must be able to understand these objects from vision, and especially in unconstrained realistic settings. This understanding must include understanding the interactions as they occur, as well as recognizing the opportunity for interaction (i.e., that a cabinet could be interacted with even when it is untouched). These abilities are beyond the capabilities of current AI systems since these largely deal with interactive objects in restricted settings such as simulation engines. This project aims to build AI systems that can learn these properties by combining knowledge from large-scale first-person-view video demonstrations of interactions by humans as well as from 3D simulators that do not include interaction. The project has the potential to enhance efforts in many other disciplines, for instance robotics or assistive technology for people, due to the ubiquity and importance of these interactive objects. Integrated with the research is a plan to support and engage the next generation of researchers in computer vision at multiple levels via research opportunities and enhanced course materials. This project aims to achieve this goal via four directions that advance the visual understanding of interactive objects. The first direction aims to build detailed 3D models of articulating objects in unconstrained first person-video. Building on this physical understanding of articulation, the second direction plans to enhance this physical understanding with information about how a human would achieve the interaction and what it might accomplish or reveal about the scene. The third effort aims to enable understanding of articulations before they occur by building associations in 3D across frames of a video, letting a system associate and learn from examples of ongoing interactions. The fourth direction connects this understanding of interactive objects with the goal of producing a 3D understanding of the full scene, by endowing 3D reconstructions of the world with beliefs about objects that may be just out of view or temporarily occluded. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.