CAREER:Towards Perceptual Agents That See and Reason Like Humans

University Of Massachusetts Amherst, Amherst MA

Investigators

Abstract

Recent advancements in computer vision systems have enabled their widespread deployment in areas like social media, healthcare, robotics, and ecology, among many others. While such applications hold exceptional promise for improving our well-being and advancing scientific discovery, the ubiquity of these intelligent systems presents new technical, social, and cultural challenges for their wide-scale adoption. This project leads an integrated effort of research, teaching, and outreach to address some of these challenges. The project develops architectures that are substantially more accurate and capable of extracting detailed information from perceptual data across different modalities. An emphasis of this work is to develop computer vision systems that can reason about data in ways that are interpretable by humans. This project also promotes diversity, engages high school, undergraduate, and graduate students in research activities, and fosters collaborations with industry and researchers in areas such as ecology and biology through workshops. This research explores new directions that improve the capabilities of visual perception and reasoning systems for analyzing image data, spatio-temporal data, and depth data. The research develops a novel class of graph-based and factorized architectures for 3D shape and spatio-temporal analysis that provide better tradeoffs between computational cost, memory overhead, and accuracy than existing models. The research develops weakly supervised techniques for learning shape and motion representations from large amounts of unlabeled data. The research also develops a novel class of techniques for transforming visual data to semantic representations such as attributes, natural language, and symbolic programs. These techniques will improve the interpretability of machine learning models and enable collaborative learning and inference between humans and AI agents. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.