CAREER: Active and Action-Centric Visual Understanding

University Of Washington, Seattle WA

Investigators

Abstract

This project develops technologies for visual semantic planning; the problem of producing ordered sequences of actions that change the current world state from what is depicted in a given image or video to the state defined by a query task. The project bridges the gap between current levels of image understanding and what is needed to actively understand the visual world to the extent that an agent can plan and perform tasks. The project develops the technology for a crucial next step in recognition: active and action-centric image understanding by semantic understanding of actions, their preconditions and effects, and visual planning. Doing so empowers several applications in healthcare, prospective memory failure care, visually impaired care, elderly care, robotics, entertainment, and education. This research addresses the visual planning problem that entails knowing what actions are, how they change the world state, and which sequences of actions change the current state to a desired one. Successful active understanding of images requires addressing several fundamental and challenging problems at the intersection of computer vision and artificial intelligence. The research is focused on the development of a framework for active visual understanding, new scalable algorithms for joint detection of actions and their arguments, new datasets and representations for actions' preconditions and effects, new algorithms for predicting the consequences of actions with intuitive laws of physics, and visual semantic planning. The developed framework is designed for active and action-centric image understanding by large-scale, semantic action recognition, modeling actions' preconditions and effects, predicting consequences of actions, and visual planning. These resources not only enable new research directions in computer vision, robotics, and AI, but also bring together some of the independent efforts across these disciplines.