Collaborative Research: RI:Medium:Understanding Events from Streaming Video - Joint Deep and Graph Representations, Commonsense Priors, and Predictive Learning

Oklahoma State University, Stillwater OK

Investigators

Abstract

While it is easy for humans to process video data and extract meanings from it, it is extremely hard to design algorithms to do so. When developed, there are many applications of this technology, such as building assistive robotics or constructing smart spaces for independent living or monitoring wildlife. Video-data capture events, which are central to the content of human experience. Events consist of objects/people (who), location (where), time (when), actions (what), activities (how), and intent (why). This project develops a computer vision-based event understanding algorithm that operates in a self-supervised, streaming fashion. The algorithm will predict and detect old and new events, learn to build hierarchical event representations, all in the context of a prior knowledge-base that is updated over time. The intent is to generate interpretations of an event that go beyond what is seen, rather than just recognition. This research pushes the frontier of computer vision by coupling the self-supervised learning process with prior knowledge, moving the field towards open-world algorithms, and needing little or no supervision. Furthermore, this project will focus on recruitment and retention of undergraduate women students through freshman and sophomore years, with attention towards underrepresented minority students at the three sites: University of South Florida, Florida State University, and Oklahoma State University. At the core of the approach is a hybrid representational hierarchy that includes both continuous representations and symbolic graph-based representations. The continuous-valued representation is the standard, vector-valued deep learning stack that ends in an embedding vector of some object or action concept in the knowledge base. The next level of the representation consists of elementary symbolic compositions of these verbs and nouns. These elementary compositions, when associated with concepts from a knowledge-base they makeup an event interpretation, containing descriptions that go beyond what is observed in the image. These symbolic levels are built using Grenander's canonical representations from pattern theory. These representations, which have flexible graph-structured backbones, are more expressive than other well-known graphical models. The specific technical aims of the project are four-fold. First, it seeks to integrate function-based continuous with energy-based Grenander's canonical symbolic representations from pattern theory into one integrated formulation based on equilibrium propagation. Second, it will research and develop ways to use and modify commonsense knowledge bases. This will help to go beyond the closed world assumption, which is implicit in the current practice of annotated data-based deep learning approaches. Third, it will develop dynamical models on graph manifolds, which will enable generative modeling of graph structures for prediction and discovery of new concepts. Fourth, inspired by finding from human perception experiments and neuroscience, it will design predictive self-supervised learning over both continuous and symbolic representations. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.