RI: Small: Grounding Probabilistic Event Logic in a Hierarchy of Video Segmentation Tubes

Oregon State University, Corvallis OR

Investigators

Abstract

The project addresses the fundamental challenge of grounding high-level semantic concepts about events into low-level video data. The key innovations include: (1) Representing events via probabilistic event logic (PEL) along with corresponding inference and learning algorithms, (2) Video segmentation into a hierarchy of space-time tubes, and (3) Robustly grounding PEL into space-time tubes via AND-OR grammars. Space-time tubes are extracted by tracking candidate object boundaries across frames, where both boundary detection and tracking are learned from training videos. PEL allows for arbitrary, probabilistic, spatiotemporal constraints among events, including the traditional compositional rules, Allen relations between time intervals, and correlations among different events. Unlike existing work, the logical nature of PEL allows humans, even non-experts, to easily inject their own knowledge into the system. PEL conducts joint, holistic inference to find the globally best parse over all events, which is grounded in an AND-OR grammar of primitive events. The AND-OR grammar uses robust graph matching of video tubes for handling uncertainty in low-level visual processing. For evaluation, two video datasets of American football and a building?s atrium are compiled, with fully annotated event labels, object tracks, and spatiotemporal segmentations. Training is provided for graduate and undergraduate students, including those from under-represented groups. The project is expected to: (a) advance the state of the art which typically focuses only on video classification; (b) make the two datasets public; (c) generate workshops/tutorials on the related topics; and (d) produce publications in the highest-impact journals/conferences.