RI: Small: The Shape of Visual Motion

Duke University, Durham NC

Investigators

Abstract

This project studies methods for describing motion in video. All visible points in the world are tagged by their identity, and trajectories of their projections on the image plane are tracked through space and time. This computation is performed globally, both in space and time, and motion discontinuities are explicitly delineated in the output. In contrast with previous techniques, which estimate motion primarily from the bottom up, starting with two frames at a time, the box of data from a video camera is carved up into tube-like regions whose shapes capture information about the motion and deformation of the objects visible in the scene. Novel methods include the projection of all visual motion onto a sparse basis of point trajectories through low-rank matrix data imputation; the use of L1 regularization in a function space that preserves boundaries; the generalization of robust estimation methods from variational calculus and quadratic programming for the efficient computation of tubes and occlusions in the multi-frame case; and several domain-specific techniques for initializing general but local optimization methods close to the global solution. The resulting descriptors enable video retrieval, medical diagnosis of heart rhythm anomalies, assessment of performance in sports, sign language recognition, traffic monitoring, surveillance, and more. The project also forms the basis for a new class on experimental methods for computer vision, the materials of which are made available online.