RI: Small: Lightly Supervised Deep Learning for Multi-Frame Visual Motion Analysis

Duke University, Durham NC

Investigators

Abstract

This project addresses the fundamental video analysis problem of determining the motion of every image point at every point in time in a video sequence. While apparently effortless for people, this problem is still a challenge for computers, especially when objects move fast, in large numbers, or in complex ways. The field of computer vision has made tremendous strides on this problem in the last few decades, but there is still ample room for improvement. This project draws on recent developments in machine learning to improve the accuracy and reliability of the estimates of point motion in video. In addition to graduate students, the project will involve undergraduates under the auspices of the Bass Connections program at Duke University. This program reaches out to students in their first college years. The thrusts of the project include the development of suitable representations of motion, the design and training of deep learning architectures, and performance evaluation. The representational challenge is paramount: While the motion of a point between two frames is a simple vector connecting the start and end point of the motion, it becomes a trajectory when multiple frames are involved. Trajectories of nearby points are often similar when they belong to the same object, but they are different when they are on different objects, and this thrust will develop the mathematics for the piecewise continuous fields of trajectories that arise as a result. Deep learning architectures and corresponding learning algorithms, at the center of the second thrust, will be re-thought to take best advantage of relations between motions at different points and times. Finally, performance evaluation will provide a nuanced understanding of the trade-offs, strengths, and weaknesses of the algorithms being developed, and will help determine what methods and parameter settings work best for what type of video. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.