CRII: RI: Learning to Predict Temporal Interestingness for Videos

University Of Florida, Gainesville FL

Investigators

Abstract

This project examines the role that implicit feedback from viewers can play in learning a temporal interestingness function for videos. The key insight is that by leveraging pupil dilation as ground truth, supervised machine learning approaches can be applied to this problem. The ubiquitous presence of cameras in every phone, and the ability to share content with the entire world have made videos a powerful tool in the hands of everyday people. This project is to address the challenge that viewers have ever-shortening attention spans, and constructing a succinct message is hard. The project provides research and training opportunities for both undergraduate and graduate students in computer vision, machine learning, and human-centered computing. This project collects a corpus of eye-tracking data as viewers watch a collection of videos, via an off-the-shelf eye-tracking device with the objective of investigating the effectiveness of a controlled brightness calibration method to separate pupillary light reflex from pupillary emotional response. The emotional response data is leveraged as dense labels for a supervised learning approach towards predicting a video interestingness function, and algorithms are developed to cut videos to their most succinct portions based on this interestingness function. A predictive model of video interestingness could potentially impact video retrieval, summarization, and search. This would impact fields as diverse as communication and online education. Further, just as research in image saliency was hugely accelerated by the use of eye-tracking data for training, validation, and benchmarking, this project can lead to a similar unification of diverse approaches, and efforts, around an implicit, temporally dense source of ground truth for temporal interestingness in videos.