RI: Small: Empowering Longer Video Understanding via Token Compression, Selection, and Reasoning

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

This project aims to advance how machines interpret video content by developing new capabilities for analyzing extended video streams; that is, ranging from several minutes to multiple hours, which is far beyond the short clips most current systems are designed to handle. As videos continue to dominate digital communication and information sharing, the ability to understand video over extended timescales is becoming increasingly essential. This research will support both live and recorded formats and encompass a broad spectrum of video sources, including footage from wearable, mobile, and fixed cameras. By equipping intelligent systems with the capacity to comprehend complex, time-varying visual information, the project is expected to drive progress in real-world applications such as interactive assistance, autonomous navigation, augmented reality, and content summarization. The primary technical challenge addressed by this project is the extreme data volume inherent in long video sequences, which can produce millions of representational units -- known as tokens -- when processed by modern vision-language models based on transformer architectures. This exceeds the context length limits of current models and hinders effective reasoning over long time horizons. To overcome these limitations, the project proposes a novel framework centered on token selection and context-aware representation. Instead of encoding entire video streams, the system will prioritize a small, highly informative subset of tokens that are dynamically selected based on both video content and user intent. The research plan integrates three core components: (1) a multi-resolution encoding strategy that adjusts token granularity to balance detail and efficiency; (2) a content- and intent-aware selection process that filters out redundancy while preserving relevance; and (3) a reasoning module that enables iterative exploration of video content to support long-term, multi-step analysis. Together, these contributions will deliver a scalable and adaptable foundation for long video understanding and support a range of emerging multimodal tasks, including temporal reasoning, object grounding, and open-ended question answering. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.