III: Medium: VOCAL: Video Organization and Interactive Compositional AnaLytics

University Of Washington, Seattle WA

Investigators

Abstract

Camera deployments are commonly used in many applications such as traffic monitoring, animal behavior tracking, autonomous driving, civil engineering, and more. Extracting value from these video streams is a key research and commercial challenge; a system that can organize and provide an interface for users to easily interact with and query large-scale video is poised to be transformative in many commercial and academic domains. Yet, the video data management systems required to develop modern video applications are still in their infancy. Existing systems have important limitations that restrict their practical use: they do not adapt easily to new domains; they have limited to no support for asking complex queries; and most systems process video streams from multiple cameras independently of one another, even if the cameras are part of a coordinated deployment. This project addresses these limitations by developing VOCAL: an open-source system for Video Organization and Interactive Compositional AnaLytics. VOCAL consists of a suite of domain-agnostic tools for end-to-end video analytics. It supports users with (1) interactively organizing video data, (2) expressing and executing complex queries, and (3) querying multi-view camera deployments. This project also provides research experiences for undergraduate and graduate students and produces materials to teach K-12 students about video management and analytics. To meet the above goals, this project contributes new approaches in databases, computer vision, and AI. It also brings together some of the independent efforts across these disciplines. In particular, VOCAL highlights the possibilities of using recent self-supervised computer vision methods to build algorithms that can make data exploration feasible for large video datasets, and thereby, allowing the rapid development of domain-specific video event recognition models. VOCAL also utilizes scene graph representations to allow users to express complex queries as compositions of simpler ones. It then develops new approaches for the interactive specification and efficient execution of such queries. Finally, VOCAL contributes new approaches to seamlessly querying multi-view camera deployments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.