I-Corps: Semantic Video - from Video to Descriptions

University Of South Florida, Tampa FL

Investigators

Abstract

The broader impact/commercial potential of this I-Corps project involves computer vision analysis of video, using both visual and auditory cues, to create descriptions of the content. The technology has a large variety of potential applications from law enforcement to surveillance to consumer applications. These include enabling the efficient storage and retrieval of large volumes of camera data. Smart surveillance systems can be enhanced with features that allows for summarization of daylong video footages as a list of security-relevant events. The technology can also allow automated organization of large collections of multimedia data. This I-Corps project involves commercialization feasibility research for a computer vision technology for expressing video content in terms of natural language text and grammar, i.e. semantics. This project builds on a video analysis framework that leverages state-of-the-art methods for object detection and action recognition in a unified formalism encoded in terms of a mathematical and statistical approach known as pattern theory. The video analysis approach can (i) handle structural variability of complex events without requiring large training data while exploiting easily available ontological information, (ii) overcome classification errors of machine learning classifiers of actions and objects, (iii) accommodate scene clutter, i.e. extraneous objects that do not in the activity present in the scene, (iv) and manage sequences of elementary events, all without retraining. The formalism allows for the easy incorporation of temporal, spatial, and logical constraints. This team has demonstrated this system on standard datasets used to benchmark performance in computer vision for human activity recognition tasks.