EAGER: Social Networks Based Concept Learning in Images

University Of California-Riverside, Riverside CA

Investigators

Abstract

The need for easy, quick, and intuitive search of visual data at the conceptual level is universal. This project will explore a novel network analysis-based approach to searching visual data, ultimately leading to visual search engines that would make searching images as easy as searching by keywords is today. Accomplishing this requires new approaches to machine learning of visual concepts. This project proposes a formal framework that unifies the ideas from social networks and semantic concept learning so multiple semantic concepts can be learned with high confidence. Specifically, the approach utilizes hierarchical co-occurrence correlation among concepts as cues to help the detection of individual visual concepts. The sources for robustness are the learning of co-occurrence patterns, similar to community structures in a social network, and their refinement over time. The success of concept co-occurrence detection will simplify management of personal image data with automatic tagging. With semantically organized personal content, the preferences of a user can be learned to provide personalization of various contents that he/she consumes online. This will have a broad impact for diverse applications ranging from information technology to physical, life and social sciences to intelligence organizations to news bureaus. Forensics analysis of digital data would be greatly speeded up as humans do not have to sift through large amount of data. Extension of the techniques to video, music, and multi-modal data would also provide similar ease for content consumption. The research team provides an environment integrating education and workforce development with research, and with recruiting and retaining a diverse group of students. Complementing the research activities will be new initiatives in education and public outreach. The project develops a transformative approach to explore the acquisition and refinement of semantic visual concepts systematically. First, it discovers the hierarchical co-occurrence patterns of concepts as underlying community structures in the co-occurrence network. The co-occurrence patterns play roles similar to underlying scene concepts at a higher level of semantics. Second, it proposes an approach for selecting visually-consistent-semantic concepts. Since concepts vary in their visual complexity, visual-semantic relatedness of each concept is investigated by quantitatively measuring the within-concept visual variability and the visual distances to the other concepts such that they can be modeled more reliably and detected more easily. Third, the project introduces a novel image content descriptor called concept signature that can record both the semantic concept and the corresponding confidence value inferred from low-level features. Finally, the project proposes techniques for scalability to handle and evaluate the performance on large databases by developing open source techniques and software tools. The results will be broadly disseminated through the project website (http://vislab.ucr.edu/RESEARCH/VisualSemanticConcepts/VSC.php), via regular releases of software tools and offering tutorials/workshops at major IEEE/ACM conferences.