EAGER: Exploring Multimedia Information Networks

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

This project is built on the interdisciplinary collaboration between multimedia and natural language processing (NLP) researchers. The goal of this exploratory research is to provide effective methods for organizing, searching, mining and reasoning with web-scale multimedia. The approach is based on formulation of a structured multimedia database, called Multimedia Information Networks (MINets) which enables new searching and mining paradigms such as keyword query of unlabeled image/video databases, query expansion in content-based image retrieval (CBIR), and measuring similarity between different modalities such as an image and a piece of text). In addition, MINets framework is expected to provide the ability to perform robust inference (e.g., recognizing objects and activities in images or videos) in the presence of noise and uncertainty. In its simplest form, a MINet is a graph where nodes are either concepts (text) or data (such as images), and links are ontological/semantic relationships between concepts, attachment of images to concepts, and visual similarity measures between images. Construction of an experimental MINets framework involves crawling the Web for a particular domain, gathering images with associated text and exploiting natural language processing, computer vision and mining techniques in establishing the concepts, associated images, interconnecting links and an ontology that supports inference. Validation against current web search engines and CBIR techniques is expected to provide a proof-of-concept for the novel MINets framework. This interdisciplinary exploratory project is expected to yield general theoretical and algorithmic MINets framework that will provide new searching, mining and reasoning capabilities for multimedia data. It will help to define new research areas in effective utilization of multimedia information sources for cross-media and cross-conceptual knowledge discovery and analysis, large-scale annotation, information fusion and inference. Project results, including open source software, annotated corpora, scoring metrics will be disseminated via project Website (https://netfiles.uiuc.edu/qi4/www/MINets.htm). This project will provide research opportunities for graduate and undergraduate students.