CAREER: Developing Contextual Cues for Just-in-Time Information Retrieval on Wearable Computers

Georgia Tech Research Corporation, Atlanta GA

Investigators

Abstract

The hypothesis underlying this project is that, with increasing interactivity and personalization of computing systems, just-in-time information retrieval agents (JITIRs) which proactively retrieve and present information based on a person's local context in an accessible yet non-intrusive manner, and which are implemented on wearable computers, will become a feature of the next stage of computing. To recover the user's context, the physical proximity of wearable computers to the user will be leveraged in designing perceptual systems that "see" and "hear" from the user's perspective. To these ends, the PI will create a Wearable Computer Workshop for rapid prototyping of new wearable sensors and systems. Although the mobile environment is extremely harsh for perceptual systems and algorithms that have been developed in the laboratory, systems that use standard video and audio for input will be preferred because they afford advantages such as automatic creation of new media databases for search, ease of annotation, availability of sensors, and applicability of results to other fields. The PI expects to adapt and exploit promising results from prior research in other domains, including face recognition algorithms, recognition of sign language gestures, hand movements involved in interacting with objects or performing tasks, gestures made in discourse, and user location systems. To support the desired model-based recognition systems, a new form of annotation derived from prior work in analyzing narration will be developed for the data collected during a user's everyday life. Ultimately, the PI will create a community of everyday users of his systems and perform studies to determine how access to JITIRs affects the wearer's use of knowledge. The PI expects this work to have broad impacts, including the development of radically new customizable wearable platforms with the ready ability to interface to new sensor technology, a sensor package directed at capturing a user's everyday life, prototype wearable face recognition and gesture recognition systems, and the expansion of traditional information retrieval algorithms to loosely defined multimedia query searches.