ITR/IM: Capturing, Coordinating and Remembering Human Experience

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

This work will develop algorithms and systems enabling people to query and communicate a synthesized record of human experiences derived from individual perspectives captured during selected personal and group activities. For this research, an experience is defined through what you see, what you hear, where you are, and associated sensor data and electronic communications. The research will transform this record into a meaningful, accessible information resource, available contemporaneously and retrospectively. We will validate our vision with two societally relevant applications: (1) providing memory aids as a personal prosthetic or behavioral monitor for the elderly; and (2) coordinating emergency response activity in disaster scenarios. This project assumes that within ten years technology will be capable of creating a continuously recorded, digital, high fidelity record of a person's activities and observations in video form. This research will prototype personal experience capture units to record audio, video, location and sensory data, and electronic communications. Each constituent unit captures, manages, secures and associates information from its unique point of view. Each operates as a portable, interoperable, information system, allowing search and retrieval by both its human operator and remote collaborating systems. An individual cannot see everything, nor remember everything that was seen or heard. The integration of multiple points of view provides more comprehensive coverage of an event, especially when coupled with support for vastly improving the memory from each perspective. The research thus enables the following technological advances: * Enhanced memory for individuals from an intelligent assistant using an automatically analyzed and fully indexed archive of captured personal experiences. * Coordination of distributed group activity, such as management of an emergency response team in a disaster relief situation, utilizing multiple synchronized streams of incoming observation data to construct a "collective experience." * Expertise synthesized across individuals and maintained over generations, retrieved and summarized on demand to enable example-based training and retrospective analysis. * Understanding of privacy, security and other societal implications of ubiquitous experience collection. The foundation for this work, the Informedia Digital Video Library, has demonstrated the successful application of speech, image, and natural language processing in automatically creating a rich, indexed, searchable multimedia information resource for broadcast-quality video. The proposed work builds from these technologies, moving well beyond a digital video library into new information spaces composed of unedited personal experience video augmented with additional sensory and position data. Tools will be created to analyze large amounts of continuously captured digital experience data in order to extract salient features, describe scenes and characterize events. The research will address summarization and collaboration of multiple simultaneous experiences integrated across time, space and people.