ITR/SY: Augmented Cognition: Combining Human and Digital Memory

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

The PI will design and build human-computer interfaces that improve human memory, which he terms infocockpits. The basic approach is to take well-understood psychology principles and apply them to the design of information displays, in particular the fact that human beings are adept at remembering information based on its location relative to their body and on the place where they were when they learned it. The implementations will use two basic strategies: multiple spatial displays surrounding the user, to engage human memory for location; and ambient context displays (both visual and auditory), to engage human memory for place. This work leverages prior collaborative efforts by the PI and Co-PI in virtual reality. The project relates to, and represents a new paradigm for, the retention of information instead of its manipulation. Although the PIs postulate just two fundamental design principles, the design space is very large. The basic research apparatus - the infocockpit - will be a computer system with a number of traditional display monitors arrayed relative to the user's body. These display screens are then placed in a room where images can be projected onto the walls. This projected imagery, plus ambient 3D surround sound, creates a distinctive place in which information is viewed. The PI will systematically vary the configuration, and will run controlled experiments, in which users access information and then later are tested for their ability to recall it, to examine the benefits of: multiple monitors arrayed around the user vs. a single monitor; the addition of projected background context; stationary vs. animated background contexts; having the context semantically related to the accessed information; having ambient and/or localized sound as part of the context; and using hierarchical places to avoid having confusions between many different contexts. The PI will partner with the Virginia Center for Digital History which will employ the new design principles to build and provide content for infocockpit systems for teaching American history; this will allow the PI to observe how the new techniques work in a real world educational application developed by others. A preliminary study conducted by the PI has found a 63% increase in users' memory capabilities; moreover, functional brain imaging assessment of the participants from this study and found that experience in the infocockpit resulted in brain activations in areas associated with spatial representation, working memory, and visualization. Thus, the question is no longer whether this approach can improve the user's ability to remember information; the question is by how much.