CHS: SMALL: Collaborative Research: APERTURE: Augmented Reality and Physio-Enhanced Robotic Gesture

University Of Colorado At Boulder, Boulder CO

Investigators

Abstract

How can a robot choose the "best" modality for drawing the attention of a human and communicating the needed information? This is the central theme of the project with the assumption that the human team mates of the robots use augmented reality (AR) based visualization. Specifically, this project plans the following three major research activities for enabling robots to communicate with human teammates in a way that is tailored to their teammates' current mental states: exploring how augmented reality technologies (such as the Microsoft Hololens) can be used to provide robots with new ways to communicate about objects, locations, and people in their environments, especially when used together with communication through spoken language; examining how technologies can non-invasively measure different aspects of teammates' mental states, including how much mental workload, perceptual workload, stress, and frustration they are experiencing; and determining how robots can choose the best way to communicate with their human teammates when the two technologies are combined, (for example, through language alone, AR visualizations alone, or both used together), based on those teammates' individual mental states. This system will then be used to test how it might improve the safety and productivity of underground workers, by allowing robots to communicate in a way that is more effective and less cognitively demanding. While the researchers will be investigating the effectiveness of these integrated technologies specifically within underground work environments, the research will also be applicable to a wide variety of areas, including eldercare, urban search-and-rescue, and space robotics, and will have broad scientific impact across both computer science and cognitive science. The above goals will be achieved through APERTURE, a novel framework integrating head-mounted augmented reality displays, a multimodal suite of noninvasive, lightweight, and field-ready physiological sensors (such as functional near-infrared spectroscopy (fNIRS), Electroencaphalography (EEG), Electrodermal Activity, Electrocardiogram (ECG), and Respiration sensors), and unmanned ground robots, within a cognitive robotic architecture. APERTURE will be built by integrating the Distributed Integrated Affect Reflection Cognition (DIARC) architecture with these robotic, augmented reality, and physiological hardware elements. The project will design and evaluate physiological sensing models, augmented reality gestural cues, and machine learning models for selecting between AR gestural cues based on neurophysiological data. The designed machine learning models will classify users' cognitive and affective states from this sensor data, and help the robots understand when and how to communicate based on users' cognitive and affective states. The novel AR approach to deictic gesture will help robots pick out the objects they are referring to through the use of visualizations displayed in their teammates' augmented reality headsets. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.