CAREER: Collaboratively Perceiving, Comprehending, and Projecting into the Future: Supporting Team Situational Awareness with Adaptive Multimodal Displays

Clemson University, Clemson SC

Investigators

Abstract

Especially in data-rich and rapidly changing environments, effective teams need to give members the information needed to develop awareness of their own, their teammates', and the overall team's current situation. However, attentional demands are high on such teams, raising questions of how to both monitor those attentional demands and develop systems that adaptively provide needed information not just through visual displays that are often overloaded, but through other senses including touch and sound. Most existing work on adaptive multimodal interfaces for situational awareness focuses on individuals; this project will address how to do this work for teams, using unmanned aerial vehicle (UAV) search and rescue as its primary domain. This includes developing conceptual models that connect individual and team-level situational awareness, algorithms that use eye gaze data to assess both situational awareness and workload in real-time, and multimodal display guidelines that adaptively present information to the most appropriate team members through the most effective modes. This work will fundamentally advance research on understanding and designing to support team interaction, leading to practical improvements in a variety of safety-critical domains. The project also has a significant educational component, providing research opportunities for both graduate and undergraduate students and conducting design activities aimed at outreach and broadening participation in STEM disciplines, including workstation design to support teams of people with disabilities in manufacturing contexts. The research work has two main thrusts. The first involves collecting baseline data through a study where pairs of novices are trained to carry out simulated UAV search and rescue tasks using a standard visually-focused interface; the team will collect situational awareness (SA) assessments using existing validated surveys, eye gaze data, and team interaction data and member characteristics. This data will be used to build two main models. The first is a model that relates team dynamics and individual member characteristics with levels of SA and performance, using qualitative analysis of recorded observational and audio data, along with focus group interviews with participants. The second is a quantitative model that attempts to predict SA using eye gaze data, using both a factor analysis of eye gaze data and Markovian models of how teams and their members transition their visual attention between interface elements and tasks to predict levels of SA. These models will support unobtrusive assessments of SA that avoid the interruptions imposed by existing surveys and are necessary for developing the adaptive multimodal interfaces that are the other main thrust of the project. This second thrust will use the models of attention and problematic tasks and contexts identified in the first study to iteratively develop a pilot suite of multimodal interfaces that combine visual, audio, and tactile information channels. These multimodal interfaces will be evaluated using a series of studies similar to the first set, with the goal of developing cost-benefit models for presentation modes and types of information that minimally interfere with teams' existing visual workload while still providing information that raises individual and team SA. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.