CISE-MSI: RCBP-RF: HCC: Understanding Human Emotions Associated with Attention Activities in Temporal Domain Utilizing Multimodal Data

Bowie State University, Bowie MD

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Emotions impact human behavior and have considerable effect on cognitive function, as they are closely linked to learning, decision-making, and memory. Since the importance of recognizing emotions has been steadily growing, identifying them has been studied in domains such as robotics, audience understanding, and therapy. In clinical settings, especially with respect to neurodevelopmental disorders, emotion analysis has been studied in an effort to help individuals with attention-deficit/hyperactivity disorder (ADHD). Adults with ADHD have difficulty attending to important details, planning task completion, modulating responses, and complex learning. Understanding emotions and how they change over time in young adults with ADHD is not yet fully understood and is the focus of this research. Project outcomes will include. an interdisciplinary solution for understanding and adjusting emotional behavior to enhance human performance. Project outcomes will enable monitoring of emotion changes for various applications such as attention training, interventions in social-emotional adjustments, and classroom-based learning, and ultimately will contribute to the development of real-time emotion monitoring utilizing wearable sensor devices. Additional broad impact will derive from building research capacity at a minority serving institute, by preparing student researchers to collaborate with faculty on cutting edge research. This research involves designing a model that can predict emotions in the near future by examining the emotions associated with attention activities over time and analyzing multimodal data such as eye-tracking, electroencephalogram (EEG), skin temperature, galvanic skin response, and oxygen saturation. The project will investigate stream-based multimodal data acquisition using a scientific workflow system that leverages both metadata standards and visual programming. Next, it will collect data from young adults with ADHD to generate multimodal metrics during an emotionality analysis task and extract emotion-attention features associated with attention activities. Subsequently, the research will focus on generating a model to forecast emotions to enhance students' learning in data science. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.