CHS:Small: Improved Cross-Subject Cognitive and Emotional State Classification Using Functional Near-Infrared Spectroscopy Data for Deep Learning

Syracuse University, Syracuse NY

Investigators

Abstract

New advances in bio-technology suggest devices to wear and measure the brain will be available to support many different activities. Future technologies may use brain activity data to adapt or customize educational software in real-time. Activity support based on interpreted brain activity could be used to reduce mental workload, modify emotional states, or help someone with post-traumatic stress disorder. However, brain activity data is complex and difficult to interpret. This project will use deep machine learning methods to overcome the challenge of classifying and interpreting brain activity data using real-time data from participants. The objective is to harness the tremendous potential of cognitive sensors and computational methods to help individuals function more effectively. Although many early successes were achieved using machine learning on brain data, several notable challenges have arisen, which significantly limit the impacts of these early successes. The technical approach in this project has three research thrusts. The investigators will develop models specifically for use on high density functional-near infrared spectroscopy (fNIRS) data. Thrust 1 involves the development of advanced deep learning techniques that are particularly well-suited for fNIRS data, and address spatial and temporal inter-relations. Thrust 2 involves development and adaptation of algorithm transparency (AT) techniques that are well-suited to shed light on brain dynamics embedded within the deep learning model structures. This will help the research team interpret the underlying structure of the models, with respect to brain spatial and temporal dynamics at the individual and group level. Thrust 3 collates the model and AT techniques developed in the prior thrusts and evaluates them using an extensive cross-subject and cross-participant fNIRS dataset. Using this data for evaluation purposes, the research team will work together to interpret results to improve upon classifier performance and model generalizability. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.