I-Corps: Nurture Neurodiversity

Texas State University - San Marcos, San Marcos TX

Investigators

Abstract

The broader impact/commercial potential of this I-Corps project is the development of a software application that can be used by people on the autism spectrum to identify the emotions of their conversational partners. The software can be used by people from different cultural backgrounds. This tool for children on the autism spectrum may enable them to understand and learn how people express different emotions through facial, body gestures, and speech to improve their communication and interaction skills into adulthood. The proposed project combines the expertise of speech-language pathologists and engineers. The technology and its commercialization may benefit people on the autism spectrum, their family members, and their therapists. Social interactions can be challenging for people on the autism spectrum. This I-Corps project seeks to integrate machine and deep learning models in a single app that recognizes and provides emoticon feedback of emotions detected. The app will be utilized by children on the autism spectrum to understand and recognize emotional states with people they interact with as a developmental tool in recognizing emotions. The app enables the microphone and camera for the necessary visual and auditory inputs. The facial expression model will classify the emotion based on the facial features, which was trained in different lighting conditions and angles. The body-gesture model will recognize the sequence of body movements to classify the emotional expression of the person and the speech-emotion model will extract frequency features to classify the emotion presented by the tone of speech. The challenge is to assemble these models to provide an accurate overall emotion classification, integrate machine and deep learning models to the code development, and provide emoticon feedback to the child on the autism spectrum through the screen. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.