RI: Small: Advancing the Science of Generalizable and Personalizable Speech-Centered Self-Report Emotion Classifiers

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

The goal of the project is to create new and personalized speech emotion recognition approaches and to use these approaches to investigate how changes in emotion are related to changes in mental health. The first step is accurately measuring how a person’s emotions vary over the course of a day, a week, a month, or even a year. However, the only approaches currently available to do so involve actively asking a user how they feel multiple times per day. Users are often willing to do this over shorter periods of time, but over longer periods of time this can be quite taxing. Fortunately, speech data are often easy to capture and conveys information about emotion. However, most approaches in speech emotion recognition are not focused on how the user feels and instead are focused on predicting how an outside group of people would label that user’s feeling. The goal of the project is to refocus automatic emotion classification on the user themselves. In the future, this will allow us to easily collect information about a user’s emotion leading to new investigations into how changes in emotions are associated with risk factors for changes in health. The goal of the presented research objectives is to advance the state-of-the-art in robust and generalizable personalized speech (acoustics + language) self-report emotion recognition classifiers and to investigate how measures created using these classifiers will allow researchers to intuit changes in mental health symptom severity in a clinical population of individuals at risk for suicidality. The field of automatic speech emotion recognition is almost exclusively focused on estimating how an outside group of observers would perceive a given emotional display (i.e., perception-of-other). Yet, when the focus is on the ultimate use cases of this technology, e.g., mental health symptom severity tracking, this is often not what is needed. Instead, symptom severity tracking often needs information about how a given individual is interpreting their own emotional experiences (i.e., self-report). For example, changing patterns in self-report are associated with changes in depression severity. Yet, these changes are currently measurable only through active participation, in which individuals are regularly asked to describe their emotional experiences using self-report measures (e.g., Ecological Momentary Assessment, EMA) longitudinally, multiple times per day, which can be quite expensive both in terms of cost and participant burden. The project team envisions a future in which audio can be passively collected and used to automatically infer self-reported emotion, but there has been limited attention to the design of such classifiers due to persistent challenges associated with accurately estimating self-reported emotion, including cognitive bias, context, and the difference between self-report and emotional experiences. The project team will accomplish these goals by: 1) creating classifiers that are robust and generalizable using new metrics that encourage models to attend to the same acoustic and language cues as human observers; 2) personalizing classifiers to users longitudinally, and 3) evaluating the effectiveness of self-report emotion classifiers by predicting changes in mental health symptom severity using an existing real-world dataset annotated with mental health symptom severity (risk of suicide). The presented approaches will forward investigations into how to use passively collected audio data to estimate changes in risk factors for health changes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.