EAGER: Exploring the Use of Synthetic Speech as Reference Model to Detect Salient Emotional Segments in Speech

University Of Texas At Dallas, Richardson TX

Investigators

Abstract

This EArly Grant for Exploratory Research aims to create neutral reference model from synthetic speech to contrast the emotional content of a speech signal. Emotional understanding is a crucial skill in human communication. For this reason, modeling and recognizing emotions is essential in the design and implementation of interfaces that are more in tune with the user's needs. Starting from the premise that paralinguistic information is non-uniformly conveyed across time, this study aims to identify emotionally prominent regions or focal points across various acoustic features. The study explores a novel approach based on synthetic speech to build reference models characterizing patterns observed in neutral speech. These reference models are used to contrast the emotional information observed in localized segments of a speech signal. The study builds a synthetic speech signal that conveys the same lexical information and is timely aligned with the target sentence in the database. Since it is expected that a single synthetic speech will not capture the full range of variability observed in neutral speech, the study explores approaches to produce different neutral synthetic realizations. After creating a parallel corpus with time-aligned synthetic speech, the study explores how well synthetic speech captures the acoustic patterns and emotional percepts of neutral, nonemotional speech. Then, a target signal from the database is compared with the properties observed across the family of synthesized signals. The study presents a novel approach to build a robust emotion recognition system that exploits the underlying nonuniform externalization process of expressive behaviors. Algorithms that able to identify localized emotional segments have the potential to shift the current approaches used in the area of affective computing. Instead of recognizing the emotional content of pre-segmented sentences, the problem is formulated as a detection paradigm, which is appealing from an application perspective. These advances represent a transformative breakthrough in the area of behavioral analysis and affective computing. The proposed models and algorithms provide numerous insights to explore and extend theories in linguistic and paralinguistic human behavior. Having established the base infrastructure for this exploratory research, several new scientific avenues will emerge that serve as truly innovative advancements that will impact applications in security and defense, next generation of advanced user interfaces, health informatics, and education. Furthermore, the scientific methods are enriching venues for interdisciplinary training and mentoring for undergraduate and graduate students.