CAREER: Advanced Knowledge Extraction of Affective Behaviors During Natural Human Interaction

University Of Texas At Dallas, Richardson TX

Investigators

Abstract

Identifying and characterizing emotional behaviors are challenging but very important research topics for enriched speech-derived analytics and human-computer interaction. This CAREER project aims to create novel algorithms to recognize spontaneous affective behaviors from speech that capture the underlying externalization process of emotions and generalize to recordings of human interactions collected under real-world conditions. The lack of generalization of current speech emotion algorithms to recognize expressive behaviors during natural human interaction is the key barrier to deploying affective-aware technology in real-life applications. Under a theoretical framework grounded in the nonuniform externalization of expressive behaviors, the project brings transformative solutions to address this problem. The proposed models and algorithms promise insights to explore and extend theories in linguistic
 and paralinguistic human behaviors. Several new scientific avenues can emerge that serve as truly innovative advancements that will impact applications in security and defense, next generation of advanced user interfaces, health behavior informatics, and education. The role of human centered technologies, especially contextualized in applications of direct societal relevance, can inspire young 
scholars into computing and engineering: from creating robust technologies for sensing, to 
actually incorporating such information as a part of advanced analytics and enhanced user experiences. As a Hispanic faculty, the PI serves as a mentor and role model for high school, undergraduate 
and graduate students involved in the Minority Scholars Symposium, Diversity Scholarship Program and Graduate Student Mentoring Program at the University of Texas at Dallas. Through lab open houses, demonstrations, and active online and social media presence, the PI is reaching out to non-traditional students, as well 
as the broader, non-technical audience interested in human behavior science. The project evaluates 
the powerful, scalable and appealing concept of using neutral reference models to contrast deviations in speech characteristics associated with emotions. The study proposes flexible, integrative and discriminative frameworks that capture the underlying encoding process of expressive behaviors including of emotion salient regions in the speech stream, intrinsic reliability of features,
 and dynamic evolution of emotions. The study considers binary and rank-based classifiers to recognize and rank-order specific expressive behaviors. The project presents speaker and lexical compensation schemes, and model adaptation strategies to increase the robustness of the proposed models. All these theoretical and algorithmic advances are carefully evaluated with naturalistic data, in which emotional content will be annotated 
with a novel crowdsourcing scheme that tracks in real time the performance of the evaluators.