CHS: Small: An Analysis-and-Synthesis Framework for Small Group Conversations

University Of Houston, Houston TX

Investigators

Abstract

As one of the most common forms of human-human communication, group conversation is essential to our daily life. Humans can intrinsically manage and regulate a group conversation via the concerted effort of gaze, head movement, and hand gesture. However, research communities only gain limited knowledge and insight on its pattern and mechanism. Understanding and modeling its structure, mechanism, and interaction patterns can not only help to reveal the science of group conversation but also find numerical potential applications in science and engineering. Computationally modeling and generation of group conversations can provide a potential transformative paradigm to empower the group conversational capability on machines and engineering systems, including virtual agents, humanoid robots, and Internet of Things, so that they can follow real-world social norms while interacting with humans. Also, it can significantly facilitate the creation of more socially-engaging and life-like digital humans in education, simulation, and virtual world applications. Furthermore, this project will support the training of a diverse cohort of graduate and undergraduate students at the University of Houston. The project addresses the research question: whether and how we can computationally analyze small group conversations and further holistically synthesize life-like group conversational motion for a variety of potential applications? Most existing research works have approached this problem using aggregated methods of analysis on the computation of various average quantities or statistical measures. This project takes a different approach and addresses this problem by computationally modeling the group conversation behaviors and leverages this model to holistically synthesize life-like group conversational motion for a variety of potential applications. The research team will first acquire, in-house, a high quality, small group conversational motion dataset that includes acoustic speech, facial expression, gaze, head movement, hand gesture, and body movement. Based on this dataset, the project will work towards the objectives by focusing on three inter-related research thrusts. The first research thrust is to develop a computational analysis model to extract and compare meaningful gaze patterns for turn-taking and turn-keeping in small group conversations. With the insight obtained from the first thrust, the second research thrust is to design a holistic, deep learning based framework to efficiently generate or synthesize life-like small group conversational animations based on various user inputs. The third research thrust is to comprehensively evaluate the effectiveness and usefulness of the proposed framework via two selected applications: small group teleconferencing within virtual worlds, and crowd simulations with group conversations. The synthesis framework can potentially transform development of applications that deal with virtual human crowds such as the ones in shopping plazas, downtown streets, and airport waiting areas. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.