CRII: CHS: Enabling Behavior Sensing via the Cloud and its Application to Public Speaking

University Of Rochester, Rochester NY

Investigators

Abstract

Public speaking is a task that people often rank as their top fear; one consequence is that even after repeatedly practicing a presentation many find they end up speaking too hastily when standing before the audience. People often desire to improve their public speaking skills, but lack of resources and social stigma may impede their ability to obtain the personalized training they seek. The PI's objective in this project is to build on his prior work and establish a research program to develop a ubiquitously available (Cloud based) automated social sensing framework that can recognize and interpret human nonverbal data (including facial expressions, tone of voice, body language, etc.), and then present constructive feedback to its users where they want and when they want. Modeling of the full range of human nonverbal behavior remains a challenging endeavor. Using the 43 muscles in our face, we can produce 10,000 unique combinations of facial expressions; modalities such as vocal tone, body language, and elements of physiology add to the complexity. While computers can now recognize basic expressions such as smiling and frowning, the automated interpretation of an individual's intent remains an active area of exploration (e.g., a smiling customer does not necessarily indicate that s/he is satisfied). This research represents a step towards developing algorithms and implementing a practical framework that can capture and interpret nonverbal data while providing meaningful feedback in the context of public speaking. Project outcomes ultimately will transform the way social skills are adapted and learned, which will have a broad impact on people with social difficulties (e.g., those with Asperger's syndrome). Human nonverbal behaviors can be subtle, are often confusing, and may even appear contradictory. While computer algorithms are more reliable than people at sensing subtle human behavior objectively and consistently, human intelligence is currently far superior at interpreting contextual behavior. This research adopts an approach that couples computer algorithms with human intelligence towards automated sensing and interpretation of nonverbal behavior in nearly real time. The PI's approach is to develop a robust and scalable Web-based sensing framework that will automatically capture and analyze an individual?s behavior by exploiting the Cloud infrastructure, without requiring any major computational resources from the end-user. The work will include three phases: development of a Cloud-enabled sensing platform for automated recognition of nonverbal behavior; development of algorithms for combining the behavioral data with human judgment using the so-called wisdom of the crowd to generate meaningful insights, interpretations, and social recommendations; and running user centric iterative studies to validate the framework for the general public as well as practitioners. The work will also lead to core contributions in designing computer interfaces. And since behavioral modeling methods typically assume a large amount of naturalistic data, preferably collected in the wild; it is therefore noteworthy that the PI's sensing framework has the potential to collect one of the largest naturalistic nonverbal datasets.