SGER: Social-Emotional Intelligence Prosthetic

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

This research will develop the first known wearable device capable of perceiving and reporting on social-emotional information in real-time human interaction. Using a small wearable camera and customized video and pattern analysis algorithms, the system will automatically read and interpret the facial expressions and head movements of the person the wearer is interacting with and will communicate its inferences to the wearer via sound or tactile feedback. The system builds on and extends the computational model of "mind-reading" developed by Rana el Kaliouby, which forms the basis of an automated system for the inference of mental states from head and facial displays in a video stream in real time. Bringing together and expanding upon recent advances and methodologies in three key scientific and technical areas - affective computing, wearable computing, and real time machine perception - the project will significantly advance the nascent ability of machines to infer cognitive-affective states in real-time from nonverbal expressions of people engaged in a range of natural interactions. The proposed work is for the first prototype of what is expected to become a sequence of increasingly intelligent prosthetics. The design and strategy will open up new lines of inquiry in understanding the role of social-emotional cues in natural interaction. The automated perception of nonverbal cues and inference of mental states are problems that continue to challenge researchers in the domains of machine vision and machine learning. These problems become even more complex if the system is required to execute in real-time and to generalize to novel situations and people, a necessary requirement in the case of an assistive technology. The project addresses open research questions pertaining to whether machine perception can be reliable enough and comfortable enough to augment social interactions in a way that improves human to human communication. The challenge is further complicated by the requirement of operating in a mobile environment with varied lighting conditions and limited processing and power. Being the first prosthetic assistive technology that is designed to augment a person's impairment in socio-emotional intelligence skills, the project has the potential to lead to a line of new research. A system with social-emotional perceptual abilities can potentially assist the growing number of individuals diagnosed with Autism Spectrum Disorder, in perceiving communication in a natural rather than a structured environment, bootstrapping their ability to learn and develop in social settings. The system will also provide a new tool for researchers on collaborative systems to more carefully examine the communication of nonverbal cues in natural everyday communication. The results of this interdisciplinary work can be leveraged in human-computer interaction, wearable computing, robotics and in future technologies with social-emotional intelligence. Results will be distributed in all of these communities through peer-reviewed publications and conference presentations, as well as through forums for attracting more minorities to scientific and technical fields. This work promotes the training and education of students by deeply involving them in the proposed research activities, with a commitment to growing the participation of underrepresented minorities in cutting-edge scientific research