CAREER: Multimodal and Multimedia Signal Processing

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

This research involves processing of multimodal signals, including speech, image, and video signals. The investigator studies such signal-processing techniques for multimedia applications, ranging from videoconferencing to immersive collaborative environments. While the research in this project covers several interdisciplinary topics, the result of the project is a platform for future research and education where concepts like collaboratories and virtual universities become reality. In this platform, the teacher can interact freely with the students and the students can collaborate remotely on projects. As a result of the interdisciplinary nature of this research, a new signal-processing curriculum is formed by the investigator, which emphasizes the interaction among several traditional courses including image processing, speech processing, image analysis, computer graphics, human-computer interaction, and communication networks. This research covers a number of interdisciplinary topics. In the area of media conversion and integration, the investigator studies audio-to-visual mapping, audio-visual speech recognition, and audio-visual person authentication. Audio-visual speech data are collected to support these tasks. In the area of image analysis and synthesis, the research focuses on facial feature tracking and face animation. In the area of geometry and texture processing of 3D objects, new algorithms for progressively representing, transmitting, indexing and retrieving 3D objects are studied. Integrating these technologies into a smart terminal that interacts with the user, and connecting these terminals to support interaction between users from anywhere at anytime, the investigator builds a networked collaborative environment. While much existing network research focuses on increasing the bandwidth and reducing the delay, this research goes one step further to add functionalities to the network to support remote collaboration. While there is much study done on human-computer interfaces, the investigator strives to put these together to create human-to-human interfaces. The goal of this research is to show that, with multimodal signal-processing technologies, it is possible to create an immersive environment in which the physical terminals and networks become invisible to the users, so that the users can concentrate on the task at hand.