CHS: Small: TICE - Telematic Immersive Classroom Environments

Rensselaer Polytechnic Institute, Troy NY

Investigators

Abstract

Since their inception, telecommunication systems have targeted the exchange between exactly two people. This holds true for the telegraph, the telephone, and typical video conference systems. Meetings for larger groups of people are usually conducted using the same personal interfaces – take, for example, a telephone conference call. This practice contrasts with a traditional group meeting, for example, or a classroom scenario. In the latter, students and teachers can communicate with each other while having an awareness of the spatial location of each participant and the ability to identify each speaker easily. The goal of this project is to develop technology that enables two distant classrooms to be joined by telecommunication devices without the need for personal communication interfaces. An electronically adjustable microphone array will be designed that can virtually focus on an active speaker. This way, echoes that would otherwise occur in a collaborative telecommunication scenario with loudspeakers will be avoided. The system will be designed such that the sounds of a speaker will emanate from the position at which they are seen on a live video feed. Project outcomes are expected to have broad impact by providing new educational opportunities for classroom students by virtually connecting them to other classrooms for collaborative learning experiences. This project will develop a spatially correct, long-distance audio connection between collaborative spaces, for example two classrooms with groups of people at each site. The missing link needed to enable such remote collaboration is an audio tracking system that can follow all participants on both sides of the connection. The approach will use two 16-channel spherical microphones as the main audio tracking devices. The key idea is that each spherical microphone will receive continuous information from additional body-worn microphones, which is utilized to ensure echo-free bidirectional communication. The body-worn microphone signals will inform the tracking system who among the participants is speaking at a given time interval and in a given frequency range. The spherical microphone is then used to track each sound source during the time intervals that the source is active and in those frequency bands that are not disturbed by other sources. Using machine-learning tools, including Bayesian methods, Kalman filters, and Deep Neural Networks, the tracking system will implement a joint bottom-up/top-down architecture. Two existing sites are equipped with immersive loudspeaker systems, each of which will enable accurate local reproduction of spatialized audio from the remote site. These sites are also equipped with collaborative virtual reality systems with immersive video capabilities, and they will serve as laboratories for this project, enabling a system to transmit bidirectional audio/video streams with spatially congruent audio/video images. As an extensions of the project, the audio-tracking system will be paired with an existing 6-camera tracking system that is mounted on the laboratory ceiling and technologies will be developed to replace the body-worn microphones with virtual microphones using the beam-forming capabilities of a spherical microphone array. In another extension, visual gestures of participants will be considered to further facilitate the telematic experience. An important component of this research will be the development of an affordable and deployable version of the collaborative virtual reality system that can be easily used in schools, for example as a ceiling-mounted system hosted in a school's gymnasium. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.