Computational auditory scene analysis as causal inference

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

Just by listening, humans can infer many details about the world around them: what someone said, whether a window in their house is open or shut, or what their child dropped on the floor in the next room. These everyday (but essential) judgments usually require us to separate the distinct causes in the world that generate sound. We hear multiple people talking at once, but can attend to the one we are interested in. We can tell whether a sound was produced in a large or small room, or an empty or furnished apartment, but can also identify what the sound was. And if an object is dropped on a table, we can usually tell the object's approximate weight but also the material the table is made of, just by listening. These abilities are critical to our interactions with the world and will be critical to reproduce in machine hearing systems for robots, automobiles, and other technologies. Here the investigators propose to investigate human abilities to decompose sound into its constituent causes and to build machine systems that can replicate these abilities. The proposed work will jointly pursue two goals. First, the investigators will build models of how sound is generated in the world. This aspect of the work will combine insights from physics and acoustics with empirical measurements of sound, focusing on how forces imparted to objects resonate within the object to yield sound, and on how the resulting sound is altered by reflections off of environmental surfaces on its way to a listener's ears. Second, the investigators will develop a computational framework to infer the most likely explanation of a sound in terms of the events in the world that could have generated it. This aspect of the work will leverage recent advances in artificial intelligence research that render such inferences newly tractable. The resulting machine hearing systems will be compared with human listeners in a series of experiments, with the goal of improving the models of sound generation and the inference algorithms in order to match human auditory abilities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.