CAREER: Simulating Nonlinear Audiovisual Dynamics for Simulated Environments and Interactive Applications

Columbia University, New York NY

Investigators

Abstract

Computer simulated virtual dynamics are increasingly recognized as important tools for creating immersive simulated environments for understanding physical systems that are too costly or perilous to investigate in real experiments. Most simulation techniques, however, are still inherently silent; they encompass only the visual modality, resulting in a limited sensory depiction of phenomena which is in sharp contrast to the real world where we are constantly immersed in rich audiovisual experiences involving both scene illumination and auditory cues. Consequently, most computer animations, regardless of how efficiently they are simulated, currently rely on manual editing to add sounds as afterthoughts, and many computational design tools are still appearance-oriented, modeling only geometry or visible motion. In this project the PI's central objective is to build efficient models for simulating nonlinear audiovisual dynamics in a realistic and synchronized way. This is a highly challenging task, as the simulation models need to faithfully resolve visible and audible details across a wide range of spatial and temporal scales because multiple scales of motion are often coupled together to produce perceivable nonlinear sound effects such as pitch shift and mode locking. The PI will address these challenges in three stages. First, he will create efficient multi-scale models which adaptively exploit underlying dynamical structures. He will then systematically evaluate these models using ground-truth data, experimental statistics and user studies. Finally, building upon his new models, he will develop tools to enable interactive parameter exploration of desired audiovisual effects. This work will seek to answer two fundamental questions: what needs to be faithfully simulated in order to produce audiovisual virtual dynamics, and how to perform the simulation at minimal computational cost? The PI seeks to develop principled new techniques for complex nonlinear audiovisual simulations, which will lay the groundwork for building immersive simulated environments and interactive applications, and also for connecting to other areas such as computational design. At a technical level, the PI's approach will bridge the longstanding gap between theoretical tools (including multi-scale analysis and dynamical system analysis) and practical computational schemes. The PI argues that by decomposing the simulated dynamics into visible and audible scales, efficient simulation models adapted to individual scales can be built. This scheme can yield methods with high efficiency while retaining simulation fidelity. It also poses a new challenge: to develop adaptive simulation models across a wide range of audiovisual scales. The PI's approach will address this challenge by exploiting dynamical system analysis at individual scales and linking the resulting models together. If successful, this research will enable new simulated environment applications with fully synchronized audiovisual effects, and lead to new ways of creating and editing multimedia content.