Predictive Distributed Musical Performance

Indiana University, Bloomington IN

Investigators

Abstract

The problem of distributed musical performance (DMP) seeks to link two or more musicians in live performance through a computer network. While the necessary bandwidth is achievable under a variety of data representations and network conditions, past attempts have been largely defeated by network latency. Building on his past work in musical accompaniment systems that developed a core technology of score following, musical prediction, and audio synthesis, the PI will explore in this project a completely new approach to DMP, feasible using ordinary "household" fast network connections, in which the remotely played musical parts are synthesized by predicting the musical evolution. Within the context of non-improvisatory music, these predictions are based both on past history during the current performance (received with latency), and past performances of the same musical piece. The simplest version of a system implementing this approach would connect a duo of musicians playing MIDI-controlled instruments by transmitting only event times over the network and basing predictions on these. This basic idea can be extended in several important ways. For instance, the approach can be applied to arbitrary acoustic (non-electronic) instruments by performing real-time score-following on the audio signal and transmitting only the event times across the network; on the other end, these event times can drive a synthesis engine that reconstructs an audio recording of the remote part using phase-vocoding. The proposed methods extend naturally to small collections of instruments, and possibly even to an entire orchestra. The principal challenge facing the PI in this research is to develop a modeling and prediction framework in which DMP is feasible in a network environment with no special latency guarantee. The model must intelligently balance the competing desires of local ensemble (each musician wants to hear a synchronized performance), and global ensemble (all musicians must hear nearly the same performance if true collaboration is to occur). The primary outcome of this project will be a modeling framework enabling real-time prediction and projection of remote musical actions, leading to a realizable implementation of DMP. Broader Impacts: Successful implementation by the PI of his ideas will lead to a unique Internet application, potentially attracting a large global following, culminating in the dream of a global chamber music society. Such an application could have a profound impact on the lives of many amateur and professional musicians, enabling otherwise impossible collaborations, and allowing broad access to the transforming human endeavor of music-making in a time of scarce resources for the arts.