RI-Small: Real-Time Planning of a Conductable Orchestra

Indiana University, Bloomington IN

Investigators

Abstract

The goal of this research is to allow a (live) conductor to create a personal musical performance by controlling a computer-driven (virtual) orchestra through gesture captured on video. The immediate focus is to provide an educational tool for conducting students, or others with some serious musical training who are capable of communicating musical intent clearly through the traditional language of gesture used by conductors. However, the PI expects that less-schooled or novice users will also be able to learn from and find enjoyment with the outcome of this research, which will culminate in a computer system that runs on generic computer hardware, and will be made freely available. The system will take video of a conductor as input, reducing this input into a two-dimensional conducting trace that describes the movement of the tip of the conductor's baton over time. The system will perform real-time estimation of the conductor's precise "state" within the composition, using an approach that fuses hidden Markov model methodology with a Kalman filter model for musical timing. Using this on-line estimate, the system will predict the location of future musical events, thus addressing the inevitable issue of detection latency. Concurrently, the system will synthesize real-time audio to follow the conducted performance, using a previously recorded performance whose timing is continually warped using phase-vocoding. The initial focus of this work will be on musical timing rather than dynamics, articulation, etc., as this is the aspect of conducting that is most clearly communicated through motion and usually also that which affords the most expressive potential and sense of "ownership" of the performance. Educated musicians find surprising agreement when evaluating the accuracy with which a musician or ensemble follows a knowledgeable conductor, suggesting that the conductor's "signal" must be relatively unambiguous. Making mathematical sense of the relationship between this signal and its meaning constitutes a challenging dimension of this research. Broader Impacts: This work will have lasting impact on conducting pedagogy, by providing a tireless and responsive laboratory for musical experimentation. The research will also make contributions to instrumental and voice pedagogy, by allowing a musician to focus on the interpretive aspects of a piece without simultaneously addressing the technical challenges. The problem of planning the orchestra's musical evolution with uncertain and continually evolving knowledge of the conductor's actions is deeply challenging; thus, this work has implications for the general domain of planning under uncertainty. Perhaps most importantly for society at large, a successful conducting system would bring the pleasure of music-making to a broad and international collection of users who might otherwise have little or no experience creating music.