HCC: Small: Multi Modal Music Intelligence for Robotic Musicianship

Georgia Tech Research Corporation, Atlanta GA

Investigators

Abstract

The PI's objective in this project is to advance the field of robotic musicianship by enabling multi-modal communication between human and robotic musicians, and by embedding knowledge-based musical intelligence in real-time musical interactions. To these ends the PI will seek to understand how temporal structures in music are represented and processed by humans, and he will develop a vision system that enables a robotic musician to anticipate a human musician's gestures. Modeled after human-human musical interaction, the artificial vision system will complement the musical listening system developed in the PI's prior work, thereby allowing a robot to better synchronize its playing with human improvisers. In addition, the PI will transcribe and analyze (using statistical tools such as Hidden Markov Models) a large corpus of works by the great classical composers as well as by masters of jazz; the off-line analysis, coupled with interviews, surveys, and focus groups, will advance our knowledge regarding cognitive and mechanical aspects of group play and the role of visual and physical cues in live performance, and will enable the PI to construct a long-term cultural knowledge base that will be added to the short-term real-time improvisation techniques already developed previously. Collectively, project outcomes will lead to a comprehensive model of human and artificial musicianship, which the PI will evaluate using behavioral measures (such as time differences in synchronization), subject questionnaires, and a Turing-like test to evaluate the quality of the musical interaction. Broader Impacts: This project will make fundamental contribution to our knowledge in areas such as musicianship, human-robotic interaction, computer assisted collaboration, and improvisation. It will help bring human-robot interaction to the general public through high visibility concerts that capture the interest and imagination of students who are not regularly drawn to math, the sciences, or engineering by creating novel musical collaborations between humans and machines. The project will serve as a testbed for future forms of musical interactions, bringing perceptual and algorithmic aspects of computer music into the physical world both visually and acoustically, which may inspire people to play and think about music in new ways. Ultimately, the research is expected to shed light on broader concepts such as human and artificial creativity and expression, and the feasibility for machines to create, or assist in creating, meritorious aesthetic and artistic products.