CAREER: Exploring Creative Expression through Music and Audio Technology

Drexel University, Philadelphia PA

Investigators

Abstract

Music contributes substantially to every culture on Earth, and the enjoyment of music is nearly universal. The appeal of music is that it provides expression to our own emotions, and it follows logically that individuals would want some creative input to that expression. Historically the primary exposure to music has been through live performance, providing audiences an opportunity for interaction with the musicians and music, but today the vast majority of music is experienced through recordings. And though recent digital audio technologies have had a tremendous impact on the world of recorded music, its fundamental nature remains unchanged: once a recording is made, that single "performance" is forever fixed, preventing any true interaction with the listener. The proposed work integrates research in digital audio technology with educational activities under a common vision of transforming the act of listening to "recorded" music into an interactive experience in which the "performance" responds to the creative input of the listener. Central to this vision is a concept known as Structured Audio, a semantic object-based representation of sound. The specific work activity consists of the following efforts: . Structured Sound Modeling (SSM) of musical instruments, which uses a novel signal processing and machine learning framework to facilitate a greater degree of musically expressive control than can be achieved with existing models for sound synthesis. . Development of the Structured Audio Platform (SAP), consisting of devices and software that employ these new instrument models to provide an expanded artistic palette for music producers and interfaces enabling nonmusicians to interact with their music by controlling creative aspects of the "performance." An outreach program that uses music technology to attract students to science, technology, engineering, and mathematics (STEM) by demonstrating the contributions of these disciplines to modern music production as well as the creativity inherent within STEM and related fields. With respect to intellectual merit, there are many methods of modeling musical instrument sounds, but most are limited in their ability to capture the full expressive range of those instruments. The project will employ a novel framework for modeling time-varying sounds using jointly estimated parameters of an excitation-resonance model to derive a HiddenMarkovModel (HMM) of expressive sound. This framework employs an analysis/synthesis scheme: input sound is parameterized by the model and can be regenerated from the model parameters. This process has been used to model the singing voice and can be applied to other instruments, resulting in high sound quality while retaining a full range of musical expression. Knowledge gained through this project will also benefit other machine listening applications, such as audio identification and sound sensing. The research activity includes the development of a custom platform of devices and interfaces that offer artists new creative options and allow non-musicians to also provide creative input to their music. A key area of research will investigate appropriate mappings from the expressive controls of SSM instruments to a simple, low-dimensional interface for musical novices. Creativity is a difficult subject to study quantitatively, but this project presents a framework from which a small subset of creative expression can be assessed and analyzed. The educational activities of this project are designed to broadly impact students at many levels. The proposed outreach program will introduce high school students, particularly from underrepresented groups, to the technology of digital music, revealing the influence of science and engineering on music performance, recording, and listening. By engaging students' affinity for popular music the program seeks to foster curiosity in science and technology. The research component of the proposal will also be directly integrated into this program via custom music devices that allow students to creatively interact with music in Structured Audio format. This confluence of engineering and art has the potential to incubate interest in both fields. Given the diminishing support for music education in the public school system, this multi-disciplinary approach may be a solution to sustaining creative arts in the curriculum. The proposed research integrates knowledge from a wide range of areas. Participating graduate students will receive interdisciplinary training in fields such as acoustics, music theory, signal processing, performance practice, and machine learning.