EAGER: Spatial Audio Data Immersive Experience (SADIE)

Virginia Polytechnic Institute And State University, Blacksburg VA

Investigators

Abstract

Although there has been much recent interest in visualization to support data analysis, sonification -- the rendering of non-auditory information as sound -- represents a relatively unexplored but rich space that could map onto many data analysis problems, especially when the data has a natural spatial and temporal element. In contrast to headphone-based sonification approaches, this project will explore the potential of "exocentric" sound environments that completely encompass the user and allow them to interact with the sonified data by moving in space. The hypothesis is that compared to existing methods of data analysis, the coupling of spatial data with spatial representations, the naturalness of interacting with the data through motion, the leveraging of humans' ability to hear patterns and localize them in 3D, and the avoidance of artifacts introduced by headphone-based sonification strategies will all help people perceive patterns and causal relationships in data. To test this, the team will develop a set of primitives for mapping spatio-temporal data to sound parameters such as volume, pitch, and spectral filtering. They will refine these primitives through a series of increasingly complex data analysis experiments, including specific analysis tasks in the domain of geospace science. If successful, the work could have implications in a variety of applications, from enhancing visualizations to developing better virtual reality systems, while developing interdisciplinary bridges between scientific communities from music to computing to the physical sciences. The project will be developed using an immersive sound studio that includes motion tracking capabilities and a high-density loudspeaker array, driven by algorithms and open source sound libraries developed by the team to support embodied, rich exploration of sonified data that is not subject to audio deviations introduced by headphone-based strategies such as Head Related Transfer Functions. The specific sonification strategies for individual data streams will be based on the primitives described earlier, focusing on sounds rich in spectra that are easier for people to localize. Strategies for representing multiple data streams will include layering multiple non-masking sounds and combining streams to modulate different aspects of the same sound (e.g., pitch and volume). To develop and validate the strategies, the team will conduct a series of experiments that gradually increase the complexity of the analysis tasks: from basic ability to perceive and interpret single data primitives, to perceiving and inferring relationships between multiple data streams, to measuring subjects' ability to perceive known causes between multiple data streams in a series of geospatial model scenarios. In these studies the team will vary the strength of relationships in the data, the size of the parameter manipulations of sounds, and the pairing of different sounds and parameterizations in order to determine perceptual properties and limitations of sonficiation strategies (similar in some ways to perception-based foundations of visualization); they will also compare both analysis performance and qualitative reactions of participants using both the exocentric environment and a headphone-based egocentric environment as a control.