CPS: Small: Real-time spatial audio on the Internet of Things

Arizona State University, Scottsdale AZ

Investigators

Abstract

As the Internet of Things proliferates, many networked devices fill the home, the office, and public spaces, often with speakers for auditory response. This project strives to use such speakers to perceptually place auditory sounds in arbitrary virtual spaces. Towards this goal, the project studies networked software systems for auditory crosstalk cancellation. The research project investigates practical foundational issues regarding: (i) the design of a visual tracking-driven feedback system for calibration and utilization of sound propagation models, (ii) the design of sounds that optimally spatialize on the spatial audio system, and (iii) the design of a networked system infrastructure to generate spatialized sounds from a computationally heterogeneous variety of Internet of Things devices. To approach such issues, the project designs and characterizes a family of system methods that invoke human-in-the-loop calibration of acoustical models through augmented reality technologies and interfaces, as well as leveraging cloud-based virtual audio propagation simulations for real-time decision-making. The project evaluates designs and methods through user studies on a prototype testbed that includes simulation frameworks, prototyped wireless audio systems and integration with existing Internet of Things devices and environments. The research offers transformative capabilities to the Internet of Things by enabling audio-based spatial guidance systems and spatial personal assistance. Such systems would be beneficial in assistive technologies for visually impaired and memory impaired individuals. The spatial guidance could also assist first responders in navigating emergency personnel towards spatially tracked individuals in foreign environments. Finally, spatial audio systems could greatly enhance commercial Internet of Things capabilities, offering the ability to place responsive sounds to allow users to e.g., locate misplaced objects, respond to spatially positioned virtual assistants, and provide auditory feedback from speaker-less devices, e.g., books, electrical outlets, exercise equipment, etc. To promote broader utility, the project integrates findings in custom open source software frameworks. Towards broadening access to computing, the project exercises outreach activities, including integrating the research into a high school summer camp, university curricula, and web-based and/or augmented reality demonstrations of audio propagation models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.