NSF-BSF: RI: Small: Enriching Scene Understanding using Computational Visual Vibrometry

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Vibrations are everywhere - often in ways too subtle or too fast for human eyes or regular cameras to detect. Floors tremble as people walk, engines hum with motion, air conditioners send ripples of vibrations through the walls, and even human voices or the beat of hearts cause tiny vibrations. These vibrations, though nearly imperceptible, carry a wealth of information about the physical world. By studying these vibrations, it is possible to learn surprising things: how full an opaque liquid container is, whether a machine is malfunctioning, or even where someone is walking in another room. Vibrations can reveal an object’s physical traits - like its weight, stiffness, or internal structure - and can even help systems understand what’s happening in places that are not directly in the line of sight. Today, however, capturing this kind of information is extremely difficult. Most vibration sensing relies on expensive, specialized equipment that can only measure one point at a time. And even if it were possible to gather all that data, making sense of it - especially in real time - is a significant challenge. This project develops technologies for sensing and understanding vibrations. The potential applications of the research are vast and impactful. Firefighters could locate people trapped in buildings. Engineers could detect structural weaknesses in aircraft bodies. First responders could remotely monitor vital signs in disaster zones. Safety inspectors can detect leakage in industrial containers holding hazardous materials from a safe distance. Beyond the lab, the project also focuses on education and outreach. Through hands-on experiences and creative programs, the team is making this cutting-edge science accessible to students and the broader public - helping everyone see (and feel) the invisible world of vibrations. This research project aims to significantly improve sensing and interpretation of vibrations by bringing together three ideas: (i) The research team is developing low-cost “visual-vibration” cameras that use everyday components - like consumer-grade cameras and lasers - to detect vibrations across space and time in complex everyday environments. (ii) By combining physics-based models with modern deep learning models, the researchers are creating algorithms that can detect hidden flaws in objects, estimate how much liquid is inside a sealed bottle, or even track activity in a room that’s out of sight. (iii) The system must not only rely on existing scene vibrations but can also send out controlled vibrations to “probe” the environment, helping it to improve the accuracy and efficiency of the extracted information. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.