CRII: RI: Sub-mm 3D Scanning of Real-World Scenes with Active Multi-View Event Sensing

Northwestern University, Evanston IL

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). State-of-the-art 3D imaging methods are not able to measure all possible classes of objects at once and still need to be tailored to a specific application. This is one of the main reasons why 3D imaging is still not omnipresent in our society, and still solely trained experts with task-specific equipment are able to capture high-quality 3D models. This project seeks to build a “one fits all” system that has the potential to change this. The targeted system enables precise 3D measurements of complicated surfaces/scenes in today’s billion-dollar industries, such as virtual reality, industrial inspection, autonomous navigation, or medical imaging. Many of these industries routinely run into particularly challenging scenarios for 3D scanning systems. Moreover, a scene-independent and precise 3D sensing system can have many applications. The produced sets of high-quality 3D data can usher the next wave in vision-related artificial intelligence research, leading to algorithms with unprecedented detection quality, prediction accuracy, or navigation precision. Given the current dissimilation of related techniques in all sectors of our modern society, everyone can profit. The project is accompanied by a comprehensive education program incorporating 3D imaging principles in a curriculum for Chicago afterschool programs to introduce at-risk youth to basic concepts in optics, image processing, and electronics. The focus of this research is to solve a long-standing problem in Computer Vision: high-resolution active 3D scanning of scenes cluttered with objects of mixed specularity and polluted by undesirable light contributions such as ambient illumination or strong inter-reflections. Existing approaches for this challenging task deliver rather sobering results or rely on large training datasets or other extensive prior knowledge, such as the geometry and reflectance of objects in the scene. An easy and flexible solution that delivers high-quality data is of significant interest for researchers in the broader computer vision community. This research distills the past decade’s research of the investigator and his colleagues. It combines previous experience in active multi-view 3D imaging concepts for different object classes with the novel detection modality of biologically inspired event sensors (which operate on a fundamentally different principle than conventional sensors). By properly facilitating the existing tradeoffs in 3D imaging and event sensing, the team will develop theory, hardware, and algorithms that lead to a fundamentally new type of 3D camera. The developed technique significantly advances the state-of-the-art and our fundamental understanding of limits. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.