RI: Small: Depth from Differential Defocus

Harvard University, Cambridge MA

Investigators

Abstract

This project will explore a new class of depth sensors. The new sensors operate by observing small changes in optical defocus through a single lens, and they require very small amounts of digital computation. The distinguishing feature of these sensors is that they can be much smaller and lower power than existing depth sensor technologies. By enabling depth sensing capabilities on smaller platforms, they help accelerate the creation of smart micro-scale systems and an effective Internet of things. Depth sensors produce two-dimensional images where each pixel's value is the distance to a scene point along a corresponding ray. A variety of these sensors exist, and they are already fueling advances in autonomous navigation, gesture-driven interfaces, robotics, and more. This research will develop sensors based on a new visual cue called differential defocus. Like the well-known passive depth cues of stereo and depth-from-defocus, this new cue avoids spending power on broadcasting light. But unlike the existing passive cues, it calculates depth using simple analytic expressions that are easy to compute. To establish differential defocus as a new way to sense depth, this project aims to discover a complete stack of knowledge, from mathematical foundations to algorithms and hardware prototypes. The mathematical foundations include a catalog of depth constraints that correspond to many forms of differential defocus, such as differential camera motion, sensor motion, change of focal length, and change of aperture. At the hardware level, the project will pursue both single-shot and multi-shot designs that incorporate deformable lenses and customized photosensors. Algorithmically, the project will explore methods for using back-propagation to fine-tune the parameters of depth computations. Going further, it will explore back-propagation into the optical dimension, in order to enable the optimization of optical and computational parameters together, in a synergistic manner.