RI: Small: RUI: Benchmarks and Algorithms for Mobile Image Matching

Middlebury College, Middlebury VT

Investigators

Abstract

This project will provide both new benchmarks and new algorithms for the mobile image matching domain. In order to drive and focus new research on mobile image matching, the existing Middlebury benchmarks will be augmented with new datasets of calibrated multiview and video sequences acquired with mobile devices, together with ground-truth geometry. The project will also contribute novel algorithmic approaches for robust and scalable image matching. Undergraduate students will be actively involved in all components of this project, in particular in the data acquisition and testing stages, as well as the authoring of online evaluation tools. The project will have several broader impacts. First, challenging benchmarks will serve as catalysts for new research. High-quality datasets are also useful beyond benchmarking in that they can aid algorithm design and enable learning approaches. Second, scalable and robust matching techniques tailored for mobile devices will enable a host of new applications with broad impacts on the population at large, including interactive 3D modeling of objects and people for social media, online commerce, and augmented and virtual reality. Finally, the project will expose undergraduates at a liberal-arts college in rural Vermont to the world of research, experimentation, and discovery. This project will augment the existing Middlebury datasets with calibrated multi-view and video sequences acquired with mobile devices from a robot arm, of challenging scenes with known geometry, derived using structured lighting. Datasets will include IMU data and flash/no-flash image pairs. The project will also explore novel evaluation metrics as well as the utilization of high-quality synthetic image sequences. A subset of the new datasets will be employed in new benchmarks for mobile 3D reconstructions tasks. The algorithmic work will contribute novel approaches for robust and scalable image matching. While the current trend in the community is to learn general models from large sets of labeled training data, this project will instead aim to learn data terms from the images at hand during the matching process. Such self-adjusting data terms will model radiometric and geometric distortions rather than being invariant to them. Another focus will be on memory-efficient approaches that avoid an exhaustive search of the full matching space while explicitly reasoning about occlusion, reflections, and transparency. Additional algorithmic techniques will include layer-based image matching algorithms, novel smoothness terms suitable for fast and scalable image matching, and novel strategies for dealing with completely textureless scenes.