RI: Small: Modeling and Recognition of Landmarks and Urban Environments

University Of North Carolina At Chapel Hill, Chapel Hill NC

Investigators

Abstract

The goal of this project is to design a scalable and robust system for modeling and representing the spatiotemporal and semantic structure of large collections of partially geo-referenced imagery. Specifically, the project is aimed at Internet photo collections of images of famous landmarks and cities. The functionalities of the system include 3D reconstruction, browsing, summarization, location recognition, and scene segmentation. In addition, the system incorporates human-created annotations such as text and geo-tags, models scene illumination conditions, and supports incremental model updating using an incoming stream of images. This system is designed to take advantage of the redundancy inherent in community photo collections to achieve levels of robustness and scalability not attainable by existing geometric modeling approaches. The key technical innovation of the project is a novel data structure, the iconic scene graph that efficiently and compactly captures the perceptual, geometric, and semantic relationships between images in the collection. The key methodological insight of this project is that successful representation and recognition of landmarks requires the integration of statistical recognition and geometric reconstruction approaches. The project incorporates statistical inference into all components of the landmark modeling system, and includes a significant layer of high-level semantic functionality that is implemented using recognition techniques. Potential applications with societal impact include virtual tourism and navigation, security and surveillance, cultural heritage preservation, immersive environments and computer games, and movie special effects. Datasets and code produced in the course of the project will be made publicly available. The project includes a significant education component through undergraduate and graduate course development.