CAREER: Geometrically Coherent Image Interpretation

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

CAREER: Geometrically Coherent Image Interpretation PI: Alexei A. Efros Abstract Image interpretation, the ability to see and understand the three-dimensional world behind a two-dimensional image, goes to the very heart of the computer vision problem. The overall objective of this proposal is, given a single image, to automatically produce a coherent interpretation of the depicted scene. On one level, such interpretation should include opportunistically recognizing known objects (e.g. people, houses, cars, trees) and known materials (e.g. grass, sand, rock, foliage) as well as their rough positions and orientations within the scene. But more than that, the goal is to capture the overall "sense of the scene" even if we do not recognize some of its constituent parts. To address this extremely difficult task, the PI proposes a novel framework that aims to jointly model the elements that make up a scene within the geometric context of the 3D space that they occupy. Because none of the measured quantities in the image -- geometry, materials, objects and object parts, scene classes, camera pose, etc. -- are reliable in isolation, they must all be considered together, in a coherent way. Having the geometric context representation will allow all the elements of the image to be physically "placed" within this contextual frame and will permit reasoning between them and their 3D environment in a joint optimization framework. During the timeframe of this proposal, the PI will develop such a framework which will allow a geometrically coherent semantic interpretation of a image to emerge. Intellectual Merit: At the core of the proposal is an effort to unify two disjoint computer vision philosophies -- the traditional "Geometry" school that deals with 3D quantities like points and surfaces, and the newer "Appearance" school that operates in terms of 2D pixel patterns. These two views are here combined into one coherent framework, where appearance and geometry co-exist and rely on each other to jointly produce an interpretation of an image. Broader Impact: There are a number of important real-world problems that will benefit from the proposed research even during its development. Direct applications of this work include: developing navigation assistant technology for the visually impaired, scene awareness for mobile robots and car safety, and creating graphical 3D walk-through environments from a single image. URL: http://www.cs.cmu.edu/~efros/ImageInterpretation/