Data-Driven Appearance Transfer for Realistic Image Synthesis

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Realistic image synthesis is a central goal of computer graphics. Major recent advances have allowed researchers to model a wide spectrum of complicated visual phenomena with a very high degree of realism. Yet, even the best computer-generated feature films are a far cry from what one might consider "real". Curiously, the problem is generally not with computer graphics being unable to model the physics of the everyday visual world -- the problem is with the world itself! It's just too complex, too noisy, too rich and vivid to be recreated from scratch by even the most skilled and patient artist. One solution is to use image-based methods and directly capture visual appearance of everything in the world -- if only it was feasible. Instead, this research effort centers on transferring appearance from a large database of stored visual data into a novel scene. The reason is that while capturing details of a particular scene is very expensive and time-consuming, obtaining similar information from some relevant scene is relatively easy. There is a tremendous amount of visual data that is already captured and available - thousands of webcams all over the world, millions of photographs placed on the Internet, depicting anything from sandstorms in Sahara to the glaciers in Alaska. And more data is being added every day. Our research is developing a unified approach for appearance transfer. Two broad scenarios are considered: transfer in image stacks (e.g. webcams) and single image transfer. In both cases, the major research issues involve: (1) grouping images and image stacks into regions with coherent material/geometry properties, (2) determining correspondence between various groups in the scene and the database, (3) and finally transferring the correct appearance from the database by combining it with the large-scale structure of the input scene.