CAREER: Large-scale Appearance Modeling

College Of William And Mary, Williamsburg VA

Investigators

Abstract

The visual appearance of the world around us is the result of complex light interactions between different surfaces and material properties that comprise a scene. Despite staggering advances in data-driven appearance modeling, the creation of accurate models of large environments remains an open problem. The reliance of most current appearance modeling methods on active lighting to probe different slices of a scene's appearance precludes their use in environments where there is limited or no control over the incident ambient lighting. Furthermore, to facilitate calibration, many appearance modeling techniques estimate the appearance of a scene from a fixed vantage point, excluding scenes too large to fit in a single view with sufficient detail. In this research, the PI will investigate two novel appearance modeling paradigms designed expressly for large-scale environments under uncontrolled ambient lighting: appearance-from-motion and appearance-by-similarity. The former exploits relations between observations from different viewpoints to infer the full reflectance behavior, while the latter seeks to identify the best match from a library of pre-existing appearance instances to a possibly under-constrained set of observations. To support these two paradigms, a novel appearance model will be developed that builds upon our intuitions regarding scene appearance. The work will focus on two common types of input: community photo-collections and targeted video sequences. Broader Impacts: This research will pave the way towards practical techniques for in-situ appearance modeling of large-scale environments, while stimulating new research in computer vision and in data-driven appearance modeling in computer graphics by answering fundamental questions as to whether we can model appearance from motion and/or by exploiting similarity. The project will have far-reaching impact not only on computer science but also on diverse fields ranging from metropolitan planning to cultural heritage to entertainment. The ability to model existing environments will be beneficial to various security and safety training programs (for example, virtual fire drill simulations of existing buildings and sites could help train and prepare firefighters and first responders). The emerging field of virtual reality therapy will also benefit from this research, by making it easier to create digital models of large-scale environments (so that, for example, patients who have suffered a stroke can practice motor rehabilitation skills in virtual reproductions of environments they encounter in their daily lives, while autistic children can train to improve their social interactions in virtual reproductions of places such as classrooms which they encounter in their daily lives).