CHS: Small: Appearance Modeling by Synthesis

College Of William And Mary, Williamsburg VA

Investigators

Abstract

Appearance modeling aims to create digital representations of materials, ranging from imaginary materials drawn by artists to reproductions of physical materials. To date, the most successful methods for creating realistic digital materials have been data-driven, where data captured from a physical material sample is used to reconstruct a digital representation. Most prior work assumes that the measurements fully constrain the reconstruction process. This project will explore a new paradigm that views appearance modeling as a constrained synthesis process instead of a reconstruction / interpolation process. This change in paradigm requires a rethinking of fundamental appearance modeling assumptions. First, constrained synthesis produces a distribution of possible solutions rather than a single most likely solution as in classic appearance modeling methods. This creates opportunities for users to participate in the appearance modeling process (for example, by selecting the "best"' solution according to subjective or artistic criteria). Furthermore, instead of building up a solution with additional appearance measurements, constrained synthesis reduces the solution distribution with each additional measurement; that is to say, it promotes a subtractive approach to appearance modeling. A constrained synthesis approach to appearance modeling also offers an elegant and scalable solution to reproducing a material's appearance from insufficient or incomplete data. This research builds on methods from computer vision and machine learning, and has the potential to advance the state-of-the-art in both fields. More broadly, the reconstruction and synthesis methods developed will be applicable to fields that model high dimensional data and for which it is difficult to obtain samples. The results from the proposed research activities will be incorporated in new and existing courses, recruitment activities at the graduate and undergraduate level, and outreach activities promoting STEM to minorities. This project will advance constrained synthesis as a new paradigm for appearance modeling through three research thrusts that explore generative adversarial networks (GANs) for: (1) unconstrained material generators, (2) constrained material generators, and (3) empowering users to author new material generators. In each of these research thrusts, the concept of a (potentially non-linear) projection that provides a mapping from the high dimensional target distribution to one or more lower-dimensional source distributions plays a central role. Associating a discriminator with each projection yields novel GAN architectures that decouple the spaces of the training data, conditions, and the target distribution. Building on the concept of projections and the resulting decoupling of distribution spaces, each of the three research thrusts endeavors to contribute to a different subfield in appearance modeling: (1) synthesizing novel instances of spatially varying materials, (2) reconstructing a material from a variable number of observations or from incomplete measurements, and (3) designing novel material generators by restricting the output of a general material generator based on user preferences. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.