CRII: CHS: Learning Procedural Modeling Programs for Computer Graphics from Examples

Brown University, Providence RI

Investigators

Abstract

Procedural modeling is used to programmatically generate visual content for instruction, simulation, animation, visual effects, architecture, graphic design, and other applications. An effective procedural model can produce a variety of detailed, visually interesting, and even pleasantly surprising results. Unfortunately, such models are difficult to author, requiring both visual creativity and programming expertise. More people could be empowered to create and use procedural models were it possible to deduce them from examples. The current project will tackle this long-standing open problem in computer graphics by building on the PI's prior work to develop a research program investigating new approaches to learning procedural models from examples by combining probabilistic programs with neural nets; programs are expressive enough to represent a variety of visual content, while neural networks provide flexible learning from data. Project outcomes will help democratize procedural modeling by allowing users to create procedural models with examples rather than by writing code, so that a wider demographic of creative professionals and enthusiasts can participate. All code and data produced will be released as open-source, to allow other researchers and developers to apply and extend the new techniques. Because graphical content is often hierarchical, (probabilistic) grammars are typically used to procedurally model it. However, such content is also characterized by continuous attributes: colors, affine transformations, and so on. While grammars can be extended to support some of these attributes, there are no general-purpose methods for learning such models from examples. Existing approaches either ignore continuous attributes or are specialized to one type of content (e.g., building facades). This research presents a new general-purpose approach for example-based learning of procedural models which generate discrete hierarchical structures with continuous attributes. The key insight is representing a procedural model as a probabilistic program whose control flow and data flow can be governed by neural networks. Like a grammar, such a program can naturally represent (possibly recursive) hierarchical structure. The neural network logic of the program can represent complex functions which generate continuous attributes such as transformations. The model is efficiently learnable with stochastic-gradient-based methods and has the potential to scale from small numbers of examples to large datasets. The initial focus will be on learning procedural models of 3D scene graphs, which are 3D objects composed of a hierarchy of parts. The research will then expand into learning procedural models from large datasets of examples, applying the techniques to domains beyond 3D scene graphs, and leveraging unstructured inputs such as images as examples. Project outcomes will include new mathematical frameworks for learning procedural models from examples, algorithms for efficiently solving the learning problem, and evaluations of the quality of content generated by learned models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.