HCC: Small: Beyond Pixels: Resolution-Independent and Discontinuity-Preserving Neural Image Representations for Computer Graphics

University Of California-San Diego, La Jolla CA

Investigators

Abstract

Images are key to modern communication, whether in the form of real photographs, computer graphics movies, outputs of physical simulations, or catalogs for e-commerce. They come with a standard representation, a regular grid of image pixels called a raster pixel grid. As successful as this grid has been, it imposes severe limitations as one zooms into images. For example to see the fine details of hair, the individual glints on a complex surface, the rich wrinkles or turbulence in a physical simulation, or individual rooms in a whole-earth representation is not possible using the raster grid. Given the rapid increase in geometric complexity and simulation fidelity, images need to be higher and higher-resolution to match the power of modern displays, and to provide fidelity for applications like digital twins and the metaverse. This project seeks a transformative change to this ubiquitous pixel representation, leveraging recent developments in neural fields to build a novel resolution-independent neural image representation. Crucially, the new representation will preserve image discontinuities that are core to many applications in computer graphics and beyond, like the boundaries of objects in physical simulation, rendering, and natural images. The contributions of the project will be threefold. First, a 2D Neural Discontinuity-Preserving representation will be created. This is achieved using a feature field, with features carefully constructed to be continuous almost everywhere, and discontinuous in the correct way over line/curve and point discontinuities. The interpolated features are fed to a multilayer perceptron to obtain the final function values. Second, the project will develops a number of extensions to higher dimensions such as videos, light fields, or radiance fields, and to jointly updating the discontinuity locations (when unknown) and the representation. Techniques will be developed to fit signals efficiently, and to significantly accelerate training convergence and memory efficiency. Third, the project will generate the applications of the representations. This will include image rendering, diffusion curves, physics-informed neural networks, view synthesis, and super-resolution. Beyond this, images are central to almost everything humans observe. The representation is expected to dramatically transform generation of computer-generated imagery and computational photography in a variety of different domains. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.