RI: Small: Bayesian Diffusion Models with Analysis-by-Synthesis

University Of California-San Diego, La Jolla CA

Investigators

Abstract

Humans can perceive the three-dimensional world from a single two-dimensional image, even though such images do not contain explicit three-dimensional information. This capability in humans stems from two main factors: 1) the image of the three-dimensional world that matches the two-dimensional image content and 2) prior knowledge about the three-dimensional world. In statistics, this visual understanding process has traditionally been modeled using Bayesian inference, which combines the likelihood of something happening with a prior likelihood. However, this once prevailing theory has been challenged in the era of big data and deep learning, where three-dimensional understanding or inference is achieved by directly learning a mapping from two-dimensional images to the three-dimensional world. This award makes a timely effort by developing a new statistical inference technique, Bayesian Diffusion Models, which updates traditional Bayesian theory with a novel methodology to build advanced visual perception and cognition systems. As a general framework, the Bayesian Diffusion Model method is expected to have a profound impact on a wide range of tasks beyond visual perception. The ever-increasing power of generative models presents an unprecedented opportunity to revisit the analysis-by-synthesis methodology by carefully integrating the generative prior into the learning and inference of the posterior. The data under study is becoming increasingly rich, encompassing images, language, and three-dimensional data. In such contexts, data-driven techniques alone are insufficient to fully capture the posterior. Furthermore, despite significant advances in generative modeling, the synthetic content produced by state-of-the-art generative models has not yet demonstrated its potential in broadly enhancing analysis and recognition tasks. Intuitively, rich augmentation from synthetic data should play an important role in improving these data-intensive analysis tasks. This project aims to bring scientific and engineering guidance in utilizing the synthetic data for the improvements to various computer vision applications, including both closed-world and open-world 3D reconstruction, policy learning, image classification, and scene understanding. The novel Bayesian Diffusion Model (BDM) framework, grounded in Bayesian theory, promises to serve as a new statistical tool for a wide array of tasks in computer vision, autonomous driving, robotics, human-computer interaction, and computational biology. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.