Collaborative Research: Theoretical and Algorithmic Foundations of Variational Bayesian Inference

Texas A&M University, College Station TX

Investigators

Abstract

Spectacular advances in data acquisition, processing and storage techniques offer modern-day statisticians a unique opportunity to analyze large and complex datasets of unprecedented richness which arise in many scientific investigations and in studies in the social and economic fields. Bayesian inference, which combines prior knowledge and data information into a posterior distribution, provides a popular paradigm for probabilistic modeling of complex multi-level datasets and for performing associated inferential or predictive tasks in a principled fashion. For most practical problems, computing the posterior probabilities require numerical approximations; to that end, sampling-based approaches such as Markov chain Monte Carlo and deterministic approximations have both received widespread attention. Among deterministic approaches based on optimization, variational approximations, also commonly referred to as variational inference, is highly popular due to its scalability to large datasets. Through this project, the investigators will explore statistical and algorithmic properties of popular variational procedures and develop new methodology and computational tools grounded on a strong theoretical foundation. The results are targeted to empower practitioners with a better understanding of situations where variational inference is likely to be successful and where potential pitfalls exist. The research will be disseminated through articles and talks at prominent outlets. Additionally, software packages for the methods developed will be made available publicly. The investigators are committed to enhancing the pedagogical component of the proposal through advising students and developing graduate and undergraduate topic courses at their respective institutions. Motivated by the increasing need to mitigate scalability issues in Bayesian computation, variational inference has tremendously grown in popularity over the last two decades as an approximate Bayesian computational technique. Despite the proven empirical successes of variational inference in large complex data domains, systematic investigations into its statistical properties have commenced only recently. Through this project, the investigators will pose a number of foundational questions to address theoretical challenges in understanding and explaining the great empirical success of variational approximations in parameter estimation, statistical inference, and model selection, coupled with applications in novel domains. The investigators will also develop general purpose sufficient conditions to certify convergence of popularly used variational algorithms. The theoretical development will employ tools from dynamical systems, functional optimization, and optimal transport, leading to a unified treatment of statistical and algorithmic aspects of variational inference. In light of this new theory, the investigators will propose modifications to existing algorithms with certifiably better convergence behaviors. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.