CAREER: Statistical Inference in High Dimensions using Variational Approximations

Harvard University, Cambridge MA

Investigators

Abstract

Modern data applications routinely involve massive datasets comprising a multitude of observations and features. To facilitate statistical learning in real time, there is an urgent need for principled and computationally efficient statistical methodology. Variational Inference methods have recently emerged as a popular choice in this context. The term "Variational Inference" refers to a general out-of-the-box strategy to develop statistical algorithms for a wide class of problems. For example, these algorithms are used as a sub-routine in text mining, generation of hyper-realistic artificial text and images, machine translation, etc. This approach is extremely attractive due to the computational efficiency of the proposed methods, and their superior practical performance. Despite these advantages, rigorous guarantees for these variational methods are still in a nascent state. This project will develop statistical guarantees for the validity of this approach in diverse settings. Subsequently, these new insights will be exploited to develop novel statistical methodology for modern data applications. The outcome of the proposed research will allow practitioners to deploy Variational Inference methods with confidence. In addition, the outcomes will add a new set of principled, computationally efficient methods to the statistician's toolkit. The PI will interweave his research and teaching throughout the research period and beyond. In particular, the PI will develop new undergraduate/graduate courses focusing on Variational Inference and mentor students (particularly those from under-represented backgrounds) with the aim of introducing them to opportunities in statistics and data science. The proposed research and educational activities will broaden participation in STEM generally, and encourage careers in statistics and data science. This project will study statistical inference based on variational approximations focusing on three concrete thrusts: (i) Statistical inference based on the Naive Mean Field (NMF) approximation for regression models, (ii) NMF approximation beyond regression and (iii) Advanced Mean Field approximations. Under theme (i), the PI will develop empirical Bayes methodology for the high-dimensional linear model, and compare Bayesian variable selection algorithms using the NMF approximation. Theme (ii) will focus on the NMF approximation for Hidden Markov Random Fields and Bayesian Neural Networks. Finally, theme (iii) will focus on certain alternative mean-field approximations. Physicists conjecture that if the number of datapoints and features are both large and comparable, the NMF approximation is no longer accurate; instead, the Thouless-Anderson-Palmer (TAP) approximation, an advanced mean-field approximation, should facilitate Bayes optimal inference. The proposed research will establish this conjecture in the context of high-dimensional linear regression under a proportional asymptotic regime. The theoretical foundations of the proposed methodology will rest on disparate ideas originating in non-linear large deviations (studied in probability and combinatorics), spin glasses (studied in probability and statistical physics) and graphical models. In turn, these ideas will be combined with classical statistical ideas (e.g. nonparametric maximum likelihood) to develop computationally efficient methods for high-dimensional inference. This cross-pollination of ideas will generate independent follow up research directions in each domain. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.