III: Small: Algorithms and Theoretical Foundations for Approximate Bayesian Inference in Machine Learning

Indiana University, Bloomington IN

Investigators

Abstract

Over the last two decades Bayesian models have become central in machine learning. Bayesian models often hypothesize latent (non-observed) variables with explanatory or predictive power toward observed phenomena. The challenge is to infer the state of these variables or a belief over that state from observed data. For example, one might try to infer a user's preferences from observations about their own behavior and the behavior of other users. The goal of this project is to develop general approximate inference algorithms that work across large families of Bayesian models so that solutions can be widely reused. The algorithmic work will be complemented by developing a learning theory for approximate Bayesian inference in machine learning. The theoretical framework will aim to prove performance guarantees for Bayesian prediction algorithms and inform the design of algorithms with desirable properties. The project will contribute to basic scientific research, advancing core goals in machine learning. The project will support training and research of PhD students and therefore will directly support human development. Through classroom teaching and outreach the project will expose a larger population of students to machine learning and its potential in applications. More concretely, the project will investigate non-conjugate Bayesian latent variable models, i.e., it will avoid the often used but limiting simplifying assumption of conjugacy. On the algorithmic side the project will aim to generalize the paradigm of variational message passing for non-conjugate graphical models, and to develop stochastic variational inference algorithms using optimal structured approximations for large sub-families of such models. The proposed sub-families will capture the properties of many important problems in the literature. Exploratory research in several specific applications further motivates the work and will be used to test the algorithms. The project will develop a new angle for theoretical analysis of Bayesian algorithms, deriving performance guarantees on their expected error. A core idea is to view variational inference algorithms through the so-called agnostic learning framework where guarantees sought are relative to the best that can be done within a specific limited class of approximations. This will provide a fresh outlook that informs the design of algorithms with desired performance guarantees. The expected scientific impact of the project is having better algorithms with well understood performance characteristics and applicable for a larger class of machine learning problems.