Bayesian Model Selection and Model Combination Using Novel Developments of Stein's Unbiased Risk Estimator

Florida State University, Tallahassee FL

Investigators

Abstract

Statistical models — probabilistic descriptions of the processes that give rise to observed data — are an integral component of modern science across various disciplines, enabling researchers to learn from the data they collect and make predictions about future events. This research addresses the important challenges of model selection and model combination within the Bayesian statistical framework. Model selection involves choosing, from a collection of candidate models, the single model that best describes the data, while model combination involves constructing a hybrid model that outperforms any single model. The Bayesian framework has gained prominence in recent decades because it enables researchers to fit complex models to data, simultaneously account for multiple sources of uncertainty, and combine the information in newly observed data with prior scientific knowledge. This research will develop new, computationally efficient techniques for estimating a model’s prediction accuracy in the Bayesian framework and will apply these techniques to the problems of model selection and model combination. In the process, it will contribute to STEM education by training statisticians at the graduate level. It will also lead to publicly available software for researchers across a broad range of disciplines. This research will include several novel projects aimed at developing Stein’s unbiased risk estimate (SURE) as a practical and computationally efficient tool for Bayesian analysis. SURE has become an established tool for model selection and parameter tuning in frequentist settings. However, SURE requires the computation of a penalty term, sometimes referred to as the generalized degrees of freedom, which adds a significant computational burden for complex estimators. Consequently, SURE has been applied considerably less for more computationally demanding Bayesian and machine learning models. This research will: (1) develop a novel expression of SURE that is straightforward to compute via Markov chain Monte Carlo for Bayes estimators of a Gaussian mean resulting from essentially arbitrary prior distributions, along with extensions to unknown variances and continuous tuning parameters; (2) introduce methodology for fully Bayesian M-open inference for both Gaussian, improving upon existing model combination/selection techniques and allowing for fully Bayesian uncertainty quantification for machine learning models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.