Theory and Methods for Modern Predictive Inference

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

In statistics and machine learning, a central inferential task is to predict the values of future data given past observations. Prediction algorithms are widely used on a daily basis, such as spam detection, health risk evaluation, weather forecasting, economy, etc., making it one of the most fundamental and important classes of statistical inference tasks. In the past decade, the emergence of deep neural networks and powerful computers have made major progress in designing and implementing prediction algorithms, resulting in an explosive development of methods and applications. These new applications call for novel statistically principled methods with mathematical justifications to fully and correctly exploit the power of such new tools. However, the unseen level of complexity in both the algorithms and datasets poses fundamental challenges to classical statistical and learning-theoretical frameworks that rely on simple problem structures. Motivated by the algorithmic and computational advances in the modern data science era, in this research project, we plan to take on several new methodological challenges and fill theoretical gaps in statistical predictive inference, including simultaneous accuracy evaluation for a large collection of prediction models, and valid statistical inference using calibrated prediction. The project also provides research training opportunities for graduate students. The research project consists of two parts. The first part will provide new insights into cross-validation, one of the most widely used methods for model and tuning parameter selection. The theoretical development will contribute to the understanding of the joint randomness of cross-validated risks, not only explaining the widely observed overfitting tendency of cross-validation but also pointing out potential solutions to correct it. This research work will further enrich and connect multiple areas of active research, including cross-validation, high-dimensional Gaussian comparison, model confidence set, and online learning. The second part aims at filling an important gap in the conformal prediction literature by developing a conformal-based hypothesis testing method beyond ex-changeability. We plan to explore connections between conformal prediction and classical topics such as Mann-Whitney rank-sum statistic, two-sample U-statistics, and semiparametric inference. The expected results will lead to new applications of conformal inference and create a new array of research problems across these related research topics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.