Model Evaluation in Modern Predictive Regimes: Case Influence and Model Complexity

Ohio State University, The, Columbus OH

Investigators

Abstract

Models are an integral component of scientific inquiries, and they can be used as effective devices in many practical areas as engineering, commerce, and governance. There have been remarkable advances in the way statistical models are defined from data for prediction or for accurate descriptions of the world we observe. While complex predictive models have routinely emerged, our understanding of these models and methods for their evaluation have been lagging. Appropriate tools for assessing models and recognizing their deficiencies, and proper ways to account for their complexity are in great need. To fill these gaps, the project aims to develop methodologies and computational tools for assessment of case influence on general models in predictive settings. And it will extend the notion of model complexity to general prediction rules for model comparison and calibration by using the overall model sensitivity to data perturbation. Results from this research will bring great benefit not only to science and engineering through the practice of refined statistical modeling, but also to society at large through applications. In particular, the project will have practical utility in outlier detection for many scientific applications, fraud/threat detection and prevention for many business applications, and detection of adversarial attacks for artificial intelligence applications. Moreover, it will advance our understanding of modern algorithmic models such as deep learning through the research on model complexity and foster interdisciplinary research. The project will provide research training opportunities for graduate students. Computational tools developed will be distributed as open-source software. To characterize the sensitivity of a predictive model to data, the PIs will develop novel approaches to case influence assessment for general modeling procedures, encompassing many modern statistical learning techniques for classification and regression. Extending case deletion statistics and case influence graph in linear regression, the project will offer a variety of new case influence measures for classification in particular. In addition, the PIs will develop efficient computational algorithms for evaluating those case influence measures by utilizing a homotopy technique to relate two modeling problems with the original data and perturbed data under various perturbation schemes. Further, the project will examine model complexity through the lens of model sensitivity to data perturbation considered in case influence assessment and extend the notion of model degrees of freedom to general modeling procedures including large-margin classifiers. This extension will be based on the relation between expected optimism and model complexity in the risk estimation framework where model sensitivities to perturbation of individual cases can be linked to the conditional expected optimism. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.