Towards Efficient Bias Correction in Data Snooping

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

The choice of a statistical model is often a critical part of data analysis because a useful model helps researchers extract relevant information from noisy data to reach interpretable findings. While scientific or economic theories do help formulate models in some applications, most data analysts have to rely on empirical models. Using the same data to select a model and then to perform model-based statistical inference is commonly known as data snooping. Unfortunately, data snooping is intrinsically risky without a careful analysis of the potential bias resulting from such practices. The primary goal of this project is to study how to understand and correct bias from data snooping and develop sound statistical inference methods. The research will provide valuable tools for scientists, researchers, and policy makers who rely on data-driven models for uncertainty assessment and confirmatory data analysis. This project focuses on regression-adjusted inference on treatment effects and inference on the best selected subgroup. The proposed work is motivated by the pressing need for more fundamental research related to the handling of "post-selection bias" in statistical analysis. The repeated data-splitting method for de-biased inference on a structural parameter (for example, the average treatment effect) enables efficient bias removal in addressing an intrinsic scientific question. The proposed inference on the best selected subgroup provides a bias-correction to a natural estimate of the subgroup effect size, and therefore reduces the risk of data-snooping and false discoveries in subgroup analysis. In the big data era, data-driven models and subgroup analyses are often used to take advantage of anticipated sparsity in the data structure or to explore data heterogeneity. The proposed research aims to provide insights, theory, and tools for more informed decision making in such endeavors. The project will involve collaborations with researchers investigating the risk of concussion as well as scientists in the biotechnology industry who routinely rely on subgroup analysis. Graduate and undergraduate students will be engaged in the proposed research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.