A Comprehensive Framework for Fully Efficient Robust Estimation and Variable Selection, with Application to High-Dimensional and Complex Data

North Carolina State University, Raleigh NC

Investigators

Abstract

This research addresses robustness in both estimation and variable selection within the context of today's complex data structures. The overarching theme of the research is that carefully specified moment restrictions combined with appropriate weighting of the data will lead to the ideal goals of full efficiency in estimation and variable selection which remains stable in the presence of atypical observations. The methodology is developed via generalized empirical likelihood, which yields estimated weights for each observation. In the process, this automatically downweights observations that may deviate from the model, thus reducing their influence. Meanwhile, the estimators have no loss of efficiency compared with the fully efficient model-based estimator if the model were correctly specified, even in finite samples. Taking this point of view allows a unified framework to the construction of robust and efficient procedures that can be developed for a variety of models. The foundation of efficiency and robustness allows variable selection to be built into the methods to handle, not only the moderate, but also the high-dimensional setting. Due to the performance of the baseline approach, the variable selection consistency under contamination and misspecification can improve on existing selection methods that rest on a starting point that may be already non-robust or less than fully efficient. Modern scientific data is characterized by a wealth of information. The data explosion has arisen in diverse areas running the gamut from drug discovery to the financial markets and even homeland security. While the massive influx of data has led to breakthroughs in these fields, it brings many statistical issues to the forefront. In particular, it can be an overwhelming task to determine the relevant predictor variables that provide a suitable model. Meanwhile, with today's complex data, this postulated model will surely be only a simplification of reality. Thus it is inevitable that some of the data will deviate, perhaps significantly, from the model, although it is still useful for the bulk of the data and can provide meaningful insight. This research targets the essential task of developing techniques to perform estimation and variable selection, while also allowing for some of the data to deviate from the model without greatly affecting the results. The methods developed from this research are robust to outliers and model misspecification, while still maintaining efficiency for both estimation and variable selection even in the presence of this contamination. Thus it will be a key component to enable meaningful results in the face of complex data.