Statistical Inferences on Massive Data

Princeton University, Princeton NJ

Investigators

Abstract

The proposal plans to develop novel statistical theory and methods for processing massive data. Four interrelated avenues are proposed for theoretical research and methodological developments: High-dimensional variable selection, large covariance estimation, large-scale hypothesis testing, and nonparametric statistical learning. In particular, novel statistical techniques are proposed to answer the following important questions: how to screen genes and risk factors with some acquired knowledge, what are the advantages of folded concave penalized methods, how to estimate the benchmark for classifications and regressions, how to deal with outliers, dependence data, and endogenous measurements, how to use homogeneity of geographical neighborhoods to enhance forecasting and inferences, how to assess uncertainty of risk measurements, how to conduct sparse principal component analysis, how to control the false discovery rates under arbitrary dependence, how to use nonparametric methods to enhance the flexibility of high-dimensional statistical learning. In addition, a novel statistical model, motivated from a financial economics theory, is proposed for estimating large covariance matrices for better understanding risk correlations and for better assessment of risks. The methods for testing the presence of endogenous variables and the celebrated multifactor pricing models are also presented and will be thoroughly investigated. Massive data collections have become routine in exploring the frontiers of science, in one case genomic studies and in another case measuring economic risks. The proposed research will advance our knowledge on understanding molecular mechanisms, biological processes, genetic associations, brain functions, social networks, economic and financial risks, supply and demands, and hence increase economic and global competitiveness. In addition, the proposed novel statistical techniques can be applied to other biological and engineering problems. The project will integrate research and education by working closely with senior undergraduate students, graduate students and postdoctoral fellows, and increase the collaborations between academia and industry by working closely with industrial partners and developing publicly available computer code for processing massive data with sound theoretical supports. The results will be disseminated broadly through presentations at seminars, conferences, professional association meetings, and the internet.