CAREER: Random matrices and High-dimensional statistics

University Of California-Berkeley, Berkeley CA

Investigators

Abstract

This research program is focused on the development of data analysis methods and of a theoretical framework for the new paradigm of high-dimensional statistical problems. The theoretical problems are concerned with spectral properties of large dimensional random matrices. More precisely, four of the main objectives of the program are: 1) further develop new covariance estimation methods; 2) further our understanding of the spectral properties of relevant large random matrices; 3) find and contribute to areas of application where this high-dimensional statistics framework is relevant; 4) train graduate students in high-dimensional statistics and make undergraduate students at least aware of possible pitfalls of classical methods and of better alternatives when available. More specifically, statisticians are now often faced with "n by p" data matrices X, for which p, the number of variables recorded per observations, is of the same order of magnitude as n, the number of recorded observations, and p and n are both large. The sample covariance matrix computed from this data is of great importance to a number of applications, as it underlies widely used methods like principal components analysis. However, the theoretical results which underlie the method, classically developed in the "small p and large n" setting, fail to apply in the "large n and large p" setting just described. Hence, a thorough study of sample covariance matrices in this setting is needed. Eigenvalues of such large dimensional matrices are of particular interest. The investigator plans to launch a multi-pronged effort to get at various kinds of properties of these objects: for instance, he plans to develop theoretical results that will allow inferential work to be done from computation of extreme eigenvalues of sample covariance matrices, develop new methods of estimation of the whole covariance matrix, and also work on the impact of naively plugging-in the sample covariance matrix as a proxy for the population covariance in certain optimization problems which depend on this latter parameter. An effort will be made to try and apply this theoretical work to real-world problems, both to raise awareness in applied communities about the pitfalls associated with high-dimensional covariance matrices, and to shape the models that will be studied to be of most relevance to applied researchers. Technological progress allows us to store and use massive amounts of data about many aspects of our daily lives. An interesting problem is to use the data to understand how certain traits depend on each other. In the stock market, we might be interested in how the behavior of one stock affects the behavior of another stock; understanding all these interrelationships leads to having a measure of the risk taken by investing in portfolios that use the corresponding stocks. Statisticians have a number of tools to deal with all these interrelationships. We can discover ways to look at the data so that, even if all interrelationships are small or weak, so each trait "should" not help us learn too much about any other trait, we might still find combinations of the traits that carry enormous amounts of information. We also know what typical values for these combinations are, so we might be able to detect unusual features in the data set by looking at it the right way. Those statistical techniques have very wide applications in various fields of science, ranging from climatology to genetics, image recognition, finance etc... Thousands of research papers are published each year that use these techniques. However, the theory that underlies these statistical techniques was created in an era where massive datasets just did not exist. This research project is focusing on theories and their applications that are better suited to handle our current massive datasets. The applications should allow us to see structure where the classical tools fail to see any and tell us when there is no structure when the classical tools tell us there is. We also have increasing evidence that our standard tools give us often very inaccurate results about our standard measures of risk or amount of information carried in combination of traits. It seems that risks might be underestimated and amount of information might be overestimated. Part of this research program will be dedicated to measuring how inaccurate the classical results are for large datasets, how much practical predictions are affected, and how a more relevant theory can be used for correcting these inaccuracies.