AF: Small: Faster Algorithms for High-Dimensional Robust Statistics

Brown University, Providence RI

Investigators

Abstract

As machine learning plays a more prominent role in our society, there is a need for learning algorithms that are reliable and robust. In modern machine learning, one often needs to work with data that are high-dimensional and noisy. Recent work gave the first efficient robust estimators for several basic statistical problems, and since then, there has been a flurry of research that obtained efficient robust algorithms for many machine-learning problems. However, one major drawback of existing algorithms in the literature is that they tend to be much slower when compared to their non-robust counterparts, or they often involve parameters that require careful tuning. To address these issues, this project aims to (i) design faster and provably robust algorithms for a wide range of high-dimensional statistical and learning tasks, and (ii) explore non-convex formulations of robust estimation and analyze their optimization landscape. This project will advance the fields of computer science and statistics, and also potentially lead to useful tools for other areas. The pursuit of faster and simpler algorithms will help accelerate technology transfer into practice, stimulate systematic approaches to robustness, and provide a positive societal impact in the long run. The education plan of this project includes incorporating the materials generated from this project into graduate-level courses at the University of Illinois at Chicago (UIC), as well as training graduate and undergraduate students at UIC, which is an urban university with a diverse student population. Designing robust algorithms in high dimensions is a very challenging task. Even for the basic problem of mean estimation, when a small fraction of the input is adversarially corrupted, no efficient algorithms were known until recently. The first polynomial-time estimators with dimension-independent error guarantees were discovered in 2016. However, given the amount of data available today, polynomial-time no longer translates to scalability in practice. Motivated by the need for faster and more practical algorithms, this project focuses on two main thrusts to expand the area of algorithmic high-dimensional robust statistics. First, the investigator would like to speed up existing algorithms and develop new robust algorithms for a broader range of problems and richer families of distributions, with the ultimate goal of matching the runtime of the fastest non-robust algorithms. Second, the investigator wants to design robust estimators that can be computed via standard first-order optimization methods. The main challenge is to find an objective function whose gradient can be evaluated using basic matrix operations while proving the structural result that this objective has no bad local optima. Concretely, the investigator plans to work on these two thrusts by targeting various aspects of the following problems: (1) robust stochastic optimization, (2) robust sparse mean estimation and sparse PCA, (3) robust covariance estimation, (4) list-decodable learning, and (5) robust learning of Bayesian networks. This project is interdisciplinary and will rely on intuition and techniques from statistics, probability, linear algebra, discrete and continuous optimization, and non-convex optimization. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.