Quantile Regression in the Big Data Regime: Online Learning, Missingness, and Causality

Washington University, Saint Louis MO

Investigators

Abstract

This research project will develop innovative solutions for quantile regression analysis of big data. Big data has become prevalent in modern society due to the exponential growth of digital information. Quantile regression is a powerful statistical tool that goes beyond the average relationship provided by traditional regression. However, big data poses fundamental challenges for quantile regression, both statistically and computationally. This project will address those challenges by developing methods that achieve computational efficiency without losing statistical efficiency. The project will contribute to statistical research on both quantile regression and statistical computation. The new methods will be of value for applications in a wide range of fields, including economics, finance, the social sciences, and healthcare. A graduate student will be involved in the conduct of the research. Educational materials and open-source software will be created for the broader research community. This research project will develop an efficient computational and inferential framework for supporting quantile regression analysis in both massive data streams and static big data. To overcome the significant challenges brought by big data's massive scale and complexity, online processing and distributed computing methods become essential for their cost-effectiveness, scalability, and real-time processing capability. However, traditional statistical procedures for quantile regression usually do not scale well to data size and frequently are infeasible when dealing with massive data. The project has three research aims. First, the investigator will develop a fast online quantile regression analysis framework for data streams to bridge the gap between computational and statistical efficiency. Statistical theory for dealing with the non-smooth objective in quantile regression will be studied. Second, a distributed computational and inferential framework for quantile regression analysis in big data with missing data for causal inference will be developed. The new methods will be applied to electronic health record data. Finally, the investigator will develop open-source software in R or Python to implement the advances from this project. Quantile regression is used in a wide range of applications when understanding the effect of variables across the distribution is important. The developed statistical theory and inferential tools will provide new foundations for quantile regression analysis in big data and hence benefit all related application fields. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.