Incremental Regression Analysis of Streaming Data: Estimating Function Theory and Applications

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

The advent of distributed data storage and parallel computing systems such as the Apache Spark has provided opportunities of innovation in data analytics and modeling. This project focuses on regression analysis of streaming data under the Spark's Lambda architecture, aiming to develop a new toolbox of Big Data analytics. Streaming data refers to a series of data batches that arrives sequentially. Such data collection schemes have become abundant lately in biomedical fields due to the booming of many AI-enhanced medical devices that are designed to monitor safety and effectiveness of medical treatments delivered by smart personalized products, or to measure real-time physiological variables such as heart beats, body temperature, and physical activity. This so-called deep phenotyping technology has significantly changed the way of information acquisition in terms of both volume and velocity. Being the most important data analytics, the regression analysis will be rebuilt in the proposed project to address various challenges from the processing of streaming data. The resulting methodology may be applied to many practical fields, where incremental learning with data streams is of primary interest. The overarching goal of this project is to develop an incremental statistical inference to address methodological challenges in regression analysis with streaming data stored in the Spark's Lambda architecture. Efficient incremental methodology requires no use of any historic raw data, rather only historic summary statistics and a newly arrived data batch. At the completion of this project the PI expects to make the following new contributions: (i) To develop a new theory of renewable estimation and incremental inference in the context of estimating functions; (ii) to develop an expansion of speed data flow architecture, called the Rho architecture, in which a new layer is added to carry over updates of inference-related quantities such as the Fisher information; (iii) to apply the proposed methodology in many important regression models, such as the generalized linear models, the generalized estimating equations (GEE), the Cox proportional hazards model, and the quantile regression model. Both python and R packages will be delivered from this project to the public. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.