BIGDATA: F: Statistical Foundation of Predictivity: A Novel Architecture for Big Data Learning

Columbia University, New York NY

Investigators

Abstract

Identifying variables that are good for prediction, especially in the context of BIG DATA, is an important challenge. The scientific literature currently lacks research that directly considers a variable set's potential ability to predict, referred to as "predictivity", as a parameter to be estimated. This project sets out to lay down statistical foundations for measures of predictivity, and proposes a novel framework for maximizing predictivity in big data learning. The research includes an application to big data in urban planning, addressing prediction problems in New York City's Vision Zero project. In collaboration with the NYC Department of Transportation, the PI and his team will identify risk factors and their combinations that are associated with traffic accidents and their outcomes, and improve accident prevention and victim outcome prediction. A novel sample-based measure of predictivity, the I-score, that is effective in differentiating between noisy and predictive variables in big data is proposed. This measure can be related to a lower bound for the correct prediction rate. Guided by this I-score, variable sets of high potential predictivity can be identified. This high predictivity often resides within complex interactions among the variables. To fully leverage the predictivity in an identified variable set, powerful classifiers based on deep architectures will be constructed. Novel strategies are proposed for scalable computational implementation of the proposed framework. Systematic evaluation of the proposed methods, comparing with current strategies, will be carried out using simulations and benchmark real data sets.