A New Stochastic Neural Network: Statistical Perspectives and Applications

Purdue University, West Lafayette IN

Investigators

Abstract

The integration of computer technology into science and daily life has enabled scientists to collect massive volumes of data. Deep learning has been developed as a major tool for big data analysis. However, the structure and parameters of the deep neural network (DNN) are hard to interpret, which can cause severe issues in human-machine trust when applying the DNN to real-life settings. To address this concern, researchers have made considerable progress in sparse deep learning, which provably leads to consistent selections of relevant variables for the underlying nonlinear system. However, the internal nodes and parameters of the sparse DNN are still hard to interpret due to the black-box nature of the DNN. The investigator will develop a new type of stochastic neural network (StoNet), which is a composition of many simple regressions. The StoNet is asymptotically equivalent to the conventional DNN in function approximation as the training sample size becomes large, while its structure and parameters are more interpretable from statistical perspectives. The StoNet can be employed to address many fundamental statistical tasks such as nonlinear sufficient dimension reduction, causal inference, missing data, and private deep learning that are difficult to handle with the conventional DNN. The StoNet bridges linear models and deep learning by its compositional regression structure, which deepens people’s understanding of deep learning. The StoNet has potentially immense benefits to the development of trustworthy artificial intelligence (AI) and data driven technologies. The research results will be disseminated to communities of interest via collaborations, publications, and conference presentations. The project will also have significant impacts on education by directly involving graduate students in the research and incorporating the research results into undergraduate and graduate courses. The StoNet provides a more general and powerful model for big data analysis than the conventional DNN. It can be employed to address many fundamental statistical tasks that are frequently encountered in modern data science. The investigator will show that the StoNet results in a novel nonlinear sufficient dimension reduction method by imposing a Markovian structure on its layers in training. The resulting method is scalable and can deal with much larger datasets than can the existing methods. For causal inference, the investigator will develop a causal-StoNet as a variant of StoNet, where the treatment variable is included as a visible unit in a middle layer of the network. The causal-StoNet provides a convenient way for modeling the outcome function and propensity score, imputing missing data, and identifying relevant covariates for high-dimensional problems. For private deep learning, the investigator will develop a varying truncation noisy stochastic gradient descent algorithm for training the StoNet. Compared to the existing private stochastic gradient descent algorithms, the proposed algorithm avoids gradient clipping and improves convergence and utility of deep learning while ensuring rigorous differential privacy guarantees. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.