Statistical Properties of Neural Networks

Stanford University, Stanford CA

Investigators

Abstract

Neural networks have revolutionized science and engineering in recent years, but their theoretical properties are still poorly understood. The proposed projects aim to gain a deeper understanding of these theoretical properties, especially the statistical ones. It is a matter of intense debate whether neural networks can "think" like humans do, by recognizing logical patterns. The project aims to take a small step towards showing that under ideal conditions, perhaps they can. If successful, this will have impact in a vast range of applications of neural networks. This award includes support and mentoring for graduate students. In one direction, it is proposed to study features of deep neural networks that distinguish them from classical statistical parametric models. Preliminary results suggest that the lack of identifiability is the differentiating factor. Secondly, it is proposed to investigate the extent to which neural networks may be seen as algorithm approximators, going beyond the classical literature on universal function approximation for neural networks. This perspective may shed light on recent empirical phenomena in neural networks, including the surprising emergent behavior of transformers and large language models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.