CIF: Small: A Probabilistic Theory of Deep Learning via Spline Operators

William Marsh Rice University, Houston TX

Investigators

Abstract

Deep learning has significantly advanced the ability to address a wide range of difficult machine perception tasks, such as recognizing objects from images, activities from videos, or text from speech. As a result, deep learning systems not only are playing a key role in emerging products and services, from conversational assistants to driverless vehicles, but they are also revolutionizing existing ones, from robotics to legal document analysis. Moreover, in the scientific realm, deep learning is enabling new ways to find patterns in large complicated datasets. This success is impressive, but a fundamental question remains: Why does deep learning work? Intuitions abound, but a coherent framework for understanding, analyzing, and designing deep learning architectures has remained elusive. This project will develop a theoretical foundation for deep learning systems by connecting them to classical and recent results from the signal processing, approximation theory, information theory, and statistics. A key goal is the development of new kinds of deep learning systems whose inner workings are explainable and interpretable. This project will have a range of impacts, from developing trustworthy, interpretable models and algorithms for mission-critical applications like autonomous navigation and decision making to advancing machine learning and signal processing education. This project builds on an elegant connection between a wide class of deep (neural) networks based on piecewise-affine, convex nonlinearities and max-affine spline operators (MASOs). The research is organized around two interlocking themes. The first theme revolves around the extension of the MASO framework beyond piecewise-affine, convex nonlinearities by linking deterministic MASOs with probabilistic Gaussian mixture models. The extended, probabilistic MASO will enable the analysis of deep networks with more general nonlinearities than those that are piecewise-affine and convex, such as the sigmoid, hyperbolic tangent, and softmax. The second theme revolves around extending deterministic MASO deep networks to a new class of hierarchical, probabilistic, generative models that generalize the feedforward inference calculations and backpropagation learning of conventional deep networks to optimal Bayesian inference via a closed-form variational expectation-maximization (EM) algorithm. The probabilistic structure will enable the full arsenal of probability and statistics methodology to be applied to deep learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.