CAREER: A Flexible Optimal Control Framework for Efficient Training of Deep Neural Networks

Emory University, Atlanta GA

Investigators

Abstract

One of the most transformative technologies of our time is deep learning, a form of machine learning that uses neural networks containing many hidden layers. Recent success has led to breakthroughs in applications such as speech and image recognition, nourishing public interest. However, more theoretical insight is needed to create a rigorous scientific basis for designing and training deep neural networks, increasing their scalability, and providing insight into their reasoning. This project develops a new mathematical framework that simplifies designing, training, and analyzing deep neural networks. Due to the broad applicability of deep learning, advances made in this project will benefit a wide range of technologies of high economic and societal impact, e.g., driverless cars, drug discoveries, and web searches. The mathematical theory supporting the new algorithms will increase the acceptance of deep learning for delicate tasks, e.g., cancer prediction and cyber-security. This project develops a new mathematical framework for deep learning based on the interpretation of deep learning as a dynamic optimal control problem involving nonlinear time-dependent differential equations. This interpretation offers a new way to analyze the successes and failures of deep learning. Advances in deep learning will be made by adapting the wide array of tools available for related optimal control problems, including numerical optimization, partial and ordinary differential equations, inverse problems theory, and parallel processing. Using these currently untapped resources provides rigorous new ways to design and train very deep neural networks that generalize well. Advances in optimal control will be achieved through new hybrid regularization methods, adaptive time discretizations, and large-scale splitting-based optimization. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.