Scaling Laws of Deep Learning

University Of Washington, Seattle WA

Investigators

Abstract

This project builds the mathematical and scientific foundations of deep learning by characterizing the fundamental quantities and general laws that govern the empirical phenomena observed by applied scientists and engineers. Deep learning is a paradigm in machine learning and artificial intelligence where statistical models are learned from data by designing networks of parameterized modules and training them on big datasets using optimization algorithms. It has widespread effects on science and society, from autonomous vehicles to online commerce and social media. Scientific research increasingly relies on deep learning for data-driven scientific discovery. This project brings together a multidisciplinary team of statisticians, mathematicians, computer scientists, and electrical engineers. Research outcomes affect many core academic disciplines and intelligence augmentation technologies by providing scientific guidelines for practical applications of deep learning. The research program addresses the concept of scaling laws. On the practical side, scaling laws greatly simplify parameter setting for large experiments and model transfer to new domains. On the theoretical side, scaling laws shed light on empirical phenomena and unify them with mathematical concision. The team builds upon recent advances in optimal transport, empirical processes, nonparametric statistics, information theory, and complexity theory, and grounds its work in empirical observations made in large experiments in natural language processing and computer vision, among other applied domains. The research outcomes provide practical guidelines for scientists and engineers who employ deep learning to tackle challenging problems, and also constitute fundamental advances in the core areas of mathematical, statistical, and computer sciences. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.