CAREER: Multi-Query Optimizations for Deep Learning Systems

University Of California-San Diego, La Jolla CA

Investigators

Abstract

Large-scale data analytics using predictive models called "deep learning" has revolutionized many digital applications, powering modern speech recognition, language translation, Web search, and more. This success of deep learning, primarily at resource-rich technology companies, has led to high interest in adopting deep learning in domain sciences, enterprise companies, healthcare, and even digital humanities. But a major bottleneck to broader adoption is the high resource cost of training deep learning models, which requires a computationally expensive empirical process with a large number of trials. This slow process raises resource costs, wastes energy, and impedes user productivity. This project tackles this problem by devising new techniques to substantially speedup up this process on deep learning systems. It will reduce resource costs and energy needs, and in turn, help democratize deep learning to more application domains. It will lead to a new open source system integrated with existing popular deep learning tools to make it cheaper, faster, and easier to adopt large-scale deep learning. The system will be used by domain scientists and also integrated into industrial products. The research will be disseminated via publications at top conferences and incorporated into new courses on data analytics systems. This project will support graduate, undergraduate, and high school students, including LGBT+ and female students. This project will improve the resource efficiency of scalable deep learning model selection, an empirical process that typically requires training dozens to hundreds of model configurations with varying data representations, neural architectures, and hyper-parameter values. Existing tools like TensorFlow and PyTorch focus on the efficiency of training one model a time, which wastes resources at scale during model selection. Some systems also sacrifice reproducibility, a showstopper for many users. This project resolves these issues by presenting a fresh database systems-inspired view of deep learning that re-imagines its executions as queries. Targeting small cluster settings, it raises the specification of three common deep learning model selection tasks to a declarative level and runs many related model configurations in one go. It proposes a suite of multi-query optimization and view materialization techniques that reduce communication costs and/or avoid computational redundancy, while not sacrificing reproducibility or prediction accuracy. The techniques combine the mathematical properties of stochastic gradient descent and the computational properties of deep learning queries with careful parallel data system design and implementation. Project website: https://adalabucsd.github.io/cerebrosystem/ This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.