Collaborative Research: CIF: Medium: Towards optimal scheduling for parallelizable machine learning training workloads

University Of North Carolina At Chapel Hill, Chapel Hill NC

Investigators

Abstract

In the past fifteen years, there has been explosive growth in the number of applications seeking to leverage Machine Learning (ML) models. Before an ML model can be deployed, the model must be “trained” by repeatedly processing examples. Each example helps the model to make progressively more accurate predictions, "learning" how to solve the problem at hand. Unfortunately, this training step requires a large amount of expensive, highly-specialized hardware and can take several hours to complete. Given limited hardware resources, it is not obvious how to allocate (share) these resources across a stream of ML training jobs. The goal of this project is to develop new resource allocation policies that allow us to produce highly-accurate ML models, quickly, and with limited resources. The main challenge in this work is that ML training jobs present a number of unique characteristics compared to other computing workloads. ML training jobs are highly parallelizable, meaning a single training job might run across multiple servers. Each job also has a large number of configuration options that affect how fast the model learns a particular problem. This project uses mathematical modeling to develop new allocation policies specifically designed for ML training jobs. This research will be accompanied by the development of new courses and mentorship programs designed to recruit students into research on the modeling of computer systems. Machine Learning (ML) models are increasingly being deployed across a wide variety of applications. The growth in ML models has been accompanied by the development of specialized hardware accelerators that help reduce the training time for ML models. However, there has not been a similar degree of specialization in the scheduling algorithms used to train ML models on clusters of specialized hardware. The question of how to best schedule ML training jobs is non-trivial. ML training jobs present many unique characteristics compared to other computing workloads. First, ML training jobs vary in their degree of parallelizability (ability to scale out across servers), with some jobs being highly elastic and others being inelastic in their scalability. It is not clear how to allocate (share) hardware resources among jobs with such different characteristics. To make things more complicated, each training job is parameterized by a set of tuning parameters called hyperparameters. The parallelizability of an ML training job varies over time as the job hyperparameters change. Hence, systems which either rely on static resource reservations or existing heuristic scheduling policies are poorly suited for ML training jobs. Finally, the inherent work associated with an ML job is not a fixed quantity. Instead, ML jobs are generally trained until the model meets a desired level of accuracy. Hence, the existing scheduling theory that favors “short” jobs does not apply in this case. This proposal develops a new theoretic framework for modeling the dynamics of ML jobs running in a shared cluster. This framework will be used to develop scheduling and allocation policies that are specialized to ML training jobs, and to prove performance guarantees on these policies. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.