CSR: Small: Latency-controlled Reduction of Data Center Expenses for Handling Bursty ML Inference Requests

Ohio State University, The, Columbus OH

Investigators

Abstract

Data centers must often over-provision the GPUs used for online machine learning (ML) inference because inference requests can arrive in a large number as bursts. Such over-provisioning results in unnecessarily high capital expenses (CapEx) for data centers. This project aims to design a holistic management framework that handles bursty ML inference requests with latency guarantees and minimized CapEx. First, the proposed framework will co-locate ML inference and training workloads efficiently on the same GPUs, with latency guarantees, for improving GPU utilization. As a result, GPUs can be used mostly for inference when a request burst comes, and then mostly for training afterwards, which can reduce the number of GPUs needed, and thus the data center’s CapEx. Second, a novel task scheduling algorithm will be designed to consolidate negatively correlated ML tasks onto the same GPUs for further reducing CapEx. Third, data center power/cooling CapEx will be reduced as well, by exploiting existing energy storage devices already equipped in most data centers to supply additional energy during a burst, in order to avoid expensive power facility upgrades. Finally, all the software, hardware, and power/cooling facilities will be coordinated as a holistic framework to efficiently manage a data center as one massive warehouse-scale computer. As ML cloud services are becoming increasingly popular, the fast-growing business demands for computing are driving data centers to run their GPUs and servers with higher performance and lower costs. The success of this project would positively impact data center design by allowing data centers to safely handle ML inference bursts, with desirable latency guarantees and minimized CapEx. Currently, in order to deal with bursty ML workloads at an increasing scale with latency guarantees, a data center must often 1) over-provision their inference GPUs and keep them separated from GPUs used for training in order to minimize the risk of resource competition that might jeopardize inference latency; and 2) upgrade the corresponding power/cooling facilities for hosting the increasing number of GPUs and servers to prepare for occasional worst-case scenarios. Such capital investment increases can impose a serious burden on ML/AI companies that run their own data centers. For start-ups and smaller ML/AI companies that rely on the cloud, the higher data center CapEx can lead to higher cloud service bills. The proposed framework can provide latency-controlled CapEx reduction for data centers, thus allowing those booming ML/AI companies to have a better chance of success. Further, this project could benefit other workloads that also perform intensive GPU computing, such as high-performance computing, big data analytics, and cloud virtual reality. The results of the project will be transitioned into new and existing undergraduate and graduate courses at The Ohio State University. Students from groups underrepresented in computing will be actively recruited in this project. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.