CAREER: QoS-aware Systems for Accelerated Datacenters

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

This project addresses the challenge of providing sufficient and efficient infrastructure to meet the increasing computational demands of Artificial Intelligence (AI) and Machine Learning (ML) applications. To support the ever-growing prevalence of ML and AI applications, data centers are incorporating specialized accelerator hardware for computation as recent work has shown that traditional CPU-based infrastructures are up to 100x less efficient than these accelerator-based designs. However, while the community has largely focused on hardware designs, currently there is very little research attention on designing system software to manage the performance and efficiency for hardware accelerated data centers. Current data center system software is mostly tailored to CPU platforms. There are significant and fundamental differences between CPUs and accelerators that impact the effectiveness of design decisions in the system software stack. This work rethinks and redesigns the system software stack for the emerging landscape of hardware acceleration in data centers. The goal of this work is to redesign datacenter systems to support acceleration at scale to meet the future computational demand. The project designs system software to schedule and allocate resources among heterogeneous platforms composed of both general purpose processors and various accelerators, managing quality of service (QoS) and achieving high efficiency. To this end, the project focuses on three pillars of innovation. First, the project designs the cluster-level scheduler for heterogeneous accelerated infrastructures to precisely predict the performance interference and maximizes the utilization without violating QoS based on such prediction. Second, the project designs an application-level acceleration manager that accurately identifies the bottleneck service, estimates potential improvement using different acceleration strategies and dynamically (re)allocates the accelerator resources across service stages, resulting in significantly improved latency. Third, the project designs a node-level resource manager for accelerated platforms that manages unique sharing and reconfiguration behaviors on accelerators to ensure QoS and high throughput. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.