CRII: CSR: Mitigating Tail Latency in Prediction-serving Systems using Coding-theoretic Tools

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Prediction-serving systems take in queries and return predictions from a machine learning model. For example, an image classification service would take an image as a query and respond with a prediction of whether there is a car in the image. Prediction-serving systems are increasingly important for a wide variety of applications today. These systems are typically run in large-scale computing infrastructures, where failures and slowdowns are common. Such unavailability results in significant increase in the query-response latency, thereby degrading the quality-of-service. The research project designs and implements a resource-efficient solution for robustness in prediction-serving systems using a tool from the domain of coding theory called "erasure-coded computation". The overarching goal of the project is to design and implement a solution for reducing the potential long latency in prediction-serving systems using erasure-coded computations. The project overcomes critical limitations in existing coded-computation solutions by employing machine learning in a novel way. Specifically, the project involves the following key tasks: (1) Designing a low-overhead coded-computation solution that efficiently supports complex non-linear functions by employing a learning-based approach; (2) Designing an efficient training methodology conforming to the constraints of real-world prediction-serving systems; and (3) System design and implementation. The software resulting from the project will be integrated into open-source prediction-serving systems for use in both research and education. Active engagement in promoting diversity in science, technology, engineering and mathematics disciplines and undergraduate mentorship will be continued. The findings from the project will be integrated into graduate and undergraduate courses in computer science. The results from the research project, including software, will be made available at http://www.cs.cmu.edu/~rvinayak/crii . This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.