CSR: Medium: Collaborative Research: GPL: General-Purpose Lambda Computing

Stanford University, Stanford CA

Investigators

Abstract

The project will design ways for computer programs to do things in a few seconds that would ordinarily take hours, by borrowing thousands of computers at once. Until recently, this was not possible or economical: the companies that rent computers required minutes of advance notice, with a minimum bill of at least ten minutes. Recently, new technology called "function-as-a-service computing" allows people to rent thousands of unreliable computers at the same time with no advance notice, for fractions of a second. This project will design software, storage systems, and programming languages to use this technology robustly for large calculations. This research aims to deliver a new model of "burst parallel" applications and systems. This model takes advantage of new function-as-a-service offerings by cloud-computing providers such as Amazon, Microsoft, and Google. In this model, fine-grained "ephemeral lambdas" enable large-scale parallel computations at low cost. With careful attention to algorithm parallelization, we argue that many both old and new data processing tasks can be adapted into a model of functions that operate on small pieces of data, executed in a burst-parallel manner to achieve interactive performance demands, in a way that can be debugged, traced, and understood. This project intends to provide the application blueprint and systems foundation to enable end-users and developers to access massive amounts of parallelism on demand, while only paying for those resources actually consumed by an application. The researchers will engage with the developer community to ensure that others are able to use and build upon the work. In addition to integrating this research into course curricula, the researchers have a formalized program for embedding under-represented minority undergraduate students into their research groups. Software will be released under open-source software licenses pursuant to the policies of the University of California and Stanford University, and distributed through the project website and through traditional means of distribution of open-source software (e.g., GNU/Linux distributions, ports collections, app stores, etc.). Data, code, and evaluation results will continue to be hosted for at least three years after project completion at https://gplambda.github.io. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.