CAREER: Optimizing Cloud-Native Databases for Storage Disaggregation

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Databases are critical software systems that manage the ever-growing volumes of data. While conventional databases run in user-controlled hardware, modern database systems are migrating to the cloud due to reduced complexity and cost in hardware and software management. Such cloud-native databases have unique architectural features compared to conventional on-premises databases, such as high scalability, elasticity, and availability. These new architectural features bring new challenges and opportunities for performance optimizations. This project will investigate new algorithms and system designs that better leverage the unique architectures in the cloud, and improve performance, cost-effectiveness, and reliability. The project will advance the understanding of cloud-native databases and add significant value to society at large. The project will build open-source prototypes of developed techniques to enable future research. The project will also modernize database courses to include hands-on exercises on cloud-native databases to resolve the severe shortage of talent in the field and promote STEM education to the public. Modern cloud-native databases adopt a unique storage-disaggregation architecture, where the computation and storage are decoupled as two separate layers of services and then connected through the data center network. Disaggregation enables the storage layer to scale independently from computation, which allows for flexible resource allocation and cost reduction, but causes the network connecting the two layers to be a new performance bottleneck due to its high latency and low bandwidth. This project aims to resolve the network bottleneck by pushing certain database functions (such as filtering, aggregation, atomic compare and swap, etc.) down into the storage layer to reduce the amount of interaction between computation and storage, while still fully leveraging the benefits of disaggregation. The project revisits the conventional wisdom of database design and focuses on optimizing the building blocks that are the most susceptible to the disaggregation bottleneck, including query execution, query optimization, distributed transaction, and real-time data analytics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.