CRII: CSR: Large-scale Systems Software Atop Scale-out In-memory Storage

University Of Utah, Salt Lake City UT

Investigators

Abstract

Research over the past 5 years on in-memory data center storage has resulted in systems that can provide on-demand access to billions of pieces of information per second. However, the benefits of these fast systems has not resulted in more powerful and interesting applications for users. The key problem is that these new storage systems are stripped down to provide speed and can only fulfill very basic requests for information. To compensate, applications must make many requests to these storage systems to perform the same operations they were able to perform with a single request to older database systems. The result is that applications spend most of the performance benefits they gain from these systems overcoming their limited programming model. This work seeks to understand the hidden costs of the limited interfaces of these large-scale in-memory key-value stores and to alleviate those costs with a new stored procedure model uniquely suited to these systems. To explore these costs, data-intensive applications from three different domains will be prototyped on today?s low-latency key-value stores, first with their conventional interfaces and then again with custom application logic built into the storage system. The results of this exploration will motivate and aid in the design of the main contribution of this work: a generalized model for safely collocating application logic inside of key-value stores. Application prototypes will include a scale-out database kernel, a simple scale-out graph database, and large-scale machine learning algorithms. A rich programming model for developing large-scale software systems will lower the barrier for building information-intensive applications: an art that requires specialized expertise. Low-latency systems equipped with fast stored procedures will be a powerful tool for building new, data-intensive systems in industry, science, and the military. They will enable deep real-time analysis of social and natural graphs; rich, interactive worlds where millions of users manipulate and modify a shared environment; the fine-grained coordination and routing of millions of autonomous vehicles on the highways in a metropolitan area; and real-time decision support for military intelligence. All components will be built as a practical and usable software system with all development done publicly and available as open-source.