SHF:Small: Distributed Key Manager for higher functionality KV Stores

Texas A&M Engineering Experiment Station, College Station TX

Investigators

Abstract

Key-value (KV) stores have been widely deployed in a variety of enterprise applications such as online retail, big-data analytics, social networks and others. Key-Value Solid-State Devices provide a key-value interface directly from devices aiming at lowering software overheads and reducing I/O amplification for such applications. With the advent of KV devices, new research is needed to architect higher-layer systems on top of these novel devices. This project plans to address the research issues to enable wider adoption of these novel KV devices. (1) Current KV stores rely on replication for providing fault tolerance. While replication is simple and effective, it incurs high storage overhead and high write amplification. Developing alternate lower-cost fault-tolerance options would support wider adoption of KV devices. (2) Many applications require support for range queries. Providing efficient indexing of keys at the KV devices could enhance these applications. (3) Currently available systems interface with block devices, and the semantic information about the KV records is only retained within the application. This project proposes to exploit the semantic information at the KV devices to provide additional functionality and optimizations for a KV store, such as near-storage processing. This project has the following objectives. (1) Design, develop and evaluate a distributed key manager to rearchitect KV stores on top of KV devices. (2) Investigate mechanisms for extracting information and organizing metadata within the distributed key manager. (3) Investigate mechanisms for efficient processing of stored data near the KV devices. (4) Explore options for architecting enhanced functionality and interfaces within the distributed key manager. (5) Investigate cost-benefit tradeoffs in supporting this new functionality within the distributed key manager. In order to achieve the project goals, the researchers plan to employ indirection, novel indexing, intelligent processing near storage and collaboration across devices. The project is expected to provide new enhancements that can be layered on top of KV devices and point to new ways of architecting applications to reduce software overheads. The proposed research is expected to lead to the development of new ways of utilizing emerging storage devices with new interfaces, enable advances in near-storage processing applications. These advances will enable important societal benefits such as big-data processing, facilitating new scientific achievements and understanding. Educational impact will include training graduate and undergraduate students with valuable research skills while advancing the state of the art in computer architecture and distributed storage systems, contributing to the technology workforce. Minority students will be involved in the project. The planned industrial collaborations related to the project are expected to hasten commercial adoption of developed systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.