CSR: Medium: Declarative Programmable Storage

University Of California-Santa Cruz, Santa Cruz CA

Investigators

Abstract

Everyone who processes large amounts of data - biologists and physicists, data scientists, and computer scientists of all stripes - must rely on distributed storage systems. Storage systems deal with conflicting demands: they must accommodate the changing workloads of the domain experts who use them, while ensuring good performance by taking advantage of changing hardware such as faster disks and networks - and guaranteeing that data is safely stored. This means that they must be rigid and flexible at the same time. Declarative programmable storage attempts to provide the best of both worlds by providing a simple, concise language for describing new storage interfaces, and using techniques from database optimization to automatically find correct and efficient implementations of the interface. Declarative programmable storage identifies key opportunities to apply classic data management systems concepts - such as declarative language and cost-based query optimization - to the domain of large-scale storage systems, allowing for cross-pollination between the storage, data management and distributed systems research communities. Additionally, this project requires novel research into cost modeling and query planning in the context of the storage domain. By decoupling the specification of new storage application programming interfaces (a task that should be undertaken by domain experts - e.g., physicists - in a high-level language) from their concrete implementations (a low-level task that can be automated), this project will free the users of distributed storage systems from the need to be experts in the internals of such systems. Consequently, these domain experts will be able to focus on innovation within their own area of expertise rather than becoming expert storage system programmers as well. These application programming interfaces can be specified once and re-optimized whenever device characteristics change. Hence the labor of these users will be made "future-proof" against the rapid evolution of storage hardware, software and configurations. Scientific results, software, and data from the work performed under this grant will be made available to the public under a free and open source software license. The project repository will be available at https://declstore.soe.ucsc.edu/ backed up by departmental backup systems to ensure long-term preservation of digital artifacts. In the long-term, this project will aim to engage open-source software communities to maintain and evolve software created under this grant. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.