CSR: Small: Developing and Implementing High Performance Erasure Codes to Tolerate Storage Failures

University Of Tennessee Knoxville, Knoxville TN

Investigators

Abstract

Computer storage systems are exploding in size and are vulnerable to an ever-increasing variety of failures. Without measures to anticipate these failures, the systems will lose data, an outcome with dire and tragic consequences. Protecting the storage systems of today and the next decade from failures that can lead to lost data is of paramount importance. Erasure codes are mathematical entitities that protect computer storage systems from losing data when they fail. With an erasure code, one computes and stores extra information from the data that is already stored, so that when disks fail or become corrupted, the original data may be recalculated from the survivors. Optimal erasure codes are well-known for small storage systems. However, larger systems exist today and will be more prevalent in the future, and the current erasure codes that protect them are suboptimal. This project attacks the unknown to develop, evaluate and implement improved erasure codes for large-scale storage installations. The project begins at the threshold of what is currently known and explores codes for larger and more complex systems. The exploration is multi-pronged, addressing the mathematical framework, structural properties and implementational features of the codes. The result will be new codes that perform better and are safer than current ones. Besides formal presentations of the codes, the project will implement and disseminate them as open-source software libraries, so that the practitioners that need them can leverage the work without having to master the complex math that underlies them.