CNS: Core: Medium: Understanding and addressing device-reliability heterogeneity in large-scale distributed storage

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

This project develops new data protection approaches to reduce both data losses and cost-inefficiency in large storage systems. Modern data-centers store data in large collections (10s of thousands) of disk storage devices. To avoid data loss in the event of a disk failure, each chunk of data is stored redundantly, such that there are multiple copies or extra code-information that allows data chunk reconstruction. Previous approaches assume that all of the disks are the same in terms of their failure rates and aging properties. But, while that was true for smaller storage systems in the past, it is not true for the large storage systems underlying today's data-driven world. The result is that some data is not protected enough and other data is protected too much (wasting money) - worse, no one really knows which is which. This project develops new tools for dynamically learning the failure characteristics of different disks, automatically determining the right levels of redundancy for those characteristics, and efficiently adapting redundancy levels as circumstances change in the system. By addressing a major flaw in how storage reliability is designed and managed, this project will make data protection stronger and less expensive. Both are crucial. Data protection is crucial, because data is core to the success of data-driven revolutions happening in medicine, science, politics, business, etc. Losing data hampers their success. Data storage cost is crucial, because of the massive dollar, power, and environmental effects of the disks and data-centers used to house and operate them. For example, early analyses suggest that this project's outcomes could potentially reduce the number of disks needed by 20%, which would represent millions fewer devices in U.S. data centers. This project will include engaging with companies and national labs that operate large storage systems in order to help move the research results into practice. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.