CIF: Small: Distributed Storage Systems from Combinatorial Designs

Iowa State University, Ames IA

Investigators

Abstract

Large scale content sharing, social networking sites and cloud storage providers require massive amounts of data (on the order of hundreds of petabytes) to be stored in a distributed manner. In these systems, node failures and loss of data are the norm rather than the exception. Thus, it is of critical importance to investigate storage techniques and algorithms that (a) allow for data recovery in the presence of node failures and (b) allow for efficient regeneration of failed nodes. It is, of course, desirable to perform the regeneration in a distributed manner and optimize performance metrics associated with it. For instance, it is desirable that the regeneration process be fast and that only a few nodes be contacted for the repair. The investigator will research storage methods that allow for very simple and efficient repair while simultaneously allowing for data recovery in the presence of a large number of failures. The research involves the investigation of distributed storage systems that are resilient to multiple failures and can be repaired in an exact and uncoded manner; i.e., the new node may produce an exact copy of the failed node by simply downloading packets from the surviving nodes, with no computational overhead. Replication codes are increasingly deployed in large-scale data centers in use by major internet companies. Advances in reconstruction codes are expected to translate into millions of dollars in savings in maintenance costs in such data centers. To ensure accessibility of the results of this research, implementation will be carried out using the open source Hadoop distributed file system, the preferred platform for cutting-edge research and development in distributed cloud storage.