HECURA: Improving Scalability in Parallel File Systems for High End Computing

Clemson University, Clemson SC

Investigators

Abstract

As high end computing systems (HECs) grow to several tens of thousands of nodes, file I/O is becoming a critical performance issue. Current parallel file systems such as PVFS2 and others, can reasonably stripe data across a hundred nodes and achieve good performance for bulk transfers involving large aligned accesses. Serious performance limits exist, however, for small unaligned accesses, metadata operations, and accesses impacted by the consistency semantics (any time one process writes data that is read by another). The proposed research would study scalable metadata operations, small, unaligned data accesses, reliability through redundancy, and management of I/O resources. The employed techniques include active caching and buffering, server-to-server and client-to-client communication, and autonomics. To enhance portability and control complexity proposed research intend to employ middleware whenever possible, and use PVFS2 file system that allows a large degree of configurability. The proposed research intends to enhance that file system so that it will scale to very large sizes.