CSR: Rethinking System Software for Overprovisioned, High-Performance Computing Systems

University Of Arizona, Tucson AZ

Investigators

Abstract

Currently, the high-performance computing (HPC) community is focused on achieving exaflop performance, which is about a 30-fold improvement from the performance of the best supercomputer in the world today. Because of practical, financial, and environmental concerns, the Department of Energy is setting a power limit for achieving an exaflop at 20 megawatts. As today's top machines generally consume between five and 20 megawatts---and yet are an order of magnitude or more away from the exaflop performance target, significant hardware and software advances in HPC systems are necessary. One way to improve hardware is to use overprovisioned systems, which contain more machines than can be fully powered simultaneously. While overprovisioned systems have the potential to significantly improve power and performance, software will need to be redesigned to support such systems. The focus of this proposal is to design and implement software infrastructure that will support overprovisioned systems. The key advance in the infrastructure is support of system-wide optimizations, i.e., optimizations that span multiple applications. This is in stark contrast to the current focus in HPC systems of optimizing on a per-application basis. The developed software will consist of a job profiler, a scheduler that performs analysis on multiple jobs at a time, and a cluster-wide run-time system that jointly optimizes multiple applications based on the output of the scheduler analysis. Achieving exascale computing is an important national priority and will impact many critical application domains, such as climate/weather, renewable energy, nuclear energy, materials science, and national security. The work described here will improve whole-system performance on power-constrained HPC systems, which is one important step towards the exascale goal. The project plans to transfer technology resulting from this research in the form of the proposed software stack via longstanding collaborations with several national laboratories.