NSCI: SI2-SSE: An Extensible Model to Support Scalable Checkpoint-Restart for DMTCP Across Multiple Disciplines

Northeastern University, Boston MA

Investigators

Abstract

Checkpointing is a technique that periodically saves the state of a long-running computer program to disk. If a computer crash occurs during the running of the program, one can then restart the program state from a previously saved "checkpoint" file on disk. The goal of this project is to discover, implement and deploy novel techniques for adapting checkpointing so as to provide a more robust capability easily usable across applications supporting the research of a variety of scientific and engineering disciplines. In particular, a problem with the classic (transparent) checkpoint model is that these packages do not model, and hence cannot recreate upon restart, communications between the original program and other external processes or programs. In this project, a virtualization model for commonly used mechanisms for communication will be developed so that on restart, external communications are emulated. Checkpointing is used across academia, industry, and government, particularly by those with long-running high performance computing programs. Thus, the project outcomes have broad applicability and value. The project has the added benefit of educating the next generation of students in valuable and highly transferable system skills. Today, transparent checkpoint-restart today is used primarily for fault tolerance, and primarily in closed systems with no external communication. DMTCP is a twelve-year old open source checkpointing project. Its currently evolving process virtualization model of checkpointing enables an application to support complex applications that interact with external subsystems. The project explores and extends a model of process virtualization in order to adapt checkpoint-restart to multiple, novel applications, and to extend its use across multiple scientific and engineering disciplines. Example disciplines that will benefit include: supercomputing (and in particular, forging a path toward practical exascale checkpointing); novel strategies for flexible resource managers (batch queues) for computer clusters that adapt to the current workload; and better support for hardware circuit emulators for Electronic Design Automation (EDA). Example challenges include the need to support transparent checkpointing over the newer low-latency networks such as Omni-Path, integration of application-specific checkpointing with transparent DMTCP-style checkpointing, the need to avoid "flooding" back-end storage during checkpointing in high-end clusters, and new types of resource managers that benefit from the flexibility of arbitrarily suspending running jobs through checkpointing. Rather than build ad hoc solutions for each of the above, this work will provide a simple model allowing end users to easily build their own extensions to support checkpointing of the external subsystems. The simple model will be derived by generalizing over solutions to many of the example challenges described above. In addition to fault tolerance, the technology holds advantages for: fast startup (checkpoint after process initialization,in order to restart and skip this phase in future sessions); debugging (e.g. checkpoint every 30 seconds); reproducible bug reports; extended interactive sessions (e.g. checkpoint before dinner and restart the next day); and so on. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.