SI2-SSE: Scaling up Science on Cyberinfrastructure with the Cooperative Computing Tools

University Of Notre Dame, Notre Dame IN

Investigators

Abstract

This project will support the continued development of the Cooperating Computing Tools (CCTools)software, which provides scientists with the ability to distribute existing simulation and analysis software codes to large numbers of computers in commercial clouds, advanced supercomputers, or both. As a result, research in fields such as physics, biology, and chemistry can be accelerated, in some cases, by a hundred- or even a thousand-fold. Staff and students involved in this project will develop new software capabilities and work closely with scientific communities to improve their productivity in advanced computing facilities. The project will also help to develop a technical workforce by training graduate and undergraduate students in advanced software engineering skills. Today's computational scientist has access to an extraordinary range of computing facilities, including thousands of cores and petabytes of storage drawn from commercial cloud providers, university clusters, or national computing resources. However, end users often want to connect these systems in ways that the designers did not anticipate. They may wish to access data at one facility, run software at a second facility, and store the results at yet another. However, combining infrastructure is much harder than it ought to be, because most resources assume that their users only live within that particular closed world. Programs built in one environment may not run in another; data stored in one system may not be accessible in another; specialized programming models may be fast on one machine, but useless on another machine. Researchers do not want to be tied down to a single system; rather, they want standard programs to execute easily across all environments. This project will create new features and support users of the Cooperating Computing Tools (CCTools) software which provides scientific users with the ability to harness large scale cyber-infrastructure for data intensive scientific applications. The key components are the Parrot virtual filesystem, the Makeflow workflow system, and the Work Queue application framework. These components can be used separately or together to take existing POSIX applications and scale them up from a single laptop all the way to national scale infrastructure. This project will advance the development of the CCTools software to meet the changing technology landscape in three key respects: exploiting container technologies, making efficient use of local concurrency, and performing capacity management at the workflow scale. This grant will support a small team of students and staff who will engage user communities to understand their needs, sustain and extend the software on current cyberinfrastructure, and perform outreach and education to users both new and old. The project will focus on active user communities in high energy physics, which rely on Parrot for global scale filesystem access in campus clusters and the Open Science Grid; bioinformatics users executing complex workflows via the VectorBase, LifeMapper, and CyVerse disciplinary portals, and ensemble molecular dynamics applications that harness GPUs from XSEDE and commercial clouds.