Collaborative Research: SI2-SSI: Swift/E: Integrating Parallel Scripted Workflow into the Scientific Software Ecosystem

University Of Notre Dame, Notre Dame IN

Investigators

Abstract

Science and engineering research increasingly relies on repeated execution of a complex series of steps (i.e., workflows) to form hypotheses; conduct experiments; analyze results; and refine theory. Computation is often essential throughout the workflow and in this case, software can improve productivity by managing the computational and data workflow. Swift is one such open-source workflow system that has been developed and widely used in diverse areas ranging from materials simulations and climate modeling to neuroscience and genomics. This project extends the capabilities of Swift by integrating it with other software systems that enable collaboration, usability, maintainability, and productivity. The new ecosystem, Swift/E, will enable scientists and engineers to more productively create and run computational workflow campaigns of larger scale, and debug, execute, adapt, and disseminate them faster and easier than has been possible to date. These workflows embody and communicate the computational methods specific to each domain of scientific inquiry. Swift/E achieves community engagement and extensive productivity benefits for a large user community through an integrated program of research, education, and software dissemination. The project engages and serves science and engineering communities by creating patterns of practice for building and sharing reusable workflow libraries, and by training students, educators, and researchers in their use. To advance the education of the next generation of computationally trained scientists, Swift/E powers a network of NSF-supported "e-Labs" that teach the concepts of collaborative parallel computational science at high school and undergraduate levels, reaching over a thousand students annually. The open-source Swift/E "ecosystem" integrates Swift with several scientific software elements that play a major role in the national and global cyberinfrastructure of today. These elements are: Swift for the parallel scripting of scientific workflow; Globus for data cataloging, management, and high-speed wide-area transport; the Web-based Galaxy workflow portal for workflow composition, execution, and collaborative sharing; Jupyter for the interactive development, testing, debugging, and assembly of high level programming and workflow languages; Python and R for productively expressing high-level computational logic; and "git" and related tools and Web portals for revision control, code dissemination and sharing, and for the collaborative engagement of developers. Swift's implicitly parallel programming language is minimal and compact. Swift provides a facility for embedding other scripting languages (currently Python, R, Julia and Tcl) into its runtime environment. This project merges newer extreme-scale "Swift/T" capabilities with the flexible and portable original "Swift/K" version to make the core Swift/E software element more powerful and flexible while lowering it?s ongoing support cost. Swift/E enhances usability by extending Swift's troubleshooting and inter-language integration facilities. And with enhanced and innovative workflow sharing archives, new training materials, and a sustained program for user support and self-sustaining and expanding community engagement, the Swift/E project engages, supports, and sustains a large global science and engineering user base.