CSSI: Frameworks: Applying Artificial Intelligence Advances to the Next Generation of Workflow Management on Modern Cyberinfrastructure

University Of Southern California, Los Angeles CA

Investigators

Abstract

Scientific research and engineering innovations increasingly depend on complex workflows that require coordination across many types of computing resources, ranging from small edge devices to powerful supercomputers. However, the growing complexity of these workflows and the distributed nature of the underlying computing and data cyberinfrastructure (CI) make it difficult for many researchers and engineers to fully access, manage, and benefit from these advanced technologies. The goal of iWMS is to apply new advancements in Artificial Intelligence (AI) techniques to design and implement a workflow management framework that empowers researchers, engineers, and educators to take advantage of the computing continuum for scientific discovery, innovation, and education. The framework promotes progress in science and engineering, powering applications that advance national priorities. The project designs and implements iWMS, an open, modular workflow management framework that integrates AI across the entire workflow lifecycle. In particular, iWMS incorporates AI models to support automated workflow composition, intelligent resource provisioning, performance prediction, real-time anomaly detection, and workflow adaptation. Techniques such as retrieval-augmented generation facilitate workflow discovery and assembly, while machine learning-based planning and monitoring services optimize execution and enhance system reliability. With the framework foundation built on a modular and integrated design, iWMS incorporates new AI methods as they are being developed in academia and industry. iWMS services feature documented APIs, enabling CI developers to contribute to and enhance iWMS over time. The framework is evaluated with real-world scientific applications and deployed across national testbeds and cyberinfrastructure platforms. In addition to delivering robust software tools and AI models, the project produces training materials, sample workflows, and public datasets to support adoption, foster community engagement, support AI research for enhancing CI, and advance the usability and resilience of the national cyberinfrastructure ecosystem. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.