STCI: Development of Stork Data Scheduler for Mitigating the Data Bottleneck in Petascale Distributed Computing Systems

Louisiana State University, Baton Rouge LA

Investigators

Abstract

This proposal will be awarded using funds made available by the American Recovery and Reinvestment Act of 2009 (Public Law 111-5), and meets the requirements established in Section 2 of the White House Memorandum entitled, Ensuring Responsible Spending of Recovery Act Funds, dated March 20, 2009. The STCI Development of Stork Data Scheduler for Mitigating the Data Bottleneck in Petascale Distributed Computing Systems project will enhance the Stork data scheduler to mitigate the end-to-end data handling bottleneck in petascale distributed computing systems and make it available for a wide range of user community as production quality software. New functionalities of Stork will include: data aggregation and caching; early error detection, classification, and recovery; integration with workflow planning and management; optimal protocol tuning; and data transfer performance prediction services. Intellectual Merit: The Stork data scheduler will make a distinctive contribution to petascale distributed computing in the areas of planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked I/O for petascale distributed applications. Unlike existing approaches, it will treat data resources and the tasks related to data access and movement as first class entities just like computational resources and compute tasks, and not simply the side effect of computation. Broader Impact: This project will impact not just traditionally compute intensive disciplines from science and engineering, but also new emerging computational areas in the arts, humanities, business and education. The PI will be collaborating with other leading institutions in the area of distributed data management such as LBNL, ISI/USC, UNC, UCSD, and University Chicago/Argonne to integrate Stork with their data management solutions and disseminate it to their user communities. The comprehensive education component of the project will include science projects and summer training camps on data-intensive computing with K-12 students (where 99% are minority students), undergraduate and graduate student training, international student/intern exchange program, and minority workshops.