CSR: Small:Data Staging and Parallel Applications in Robust Desktop Grids

University Of Maryland, College Park, College Park MD

Investigators

Abstract

This project is an investigation into algorithms and mechanisms that will allow sophisticated staging of input and output data in desktop grids. Data staging allows the system to store data semi-permanently in the underlying peer-to-peer structure, and to run multi-node jobs (applications) whether they be tightly-coupled parallel applications, arbitrary work-flows, or anything between. We are building support for these application types by extending our current desktop grid infrastructure in three distinct areas. We are developing cluster identification techniques that can define arbitrarily-sized virtual clusters through both passive and active network measurement. We are incorporating virtual cluster descriptions into the underlying peer-to-peer infrastructure to allow the scheduling algorithms to map multi-node jobs to the clusters. Finally, we are incorporating data placement into the underlying infrastructure; data is placed according to use and process binding. This work will impact several research areas, including that of distributed and decentralized scheduling, application description, network characterization, and storage networks. In all of these areas our work will explore the tension between local autonomy and global, aggregate objectives. The algorithms and techniques will have broad applicability across a wide range of emerging distributed and collaborative applications. However, the work described here will also explicitly and immediately impact the quality of research conducted by our collaborators in astronomy and elsewhere. The ability to run parallel applications, and those with more complicated inter-relationships, will enable whole new classes of scientific applications to be run on top of ad-hoc grid-like systems.