Large Scale Data Transfer: Addressing a Critical Need with a Low Cost Internet2 Service

Internet2, Washington DC

Investigators

Abstract

Since the inception of the Internet, the problem of sending files between users that were too large for most servers has plagued users. Often the simple process of getting the sender and recipient to set up accounts on a common service to allow exchange of their respective identities results delays in actually sending a file, as this information needs to be known before a file exchange can be initiated. Internet2 pursues in this work the creation of a general bulk data transfer service that will allow researchers to stage datasets in the 10MB-1TB size range for short periods of time. An initial data store of 100TB will be provided as a short-term repository, allowing robust implementation to meet the needs of the community as none of the space will be allocated long-term and will be available for re-use quickly. This work focuses on modest functional goals (solving only the problem of exchanging files, rather than their long term storage and sharing) while emphasizing a global reach to work across a large population with no advanced user planning or provisioning. The intellectual merit of this work lies in its unique solution to a problem affecting a broad scope of research communities, particularly those that deal with large file data transfers, and in the value it will offer through a solution that already enjoys international deployment. The existence of a simple but comprehensive solution that is independent of any community infrastructure will be a critical component of the overall R&E cyberinfrastructure. FileSender, the core software to accomplish this, has evolved into an international open-source community that is gaining significant traction. By providing a service based on FileSender in the U.S., this project will capitalize on and extend the potential for this to become a ubiquitous and common solution throughout the international research and education community.