Collaborative Research: SI2-SSI: Adding Volunteer Computing to the Research Cyberinfrastructure

University Of California-Berkeley, Berkeley CA

Investigators

Abstract

The aggregate computing power of consumer devices - desktop and laptop computers, tablets, smartphones - far exceeds that of institutional computing resources. "Volunteer computing" uses these consumer devices, volunteered by their owners, to do scientific computing. In addition to providing additional, much-needed computational resources to scientists, volunteer computing publicizes scientific research and engages citizens in science. BOINC is the primary software system for volunteer computing. It was developed at UC Berkeley with NSF support starting in 2002. Until now, BOINC has been based on a model of independent competing projects. Scientists set up their own BOINC servers, port their applications to run on BOINC, and publicize their projects to attract volunteers. There are about 40 such projects, in many areas of science: examples include Einstein@home, CERN, and SETI@home (astrophysics), Rosetta@home and GPUGrid.net (biomedicine), Climateprediction.net (climate study), and IBM World Community Grid (multiple applications). Together these projects have about 400,000 active volunteers and 12 PetaFLOPS of computing throughput. This model, while successful to an extent, has reached a limit. The number of projects and volunteers has stagnated. Volunteer computing is supplying lots of computing power, but only to a few research projects. For other scientists, there are two major barriers. First, creating a BOINC project has significant overhead: learning a new technology, creating a public web site, generating publicity, and so on. Second, volunteer computing is risky and uncertain; there is no guarantee that a new project will attract volunteers. This project aims to break this barrier, and to make volunteer computing available to all scientists doing high-throughput computing, by replacing the competing-projects model with a new "central broker" model. The new model has two related parts: 1) the integration of BOINC with existing high-throughput computing facilities such as supercomputing centers and science portals. Jobs currently run on cluster nodes will be transparently offloaded to volunteer computers. Scientists using these facilities will see faster turnaround times; they'll benefit from volunteer computing without even knowing it's there. 2) The project will change the volunteer interface so that participants sign up for scientific areas and goals rather then for particular projects. For example, a participant might sign up to contribute to cancer research. A central broker, to be developed as part of this project, would dynamically assign their computing resources to projects doing that type of research. This project mobilizes public support for and interest in scientific research by encouraging "volunteer computing" and engaging citizens in the conduct of the research itself. It simultaneously advances NSF's mission to advance science while broadening citizen engagement. The first year of this project will prototype each of these parts, and will integrate BOINC with TACC and nanoHub. Integrating BOINC with existing HTC systems involves several subtasks: 1) Job routing: modifying existing job processing systems used by TACC and nanoHub (Launcher and Rappture respectively) to decide when a group of jobs should be offloaded to BOINC. This decision might involve the estimated runtime of the jobs, input and output file sizes, data sensitivity, the deadline or priority of the jobs, and the identity of the job submitter. 2) Job format conversion: mapping job descriptions (input/output file specifications, resource and timing requirements) to their BOINC equivalents. 3) Application packaging: adapting existing applications (such as nanoHub's simulation tools and TACC's Autodock) to run under BOINC. We will use BOINC's virtual machine facility, which packages an application as a virtual machine image (VirtualBox or Docker) and a program to be run within the VM. This allows existing Linux applications to run on consumer desktop platforms such as Windows and Mac, as well as providing a strong security sandbox and an efficient application-independent checkpoint/restart mechanism. 4) File handling: moving input and output files between existing storage systems (typically inaccessible from outside firewalls) to Internet-visible servers. This will use existing BOINC components that manage files based on hashes to eliminate duplicate transfer and storage of files. 5) Job monitoring and control: adapting existing web- or command-line based tools for monitoring the progress of batches of jobs, and for aborting jobs, to work with BOINC. This will use existing Web RPCs provided by BOINC for these purposes. This project will carry out these tasks by designing and implementing new software as needed, testing for correctness, performance, and scalability, and deploying it in a production environment. The second part of the project - a brokering system for allocating computing power based on volunteer scientific preferences - will be designed and prototyped. This involves several subtasks: 1) Designing a schema for volunteer preferences, including scientific areas and sub-areas, project nationality and institutions, specific projects and applications, inclusions/exclusions, and so on. 2) Designing a schema for assigning attributes to job streams (e.g. their area, sub-area, institution, etc.), and for assigning quotas or priorities to job streams. 3) Designing a relational database for storing the above information. 4) Designing and implementing policies for assigning volunteer resources to job streams in a way that respects volunteer preferences and optimizes quota, fairness, and throughput criteria. This will be implemented as a BOINC "account manager" so that volunteers see a single interface rather than lots of separate projects and web sites.