OPTIMIZATION OF SPIDER FOR PARALLEL MACHINES

University Of Calif-Lawrenc Berkeley Lab, Berkeley CA

Investigators

Linked publications & trials

Abstract

SPIDER is one of the most frequently used packages in EM-related single particle processing, currently in use by at least 55 research groups. This project addresses two needs encountered in striving for higher resolution: the need for massively parallel and distributed processing, and the need for overall optimization and automation of the processing flow. Both needs are being addressed in a way that is platform-independent and likely to benefit a large user community. SPIDER is currently parallelized on a fine-to medium-grain level of computing, making use of OMP (open multiprocessing) directives for Shared Memory Processing placed within the SPIDER code. One of the aims of this proposal is to add a high-level, coarse-grain parallelism at the process or SPIDER job level, which does not require shared memory. Processing will be partitioned into independent jobs on individual images, sets of images, or stacks of images. These jobs will be coordinated and scheduled on available processors through a "publish and subscribe" system. This approach has several advantages, including exhaustive yet flexible use of available resources and easy portability to a large range of platforms. Overall optimization and partial automation of large, complex tasks formulated in SPIDER procedures will be achieved as part of the Reconstruction Engine project, which forms the second Specific Aim of this proposal. The development of the Reconstruction Engine will be tightly coordinated with the development of parallel processing under Specific Aim #1. The work will be coordinated with the team that develops SPARX (Project B) by developing interchangeable modules that facilitate ongoing comparisons of software performance. Interactions with the teams in Projects C, D, and E are foreseen to further enrich the capabilities and increase the efficiency of the software serving the Structural Biology Community.