Virtual Data Set Services Enabling New Science at NSF Facilities

University Of Chicago, Chicago IL

Investigators

Abstract

Scientific facilities supported by the National Science Foundation such as the Daniel K. Inouye Solar Telescope (DKIST), National Center for Atmospheric Research (NCAR), and National Ecological Observatory Network (NEON) collect enormous quantities of valuable data about the world in which we live. These data can be used for scientific and societal benefit: to make breakthrough discoveries about sun's behavior and magnetic field, and our changing environment; to improve the speed and accuracy of forecasting of severe storms and destructive wildfires; to predict disruptions to electrical systems from solar flares; and many other purposes. Before such vital scientific data can be used effectively, they must be delivered rapidly, efficiently, and reliably to the people who need them. Researchers need interactive community access to hard-to-obtain data located in large data archives. Because the NCAR, DKIST, and NEON data archives cannot feasibly provide the computing resources needed for all analyses, end-user scientists need to be able to define, navigate, download, and analyze data subsets. Current web-based tools are not up to these tasks. The Virtual Data Set Services Enabling New Science at NSF Facilities project will tackle this challenge by developing new methods for organizing, packaging, and rapidly transporting data. A key innovation will be the development of methods for defining, sharing, and manipulating "virtual data sets," data collections extracted "on the fly" from the vast holdings of scientific facilities for a specific purpose. A researcher may define a virtual data set much as a shopper assembles products in an online "shopping cart." Once defined, a virtual data set can then be transferred to a remote computer for analysis, shared with colleagues, or extended for future projects. We will develop new services to (a) enable definition of, navigation over, and selective access to virtual data sets from petascale data archives of the scientific facilities, and (b) ensure reliable, automated, efficient, and secure replication and access of entire data sets or data subsets between a petascale data archive and other locations, to include both end user computers and remote mirrors intended to accelerate data access by community members. These new services will be constructed on top of the Globus platform, already heavily used within NCAR's Research Data Archive (RDA) and many other research data centers. The ultimate aim is to integrate the new services into operational systems in collaboration with DKIST, NEON, and NCAR/RDA. The results will be evaluated in the context of demanding science applications in partnership with solar physics, atmospheric science, and ecology researchers. This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer and Information Science and Engineering. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.