Elements: Empowering high-performance remote access to adaptive particle simulation data

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

This project supports the development of new cyberinfrastructure (CI) that enables rapid and efficient access to the massive datasets produced by computer simulations of galaxy formation. Such simulations are crucial for the development of new theories about where our galaxy came from and how it formed; for helping aid the interpretation of data from new telescopes like the James Webb Space Telescope; and for providing predictions for new surveys like the Legacy Survey of Space and Time that is being conducted by the Vera Rubin Observatory. Analyzing the data produced by these simulations is almost as demanding as running them because they produce petabyte-scale datasets representing hundreds of thousands or even millions of galaxies. Luckily, most analysis is only interested in a handful of objects at a time, but no software currently exists to let astronomers retrieve these critical subsets of the data easily or efficiently, leaving analysis siloed in massive high performance computing clusters. Socket addresses this problem, allowing easy selection of data subsets over a standard internet connection, significantly widening its audience. Socket will not only allow more people to do more research with this data more quickly but is also designed to empower modern Artificial Intelligence and Machine Learning-based research. Socket is an open, high-performance data access platform for adaptive particle simulations. It is primarily focused on cosmological galaxy formation simulations, where hundreds of thousands of galaxies are simulated simultaneously to capture their complex interactions. Analysis of these data (where an individual snapshot may be over 10 TB) frequently represents a filtering by galaxy properties to a small subset that may only consist of 100 MB. Data sharing is currently restricted to copying entire snapshots using high-performance networks or (rarely) by allowing a ‘compute to data’ approach. Both approaches are suboptimal; either the data consumer must have masses of storage available, even if their analysis is not compute-intensive, or learn and set up a new computing environment. By leveraging websockets and RESTful Application Programming Interfaces (APIs), socket will provide Findable, Accessible, Interoperable, and Reusable (FAIR) access to this valuable data. Using the socket CI, this project provides rapid access to spatial subsets of this highly adaptive data through the development of a novel, metadata-rich (and hence Machine Learning- and Artificial Intelligence-ready), standard for data storage using HDF5. Socket enables broad community engagement with leading simulation suites by removing traditional access barriers and enabling high-quality, remote analysis workflows, reducing friction and providing a multiplicative effect to productivity. This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Astronomical Sciences in the Mathematical and Physical Sciences Directorate. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.