OAC Core: RINAS: Data I/O CyberInfrastructure for Extreme-scale Foundation Model and Generative AI Training on HPC

University Of Virginia Main Campus, Charlottesville VA

Investigators

Abstract

The RINAS project researches techniques to enable modern AI to be trained using fewer computer resources. Large Language Models (LLMs) have significant computer resource demands, and all modern AI applications across different areas and with various data types will exhibit similar demands on the underlying cyberinfrastructure. AI training is unusually sensitive to the performance of the data center disks on which training data is stored. This is due to the large data sizes, which prevent in-memory storage and necessitate training involving many iterative epochs, during which the full dataset must be read and processed. The project will use and extend an open-source data formatting and compression system developed by the Apache Foundation. Furthermore, it will utilize traditional optimizations, such as parallel computing (where multiple jobs run simultaneously), and modifications to the AI (Artificial Intelligence) models that accelerate training without significantly compromising model accuracy. AI itself will be utilized to enhance the performance of these jobs further. The new software developed will be deployed on supercomputers run by the National Science Foundation (NSF) and the Department of Energy. It will allow the development of new AI models that will benefit the nation across both industry and academia. The project will collaborate with the AI development and use community through three alliances, each having over 100 organizational members from industry, academia, and government. The project, RINAS, is a new open-source software library designed to solve data input/output (I/O) bottlenecks in large-scale deep learning. RINAS will integrate with popular AI frameworks like PyTorch and TensorFlow and build upon Apache standards like Parquet and Arrow to enhance efficiency. Early prototypes have shown a 50% increase in end-to-end throughput and a 40x improvement in I/O performance. The project will be executed over three years in six stages. RINAS I & II will establish a highly efficient and scalable data pipeline utilizing parallel I/O and novel data shuffling techniques for training large language models. Prototypes have been described in two papers. RINAS III & IV will expand support to include a rich set of data types, such as text, sequences, and images, and provide an interface to existing scientific formats, including HDF and NetCDF. RINAS V will incorporate intelligent AI methods, such as data-efficient training that prioritizes high-quality data for resampling. RINAS VI will focus on integrating the library into larger systems used in Industry and Academia. The University of Virginia will work with industry collaborators, National Government-supported high-performance computing (HPC) providers, and other universities. While primarily aimed at AI, RINAS will also benefit large-scale scientific simulations requiring high-performance data processing. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.