CRII: OAC: Online Optimization of End-to-End Data Transfers in High Performance Networks

Board Of Regents, Nshe, Obo University Of Nevada, Reno, Reno NV

Investigators

Abstract

With the advancement of computing and sensing technology, the amount of data generated by scientific applications is growing rapidly. To accommodate this growth, high speed networks with up to 400 Gbps capacities have been established. Despite the increasing availability of high-speed wide-area networks and the use of modern data transfer protocols designed for high performance, file transfers in practice attain only a fraction of theoretical maximum throughput, leaving networks underutilized and users unsatisfied. This project aims to develop a real-time transfer tuning algorithm to optimize file transfer throughput in high speed networks. Improved data transfer performance does not only enable efficient execution of distributed scientific applications but also fosters collaboration between scientists at geographically separated institutions by reducing time it takes to share data. This project complements the efforts to build next generation networking infrastructure by offering a novel solution to maximize utilization. The project also facilitates the development of a graduate level high-performance networking course at University of Nevada, Reno, and contribute to the education of undergraduate, female, and under-representative students. Therefore, this research aligns with the NSF's mission to promote the progress of science and to advance national prosperity, and welfare. It is critical to fully utilize available network bandwidth to meet stringent end-to-end performance requirements of distributed scientific workflows. Yet, existing data transfer applications (e.g., scp, bbcp, and ftp) fail to saturate the available network bandwidth due to several factors, such as end system limitations, ill-designed transfer protocols, and poor storage performances. Application-layer transfer tuning offers a comprehensive solution to enhance transfer throughput significantly and can be applied with only client-side modifications. However, finding optimal configuration for application-layer parameters is challenging due to large search space and complex dynamics of network and storage subsystems. This project applies state-of-the-art online convex optimization to application-layer parameter tuning problem as it offers performance and convergence guarantees even under complete uncertainty. In addition to being fast and optimal, online learning algorithms can guarantee the fair distribution of resources among users when combined with game-theory inspired utility functions. The project aims to improve the performance of large streaming applications under dynamic network conditions through anomaly detection and mitigation. It has three unique and innovative aspects: (i) It uses state-of-the art online learning algorithm to fine tune application-layer parameters in real-time. (ii) It improves accuracy and efficiency of sample transfers to minimize the overhead of real-time tuning. (iii) It offers quality of service for delay-sensitive transfers (e.g., high-speed streaming applications) through continuous performance monitoring and adaptive tuning. This project is jointly funded by Office of Advanced Cyberinfrastructure (OAC) and the Established Program to Stimulate Competitive Research (EPSCoR). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.