CC* Integration-Large: SciStream: Architecture and Toolkit for Data Streaming between Federated Science Instruments

University Of Chicago, Chicago IL

Investigators

Abstract

Scientific instruments are capable of generating data at very high speeds. However, with traditional file-based data movement and analysis methods, data are often processed at a much lower speed, leading to either operating the instruments at a lower speed or discarding a (significant) portion of the data without processing it. To address this issue, SciStream project will develop software tools to stream data at very high speeds from scientific instruments to supercomputers at a distant location. SciStream hides the complexities in network connections from the end user and provides a high level of security for all the network connections. The data producers (e.g., data acquisition applications on scientific instruments, simulations on supercomputers) and consumers (e.g., data analysis applications on high performance computing systems) may be in different security domains (and thus require bridging of those domains) and may, further, lack external network connectivity (and thus, require traffic forwarding proxies). SciStream establishes necessary bridging and end-to-end authentication between source and destination, while providing efficient memory-to-memory data streaming. Through the exploration of architectural and design choices and addressing issues of control protocols and security, SciStream will advance the understanding of the challenges in supporting high speed memory-to-memory data streaming between remote instruments in federated science environments. SciStream will benefit all scientific applications that require memory-to-memory data streaming between distributed instruments. Recent trends suggest that this is an important and growing requirement for many scientific applications. SciStream will help significantly reduce the time to solution for these applications, resulting in improved scientific productivity and thus far-reaching benefits for society. Key design choices such as application-agnostic streaming and support for best-effort streaming will make SciStream appealing to a broader science community. SciStream will engage with domain scientists, campus computing centers, and a scientific user facility to reach a wider audience. Through on-campus programs at the University of Chicago, SciStream will train under-represented students in networking. Additional details on SciStream can be found here: https://scistream.github.io/ This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.