CC* Integration: Service Analysis and Network Diagnosis (SAND)

University Of Nebraska-Lincoln, Lincoln NE

Investigators

Abstract

Increasingly, science has become a "team sport" with projects less likely to be conducted by a single investigator or university group but with multi-institution collaborations spanning many universities and national laboratories. As these collaborations have become data-intensive, highly-performant network links connecting their distributed science platforms have become increasingly important. While true across all science, some collaborations, like those at the Large Hadron Collider (LHC) at CERN, transfer millions of gigabytes across global networks today and anticipate orders of magnitude increases within the next decade. Without performant networks and efficient data transfer and access services, the time to science will be greatly compromised. In this context, high-speed, long-distance networks see their performance degrade surprisingly quickly in the face of modest error rates. Accordingly, network engineers and researchers use sophisticated tools to monitor the network and transfer services. Without a means to aggregate and correlate network tests, performance measurements, and application response, they at best can only reveal a small piece of the overall problem. This project focuses on techniques that better combine, visualize, and analyze disparate network monitoring and service logging data, providing a comprehensive picture critical to the engineers and scientists relying on the network. This will allow problems to be located and fixed more quickly, reducing the time to science. The "CC* Integration: Service Analysis and Network Diagnosis (SAND)" project brings together an experienced team that has been working for more than a decade on wide area data transfers for large scale science. The project develops a network monitoring archive and analytics platform, SAND-NMA, which integrates widely used data analytics tools (such as ElasticSearch, Kibana and JupyterLab) with infrastructure components (perfSONAR, HTCondor) and application sensors. Data from disparate sources are published to a messaging bus providing low-latency metrics describing the performance of research platforms on a global scale. Exploratory work is performed to provide engineers with pragmatic tools to identify and locate problems and perform analytics to understand the long-term evolution of network and higher level service performance. Programming interfaces allow external cyberinfrastructure (such as workflow management systems) to incorporate the network monitoring feeds into their decision making engines. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.