CC* Integration-Small: Network cyberinfrastructure innovation with an intelligent real-time traffic analysis framework and application-aware networking

University Of Nebraska-Lincoln, Lincoln NE

Investigators

Abstract

Intelligent analytics approaches leveraging machine learning techniques offer new capabilities to analyze, model, predict and optimize traffic for high-throughput distributed computing workflows. These techniques can be greatly enhanced by access to real-world data from the edge (campus networks) and the core (Internet2) as well as Just-In-Time (JIT) machine learning approaches. Such a design allows for run-time deployment of the models at the campus cyberinfrastructure to make real-time network decisions. Network flow data collected from these cyberinfrastructures for analysis quickly scales up in size, making it infeasible to perform analysis of network flows in a realistic and timely manner. There are intrinsic difficulties stemming from data storage, its formatting and types as well as the manner in which traditional analysis is done to study network flow data. Although advances have been made in the past several years in how data could be handled efficiently, the new techniques have not been integrated well into the network operations. Improvements need to be made in the way network flow data is analyzed by exploiting the modern data storage formats and the intrinsic properties of the network flow data as well as by developing efficient data structures and algorithms. Recent advances in networking allow for fine-grained network control policies to be managed by network applications. Although it is possible to improve the overall performance of scientific data transfers end-to-end, problems exist with managing resources and differentiating network services at the experiment/site level. Designing and developing intelligent network analysis by JIT machine learning paradigms strengthened by a scalable network flow analysis framework for an application-aware control of the network in high-throughput computing frameworks is the goal of this project. The project is strengthened by collaborations with Holland Computing Center (HCC) at UNL, Open Science Consortium (OSG), Argonne National Lab (ANL) and Internet2. The techniques and frameworks developed in this project will be made available to the open-source community, thus benefiting other science application use cases in Research and Education (R&E) networks. Enriching the education opportunities for UNL School of Computing students and conducting outreach events for the broader community are important objectives of this project. The project aims to transform the current cyberinfrastructure networking approach by (1) gaining insights in real-time by the development and integration of online-offline approaches to machine learning (unlike traditional offline approaches) that can be deployed in data centers for real-time network traffic analysis and prediction; (2) scalable analysis of network flow data by implementing the developed theoretical models for transforming, indexing and building search techniques to study the network flow data at internet-scale in real-time and (3) application-aware control of data transfers by application-aware software defined networking (SDN) control strategies to provide greater flexibility in network management and service differentiation for scientific data transfers on campus cyberinfrastructures. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.