NeTS: Medium: From Packets to Insights: Programmable Streaming Analytics for Networks

Princeton University, Princeton NJ

Investigators

Abstract

The ability to monitor Internet traffic on our communications networks is of critical importance to our nation's economic prosperity and national security. For communications networks to run well, however, network operators must be able to manage them: they must be able to detect, diagnose, and fix problems that degrade the performance of the applications we use, and they must be able to detect and mitigate attacks against the infrastructure.  To ensure that computer networks are secure and perform well, network operators need to gather measurements to detect attack traffic, diagnose performance problems, identify flaky equipment, drive traffic-engineering decisions, and more. Although network devices provide reasonable mechanisms for monitoring the control plane -- that part of the network that is responsible for routing packets/information through the network.  Tools and mechanisms for monitoring the flow of network traffic remain primitive (e.g., ping and traceroute for active measurement, Netflow and sFlow for passive measurement). These measurements provide coarse statistics about network traffic or conditions, but they provide at once both too little information (because they obscure important details about the flows, such as packet timings, queue sizes, and loss rates) and too much information (because, for any particular question about performance or security, the operator needs detailed information about a few flows, as opposed to coarse information about all of them). This project aims to develop measurements that are "just right" for each of the above tasks and to design a data-analytics platform for querying the data that can be used to diagnose and mitigate network problems.   Two technological trends enable fundamentally new paradigms for network measurement. The first trend is the rise of programmable network hardware -- including reconfigurable application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and network processors -- that are fast and inexpensive enough for use in commodity switches, and also programmable in target-independent languages like P4. The second trend is the emergence of scalable streaming analytics platforms, such as Spark Streaming and Apache Storm. These platforms make it possible to express queries based on streams of tuples and efficiently filter and aggregate the data. Using the programmable functionality from switches, one can define the types of tuples that a switch exports, and even perform simple computations over the tuples directly in the data plane. Given input tuple streams from one or more switches, the stream processor can compute the answer to a high-level query. This project is developing a streaming analytics framework that addresses these challenges. The researchers will develop a query language with familiar programming paradigms from existing streaming analytics platforms, which they will extend to support domain-specific primitives. They are also developing a runtime system that partitions this query across the stream processor and the switches in the data plane. Queries will entail network-wide aggregation, iterative "drill down" capabilities, and joins with external data sources (e.g., routing, application identification). The researchers are evaluating the feasibility and usability of this platform in the context of a wide-range of security and performance diagnosis queries that arise in operational networks.