CCSS: Small: Universal Feature Selection in Integrated Monitoring of Large Networks

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

The key challenges of monitoring and managing large networks comes from two sources. First, the network measurement data is multi-modal, with data of different forms, qualities, meanings, and overall several orders of magnitudes higher dimensionality than the current techniques can jointly processing. Second, such network monitoring results are often needed to serve multiple purposes, to answer queries about different aspects of the network behaviors, to understand at a microscopic level of how the interconnected components interact. The goal of this project is to apply a new geometric approach of feature selection to reduce the high dimensional network measurement data into a much more manageable lower dimensional feature space, where efficient algorithms can be applied to understand network behaviors, identify suspicious activities, and optimize network resource allocations. The success of this project can lead to new interfaces to network status, better network operations, as well as new approaches to improve cyber security. The main technical contribution of this project is based on a novel formulation of "universal feature selection", with the goal of finding features from the data that is universally informative to a collection of possible queries. A geometric-based analysis method is developed to give quantitative solutions to such problems. Comparing to the tradition information metrics that only quantify the volume of information, the geometric framework helps to give rise to a new notion of "information vector", which can be used to quantitatively measure the meaning of a piece of information and the relevance to a specific query. Feature selection based on such a geometric structure can thus connect the information processing operations to these new information metrics, providing guidance of optimization and theoretical guarantees of the performance. The project is thus based on a theoretic framework that is general, connecting several different branches of statistics. The network monitoring problem is a particularly