NSF-BSF:CIF:Small: Searching for the Rare: an Active Inference and Learning Approach

Cornell University, Ithaca NY

Investigators

Abstract

This project addresses the problem of searching for a few rare events of interest among a massive number of possibilities. The rare events may represent opportunities with exceptional returns or anomalies associated with high costs or potential catastrophic consequences. This problem arises in a broad range of applications, ranging from communications and infrastructure systems, cyber-security, to social-economic networks, and is particularly relevant in the era of increasing network size and abundance of data. The multidisciplinary nature of this project also provides a rich research experience for both undergraduate and graduate students. The scientific objective is to develop general design methodologies for detecting rare events quickly and reliably when the total number of hypotheses is large, the observations are noisy, and the prior knowledge on the rare events may be as little as "they are different from the nominal." The project consists of three steps that represent a logical progression in scope and level of difficulty: (i) achieving optimal sample complexity with respect to detection accuracy through active hypothesis testing; (ii) achieving optimal sample complexity with respect to the dimension of the search space by exploiting inherent hierarchical structures; (iii) tackling unknown models by integrating online learning with active inference. The holistic treatment within a principled theoretic framework draws inspirations from and contributes to fundamental theories of active hypothesis testing and statistical and machine learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.