III: Large: Discovering Complex Anomalous Patterns

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Many of the most interesting and valuable discoveries that can be made from data arise not from the evaluation of single records, but from identifying a set of records that are anomalous in some interesting way. Together they may indicate for example the emergence of a disease outbreak or new patterns of criminal activity. One can view pattern discovery as an interactive process between data analysis algorithms and human users who have expertise in the domain. This research will develop an integrated framework of probabilistic methods to interact with the user in detecting, characterizing, explaining, and learning anomalous patterns over groups of records. The focus is on the many situations where the data (and the probabilistic patterns to be discovered) are not appropriate for using other existing techniques, such as graph mining or frequent sets. The proposed methods will search over arbitrary subsets of records and evaluate their correspondence to known, potentially very complex, probabilistic patterns, or their failure to match baseline data under various learned statistical models. These methods will assist the user in understanding and modeling the discovered, previously unknown anomalies to be identifiable as a known pattern when encountered in the future. Intellectual Merit This collaborative team of researchers will develop, implement, and evaluate a general, comprehensive, and widely applicable probabilistic framework for pattern discovery. The proposed work will address these challenging and important research questions: - How can machine learning concepts such as classification and anomaly detection be generalized to consider groups of records rather than single records? - How can a detection algorithm simultaneously detect and differentiate between known and currently unknown pattern types? - How can an algorithm explain clearly to a user what pattern was found and why? - How can an algorithm learn new pattern types through feedback from a user? The ability to detect, characterize, explain, and learn patterns from groups of records in massive datasets will provide a qualitatively new approach for advancing discovery of knowledge from data. Broader Impact Although the applications for these algorithms are innumerable, development and testing will be prioritized in the areas of patient care in the intensive care unit (ICU) and aircraft fleet maintenance. Through the team's existing collaborations, the algorithms will also be used during the project in other areas including food safety, scientific discovery in astronomy sky surveys, and detection of geographic hot-spots of criminal activity. Together, these applications will demonstrate the methods' value across a wide spectrum of domains and tasks. Key Words: anomalous patterns; pattern discovery; probabilistic models; incremental learning.