ATD: Collaborative Research: Algorithms and Data for High-Frequency, Real-Time Anomaly Detection

New York University, New York NY

Investigators

Abstract

The rapidly burgeoning amount of digital data from Internet and mobile-enabled communications can offer low-cost and high-resolution views into human behavior across areas such as health and socio-economics. Personally-generated data from Internet and mobile-connected sources offer unique insight, capturing aspects of human behavior that would be taxing or impossible to quantify through other data sources. Moreover, the data is often available in real time and can be linked to specific locations. This research project addresses the statistical challenges inherent in using such unstructured spatio-temporal data sets for detection of anomalous events. Such data requires new statistical approaches to pre-process and extract forms from the data that can reliably be used for event detection. Further, the continuous nature of the data means that what constitutes anomalous behavior depends on the time-scale and on the type of underlying event. This project aims to develop 1) approaches for generating relevant features from social media data that account for the observational nature of the data and can be used in spatio-temporal models of real-world behavior and 2) a new multi-scale approach to modeling dependence structures that uses new information to continuously refine the model and accurately assess anomalies. The approach in this project is both suited to and harnesses the continuous and observational nature of social media data. The research will be validated on empirical data sets, demonstrating practical utility. It is anticipated that the results will be applicable to further the use of publicly-available geospatial data sources and understand human dynamics that are not measurable through other means.