ITR Collaborative Proposal: Aurora - Enabling Stream-Based Monitoring Applications

Massachusetts Institute Of Technology, Cambridge MA

Investigators

Abstract

There is a host of existing and newly-emerging applications that require sophisticated and timely processing of fast, high-volume data streams. Examples of these applications include environmental monitoring, portfolio management, asset tracking, and traffic management. These applications, often referred to as monitoring applications, track data from numerous continuous data streams (coming from such sources as sensor networks and stock feeds), filtering them for signs of abnormal activity, and processing them in a timely fashion for purposes of aggregation, reduction, and correlation. Currently, monitoring applications are commonly implemented using custom, application-specific code - there is no general-purpose software infrastructure that can efficiently and effectively meet the processing and performance requirements of these applications. The project will investigate the key issues related to the design and architecture of general-purpose data-stream processing systems by developing proper abstractions, mechanisms, policies, and protocols. In particular, the project will develop novel stream-oriented languages, query execution and optimization techniques, and resource allocation algorithms. The project will also address the challenges and opportunities that arise in distributed stream processing by addressing issues such as stream-oriented load sharing, dynamic operator placement, and high availability. In both cases, the algorithms will be driven by the Quality-of-Service expectations of the target applications.