ITR: DataMotion - Dealing With Fast-Moving Data

Stanford University, Stanford CA

Investigators

Abstract

The goal of this project is to build a new infrastructure, called DataMotion, for managing and analyzing large volumes of dynamic and diverse data. Today, most information systems ? even those handling multiple, distributed, heterogeneous data sources ? are based on stored and relatively static data sets. However, much of the most critical information today is highly dynamic, coming instead in the form of multiple, rapid, time-varying data streams. The volume is so high that it is difficult to store all the information in conventional databases. A much better approach is to route the stream to the interested users, while in the process filtering according to user's interests, combining the stream with other relevant streams, and performing real-time analysis of the data whenever possible. DataMotion enables such distributed, real-time processing. This project will consist of 4 inter-related research thrusts: (a) How to perform traditional database and data mining operations on streams; (b) How to generate streams, and how to present streams to users; (c) How to route streams to distributed users with differing information needs; and (d) How to ensure the security and privacy of streams.