ITR: A Petabyte in Your Pocket

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

In 2015, for a few hundred dollars a year, everyone will have a personal petabyte database (PetDB) that can be accessed from any point of connection, with any device from a high-powered workstation to a PDA. Each individual's PetDB will be their evolving and customized view of all the on-line digital data that exists anywhere. It will store and organize any kind of digital data, without losing structure or information. All this data will be queryable and arranged by type, content, structure, association and multiple categorizations and groupings. The PetDB is an example of what could be done with a new generation of software infrastructure we term Net Data Managers (NDMs). The object of this research is not to build a PetDB per se; but to design and implement the NDM technology upon which PetDBs and other applications could be readily built. NDMs are a departure from current database management systems. They focus on data movement, rather than data storage and must handle data of arbitrary types, without necessarily having a matching database schema. They must execute queries over tens of thousands of information sites, while monitoring possibly millions of triggers over rapidly changing information sources. The first exploratory steps have been taken towards NDMs with initial work on XML querying, text in context indexing and searching, multi-trigger planning and data stream processing. This work has demonstrated some of the basic capabilities of NDMs. Continued research will address the challenging fundamental problems that must be solved to extend these capabilities to exploit the full capabilities of NDMs. At the same time, the scale at which NDMs operate will be extended, in terms of numbers of users, tasks and sites, as well as data volumes, through distributed and parallel implementations.