III-COR-Medium: Design and Implementation of the DBO Database System

William Marsh Rice University, Houston TX

Investigators

Abstract

Commercial and government entities now spend around $10 billion per year on software and hardware systems for managing "data warehouses", which are very large electronic data archives. Despite the size and importance of this marketplace, existing data management solutions can be painfully slow (see, for example, www.tpc.org/tpch/ for recent benchmarking results). It is now possible to spend millions of dollars of hardware and software for a system that still takes hours to answer simple analytic questions. This is unfortunate, because there is much knowledge to be gained by interactive exploration of electronic archives. Very long processing times make it likely that the data will stored away, and never looked at again. The DBO Database System project is concerned with the design and development of a unique system called DBO. Like traditional relational database systems, DBO can run database queries from start to finish and produce exact answers over very large archives. However, unlike any existing research or production system, DBO uses sampling algorithms to produce a statistical estimate for the final query answer at all times throughout query execution. An example of the sort of estimate produced by DBO is, "There is a 95% chance that the true answer is between $1.75 million an $1.80 million". The longer a user waits, the more accurate the estimate becomes. The potential benefit of such an estimate is that a user can stop execution whenever satisfied with the accuracy of the estimate, which may translate to dramatic time savings during exploratory processing. In this way, the goal of the DBO project is to render interactive data analysis a reality, even over the largest databases. All scientific and technical materials produced by the project, as well as any software available for download, can be obtained from http://www.cise.ufl.edu/~cjermain/DBO .