Small: The MCDB Database System for Managing and Modeling Uncertainty

William Marsh Rice University, Houston TX

Investigators

Abstract

The MCDB Database System for Managing and Modeling Uncertainty Analysts working with large data sets often use statistical models to ``guess'' at unknown, inaccurate, or missing information associated with the data stored in a database. For example, an analyst for a manufacturer may wish to know, "What would my profits have been if I'd increased my margins by 5% last year?" The answer to this question depends upon the extent to which the higher prices would have affected each customer's demand, which is undoubtedly guessed via the application of some statistical model. The MCDB project is concerned with the design and implementation of a prototype database system called the "Monte Carlo Database System," or "MCDB" for short. MCDB allows an expert-level analyst or statistician to attach arbitrary stochastic models to the database data in order to "guess" the values for unknown or inaccurate data, such as each customer's unseen demand function. These stochastic models reside in the database, and are always up-to-date in the sense that they are parameterized on the current state of the database (using each customer's most recent purchases in the above example). The project attacks a number of key intellectual and scientific challenges. Most of these are related to the fact that for performance reasons, it is not possible to materialize one thousand stochastic instances of a one terabyte data warehouse, and query each of them in sequence. Novel methods for avoiding such materializations are being considered, such as skipping Monte Carlo trials that produce data which will never be used to answer a specific query. The project also considers statistical challenges, such as generating database instances that fall far out in the tail of the answer distribution, which is necessary for specific applications such as risk assessment. Further information is available at http://mcdb.cs.rice.edu.