Constructing Probability Models for Large Corpora of Well-Informed but Probabilistically Incoherent Judgments

William Marsh Rice University, Houston TX

Investigators

Abstract

This interdisciplinary team (Psychology, Computational and Applied Mathematics, Statistics, Economics, Computer Science) has a goal of overcoming the weaknesses in inference engines in expert systems, decision support systems or in knowledge discovery systems that often work in environments with uncertain characteristics. The current techniques rely on Baysian theory and do not perform well in situations in which conditional independence cannot be guaranteed, or the probabilities provided by experts may not be sound. Since inferences based on probability calculations offer the best guarantee of sensible assessments of chance, efficient schemes have been developed for computing probabilities over complex event spaces. Underlying all such algorithms is a "probability model," i.e., a representation of the chances of various combinations of events. In turn, probability models are constructed from an initial set of facts about uncertainty in the environment. These facts can sometimes be extracted from databases, using relative frequency as probability. Often, however, the needed probabilities must be obtained from an expert, who responds intuitively. Reliance upon experts raises the specter of incoherence, i.e., judgments that cannot be reconciled with any probability model at all. Indeed, maintaining coherence across a large set of judgments is both computationally and psychologically taxing, and seldom achieved. Incoherent judgment on the part of a single judge is compounded when it is desired to integrate the opinions of several judges. To exploit potentially incoherent and inconsistent judgments, special optimization algorithms are used to construct a compact probability model that best approximates all the judgments in play. The algorithms are tested by applying them to a body of expert opinion in some complex domain. Development of the algorithms will facilitate the automatic construction of artificial expert systems. Whenever a body of expert judgment can be assembled, the algorithms can be applied in view of creating a compact representation of the collective wisdom of the judges. The results of the theoretical research on the probability models will be applied to the analysis of air quality policy in Houston. A large probabilistic database will be established by culling measurements from air quality control stations around the city, expert judgements in environmental science and medicine, and the output of econometric and environmental models of the region. The project has the potential to have a significant intellectual impact in probability, applied learning, and datamining research communities and also provide a useful tool to environmental researchers and Houston decision-makers. http://www.ruf.rice.edu>/~osherson