Doctoral Dissertation Research in DRMS: On the Evaluation of Beliefs: A Method for Assessing Credibility in Subjective Probability Judgment

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

Uncertainty is a pervasive feature of the world we live in. In the face of uncertainty, decision makers cannot observe probabilities directly and must instead base their choices on subjective beliefs. These beliefs are often expressed as subjective probability judgments (SPJs), and their quality is constrained by the knowledge of the individuals who hold them. Consequently, decision making under uncertainty requires that decision makers account for the quality of the judgments at their disposal. In many of the areas that concern us the most (e.g., health, safety, and the protection of the environment), however, decisions often hinge on the likelihood of single events such as the success of a surgery, the growth of a mutual fund, or the melting of the polar ice caps. As a result, evaluating SPJs is either impractical or impossible in many areas of decision making. To circumvent this issue, research on judgment under uncertainty has often been forced to evaluate SPJs relative to (a) laboratory situations in which base rates or relative frequencies are known; (b) sets of judgments for which probability theory demands some qualitative ordering or relationship; or (c) the observed outcomes of uncertain events. Thus, while informative, the extensive body of research on judgment under uncertainty is often of little use to real-world decision makers when attempting to identify their best course of action, ex ante. This dissertation develops a method for assessing the quality of SPJs when benchmarks (e.g., base rates, outcomes) are unknown. Specifically, the present research develops a method for measuring the degree to which an individual's SPJs tend to agree with the optimized, aggregate "wisdom of the crowds", also known as credibility. To do this, we will conduct several probability forecasting tournaments and regress each individual's SPJs on optimized aggregates, calculated using methods developed by the Good Judgment Project. The estimated parameters of these models are indices of an individual's bias; expertise; and acuity (measured as the standard error of the regression) in subjective probability judgment. This work improves upon previous methods for assessing uncertainty in at least two ways. First, the credibility framework developed here can be used to evaluate SPJs when normative benchmarks (e.g., base rates, outcomes, repeated measurements) are unavailable or unknown. Second, preliminary evidence suggests that modeling judgments in this way can provide decision makers with an empirical method for correcting "errors" and "biases" in subjective probability judgment. Additionally, because predictions in domains such as military intelligence, climate science, and epidemiology tend be highly uncertain and errors prohibitively costly, decision makers in these domains may benefit from even marginal gains in their ability to evaluate SPJs. The present research facilitates this evaluation by providing straightforward measures of a source's quality, independent of empirical outcomes. In doing so, this research improves the quality and evaluability of decision making across a wide variety of domains.