Algorithmic Methods for Combining Expert Judgments

Harvard University, Cambridge MA

Investigators

Abstract

Decisions about environmental protection, homeland security, and other critical issues are informed by mathematical models used to estimate the risk and consequences of alternative actions. Many of the parameters in these models cannot be measured directly and so models rely heavily on expert judgment. In recent years, formal structured methods for eliciting judgments from a panel of subject-matter experts have been developed and applied to important public issues (e.g., nuclear power safety, air pollution). In structured expert judgment, each expert reports his or her beliefs about the value of a parameter as a subjective probability distribution. For decision making, it is useful to combine the experts' judgments into a single probability distribution. This research investigates the performance of alternative combination rules, including a simple average, a weighted average with weights based on the experts' performance on variables for which the true values become known, and a method that updates the decision maker's distribution in accordance with Bayes' Rule. The research compares the performance of these methods by using synthetic expert judgments in which the number of experts and the qualities of their individual judgments (e.g., precision, calibration) are known. The project will also use real expert-judgment data obtained for environmental risk assessment to see how sensitive results are to alternative combination methods and analytic choices. The work should contribute to better decision making through improved understanding of methods for using experts' judgments, including questions of how many and how diverse a set of experts to involve and how to synthesize their responses.