Framework for Statistical Evaluation of Complex Computer Models

National Institute Of Statistical Sciences, Durham NC

Investigators

Abstract

Though inherently a statistical issue, model evaluation lacks a unifying statistical framework; this project supplies such. The foundation of the framework is the use of Bayesian techniques to quantify the degree to which a model captures an underlying reality; develop theory and methods that allow dual use of data in both estimation of model inputs and evaluation of outputs; select evaluation functions, by which a model and reality are compared and through which flaws (causes of "invalidity") are found; and to design the collection of field and computer simulation data. The building of the framework relies on specific formulations of problems, motivated by testbed examples such as subsurface fluid flow models - where the computer model is deterministic but uncertainties are present in the model inputs and specifications - and traffic simulators - where the model is intrinsically stochastic, in addition to having uncertain inputs. Computer models are everywhere and evaluating their fidelity to reality is central to assessing their effectiveness in understanding real phenomena (such as flow of pollutants through soil and into groundwater) and predicting results of innovative technologies (such as new signal timing strategies to relieve traffic congestion in urban networks). This project develops a statistical structure and basis for such evaluations applicable across the scientific and technological landscape. In the process, the project creates the ingredients for a virtual laboratory to disseminate results, broaden involvement of other researchers (and users), and establish a unique educational and training environment.