Bayesian Optimization for Exploratory Experimentation in the Behavioral Sciences

University Of Colorado At Boulder, Boulder CO

Investigators

Abstract

This research project will develop an exploratory experimentation methodology for human behavioral research that will allow cognitive scientists to efficiently identify optimal conditions -- those leading to the most robust learning, the fastest performance, the fewest errors, the best decisions and choices. The tools to be developed will allow scientists to answer questions they cannot currently address due to the massive data collection effort required. To understand and predict human behavior, scientists typically perform controlled experiments that compare a small, carefully chosen set of experimental conditions. For example, in designing instructional software, a comparison might be made between two techniques for teaching students. The finding that one technique obtains reliably better outcomes has both practical and theoretical implications. However, this result does not answer the question one often wishes to ask: what is the very best possible technique? The methodology to be developed will allow scientists to evaluate many experimental conditions with only a few participants, in contrast to the traditional controlled experiment which evaluates only a few conditions each with many participants. A key product of the project will be black-box software that researchers in various disciplines of the cognitive sciences can use to apply exploratory experimentation to problems in their own field. Experimental studies also will be conducted to demonstrate the breadth of the approach in domains including: concept acquisition, color aesthetics, formal instruction, and the design of usable and engaging software. The project will extend Bayesian optimization methods to human experimental research. Bayesian optimization has long been used in the geostatistics community for inferring unobserved properties (e.g., oil reserves below the earth's surface) from costly measurements (e.g., drilling tests). In the current project, the "landscapes" being explored are defined over possible conditions (e.g., training strategies), the unobserved properties are internal cognitive states of the human observer, and the measurements are obtained via behavioral evaluations (e.g., assessments of learning). To apply Bayesian optimization methods to a range of human experimental research, mathematical models will be developed for multiple behavioral response measures, including choice, ranking, rating, latency, and free recall. The exploratory nature of the approach requires heuristics for sequentially selecting experimental conditions to obtain maximally informative data given prior observations. Various heuristics will be evaluated in the context of behavioral research.