EAGER: Toward a General Framework for Optimal Experimentation in Computational Cognition

Howard University, Washington DC

Investigators

Abstract

Cognitive science aims to gain detailed insight into the underlying mechanisms of cognitive tasks. To achieve this, researchers change factors affecting a task in an experiment and observe responses. One way to learn from such an experiment is to build and test a computational model of stimulus-response relationships. However, the level of detail with which a model describes a task requires as much detail in supporting data. As observations from experiments are often expensive (e.g., child subject), this is a rather significant barrier. The method of optimal experiments can be a solution. It can optimize the selection of stimuli to maximize inference from responses. Nonetheless, the difficulty in applying the method to each new experiment has been a stumbling block. This project proposes to lay the foundation for a general framework for optimal experiments. The goal is to make it applicable to a wide range of modeling problems in cognitive science. This will help cognitive scientists to develop quantitative accounts of cognitive tasks effectively. Further, the method has the potential to accelerate scientific discovery broadly in social and behavioral research. Conducting cognitive science experiments guided by optimal interaction with subjects toward a clear, quantified inference goal is a powerful idea. Such a method is particularly enticing for behavioral experiments in which the amount of noise in response is so great as to require many repeated measurements. Despite its groundbreaking potential for cognitive modeling research, the method of optimal experimentation is out of reach for most researchers in the field. The formidable task of implementing it for each unique experimental paradigm has been an obstacle to the realization of the methodology's promising power. The project focuses on establishing the technical feasibility of optimal experiments in arbitrary cognitive modeling contexts. The proposed research will define the need for the methodology in the field clearly, identify suitable computational strategies, and test alternative algorithms in simulation studies. The performance of algorithms under consideration will be evaluated on a testbed of modeling paradigms whose successful treatment would transfer to a wide range of similar problems. The project aims to create a tangible blueprint for a general-purpose methodology for optimal experimentation in computational cognition. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.