SHF:Small:A Domain-Specific Language for Designing Cognitive-Science Experiments

University Of Utah, Salt Lake City UT

Investigators

Abstract

When a cognitive scientist designs an experiment, the design typically specifies factors of interest, the way that multiple factors are combined, and constraints that avoid combinations that are not interesting or that balance the frequency of combinations. Computers have enabled more sophisticated experimental designs, since a program can generate the steps for an experiment, but writing a program that correctly handles an experiment's constraints can be difficult. Worse, subtle mistakes in the program can bias the generated stimuli and invalidate the experiment's results. The project's novelties are a programming language that lets a scientist describe an experiment's design in higher level terms --- that is, higher than a general-purpose language's loops and conditionals --- and an execution engine to generate an experiment's stimuli from its description. The project's impacts are to make experiment design faster, to provide a better way for scientists to communicate their experiment designs, and to improve the reproducibility of experiments by ensuring statistical properties of the experiment's stimuli. The key to good statistical properties for generated stimuli is to uniformly sample the space of solutions to the experiment's constraints. In a simple experiment where all factors are crossed and combinations are weighted equally, random selection is unbiased and easy to implement. The programming problem is considerably more difficult when constraints are imposed on the combinations, such as having twice as many of one kind of stimuli in a row, versus having some other kind of stimuli in a row. State-of-the-art tools, such as UNIGEN, can sample from a constrained space, but the space must be expressed as a SAT formula, which is far from the way that cognitive scientists think about their designs. The project will combine this sampling technology with tools for hosting and compiling domain-specific languages, allowing an experimenter to work in domain terms while bridging the gap between the experimenter's view and the implementation mechanics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.