Embodied Decision Making: The Influence of Action Errors on Reinforcement Learning

University Of California Berkeley, Berkeley CA

Investigators

Linked publications & trials

Abstract

? DESCRIPTION (provided by applicant): This proposal explores the interaction of processes involved in action selection and action execution. This interaction is essential for understanding how people learn to make optimal decisions and develop complex skills, as well as for explicating how disorders of the motor system may impact cognition, a question that has been of central interest in studies of degenerative diseases of the cerebellum and basal ganglia. The interaction can be appreciated by considering that successful decision making requires (at least) two fundamental abilities. First, an agent must be able to evaluate the value of different options in the environment, using that information to choose the option that will maximize reward. Second, the agent must be able to execute a response to indicate the selected option. Traditionally, models of decision making have focused on the former and ignored the latter. However, in many real-world situations, errors in execution are the primary impediment to successful outcomes. A tennis player may correctly opt to use a backhand swing instead of a forehand to return a serve, but fail to execute the action properly. Or in a more mundane example, a person might choose to take a sip from the wine glass rather than the water glass, but fail to reap the expected reward because she knocks over the glass by reaching in a clumsy manner. In this example, the issue is whether a person values wine less (due to the failure to obtain the expected reinforcement) after the clumsy reach, or whether the error is attributed to the execution system, with the outcome precluded from influencing future choice behavior (assuming we are equally competent in reaching for water or wine). The proposed work will examine the psychological processes and neural systems through which action execution and action selection interact. To this end, two specific aims will be addressed. 1) Our pilot work demonstrates a striking difference in choice behavior depending on whether failed outcomes are attributed to a property of the object or a limitation of the execution system. A series of computational models will be developed that have the potential to account for this difference. In a symbiotic manner, behavioral data from a series of experiments will be used to evaluate the models, and the models will be used to generate and test specific predictions. 2) Identify the neural regions involved in the interaction of action errors and selection processes. Of special note here is the idea that cerebellar-based representations of action execution errors might serve a dual-purpose, improving future action execution and providing a gating signal to constrain learning processes that underlie action selection dynamics. The experimental plan for this aim entails the integrated use of functional imaging and neuropsychological studies. At the completion of this project, the studies will help provide an integrated picture of how action selection and action execution processes interact in the human brain to optimize behavior.