EAGER: On the Optimal Rewards Problem

Regents Of The University Of Michigan - Ann Arbor, Ann Arbor MI

Investigators

Abstract

The specification of goals in the form of utility or reward functions is a cornerstone of most approaches to developing autonomous artificial agents, and to understanding and shaping the behavior of natural biological agents - in fields ranging from control, AI, economics, psychology, and ethology. But in practice there are actually two notions of goals: the (human or evolutionary) designer's goals and the (artificial or natural) agent's goals. Should these be the same? The conventional (implicit) answer is "yes", but new work by the PIs shows that the answer may be "no" for computationally bounded agents. We define a new problem in agent design, the optimal rewards problem, whose solution is a reward function to assign to the agent so that in attempting to maximize its reward the agent best achieves the designer's goals. This project explores optimal rewards, through computational experimentation and analysis, on two broad fronts. We will develop new principles and algorithms for bounded planning agents, demonstrating increased performance over using conventional rewards, even when taking into account the cost of finding the optimal rewards. We will make significant advances in two areas of behavioral economics and ethology: the understanding of subjective utility in humans, and the understanding of foraging behavior in animals, by using optimal reward theory to rigorously derive subjective reward functions that take into account the computational limits of the natural agents. The results of this work, disseminated through published theory and software, should lead to foundational changes in the way we understand and design incentive structures for humans, animals, and artificial agents, and thus to significant practical benefits for any methods in engineering, education, economics, and health that depend on finding good incentives for desired behavioral outcomes.