EAGER: Income Learning: A New Model for Behavior-Analysis-Inspired Learning from Human Feedback

Washington State University, Pullman WA

Investigators

Abstract

As virtual agents and physical robots become more common, there is an increasing number of complex tasks they can usefully perform to assist humans. These tasks are typically formalized as sequential decision tasks, where robots and agents perceive states, take actions, and receive a reward feedback signal. In practice, there is a critical need to learn directly from human users if such machines are to accomplish tasks outside of those pre-specified by the original developments. Machine reinforcement learning (RL), a paradigm often used for solving sequential decision making tasks, was originally developed with inspiration from animal learning research from the applied behavior analysis (ABA) community. Existing RL approaches operationalize a limited set of ABA principles effectively; however, there are additional principles and properties from ABA research that are not well encapsulated in the existing RL formalisms, and that are likely sources of new inspiration for designing more effective RL techniques capable of learning from human teachers. This project will (1) take combine principles from ABA and RL to produce algorithms that can learn more effectively from humans, (2) evaluate these algorithms in both virtual agents and on robot platforms, and (3) investigate whether and how non-expert humans can construct sequences of tasks of increasing difficulty, similar to how expert animal trainers shape tasks. Insights from these user studies will be leveraged to further improve our algorithms' abilities to learn from human trainers. Once successful, this project will make critical progress towards allowing non-technical users to be able to teach virtual and physical agents to perform complex tasks in a natural setting, familiar to many from previous experience in training household pets. This project is a part of a larger effort between Washington State University (WSU), North Carolina State University, and Brown University. The WSU effort will focus on implementing the proposed family of machine learning algorithms, called Income Learning (I-Learning). As these algorithms are co-developed by the three universities, WSU will design user studies to evaluate when and how the principles behind I-Learning allow it to outperform other existing algorithms at learning from human feedback. WSU will primarily focus on 1) virtual agents, allowing test learning via crowdsourcing, as well as testing on 2) physical robots and study if embodiment changes user's perceptions and actions, or the algorithms' learning efficacy. Additionally, WSU will investigate 3) human curricula design. Expert trainers can shape the behavior of animals, increasing task complexity over time, so that the animals can learn a sequence of tasks much faster than if they trained directly on the final, difficult task. WSU will run user studies on crowdsourcing platforms to better understand how non-expert humans design curricula for machine learning algorithms in sequential decision tasks, and investigate how these design decisions can inform algorithm design.