CRII: RI: RUI: Performance guarantees for online apprenticeship learning with unknown features

University Of North Carolina At Asheville, Asheville NC

Investigators

Abstract

The surge of interest in robots that can be trained to perform in industries such as manufacturing and healthcare increases the need for improved learning methods. In one such method, apprenticeship learning, a robot learns to perform a task by watching an expert. This project's goals are to decrease the time required to set up the robot for learning and to offer college students hands-on robotic research activities. Most related work describes techniques that require the robot's programmer to identify features of the task. This project will reduce the programmer's setup work by using automatically generated features. A new method ensures the accuracy of the robotic learner by determining the number of observations required of the expert. Maximum Causal Entropy Inverse-Reinforcement Learning learns feature weights from demonstration, and like other maximum entropy models, offers strong performance guarantees and analysis possibilities. Proven generalization bounds are available that allow an estimate on the number of observed samples needed for a given expected level of error. However, they require knowledge of the covering number or complexity of the feature functions and/or known limits on the feature weights. When features are automatically extracted from a robot's sensor stream it is likely that many spurious features will be selected for use which could greatly increase the estimated number of samples needed, rendering the technique impractical. This project is developing an iterative, online variant of the maximum causal inverse-reinforcement learning algorithm that runs during the demonstrations and selects high-valued features as a critical subset which are then used to calculate the sample bounds. Once the required number of samples have been observed an offline inverse-reinforcement learning technique is run to ensure the feature weights are learned accurately. The new algorithm will be evaluated on a robot tasked with sorting previously-unknown objects. In this task, students will demonstrate the sorting of objects, then the robot will be required to do the same. Afterwards, the robot will be reset and the experiment repeats with a new set of objects. Critically, the software on the robot should not be changed or updated between these tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.