RI: Small: Active Testing for Evaluating Reinforcement Learning Agents

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

Applications of artificial intelligence (AI) are exploding and the demand for trustworthy autonomous systems has never been higher. Before an autonomous system is widely deployed and its decisions have real-life consequences, it is critical to be able to evaluate the expected outcome of letting the system make those decisions. Evaluation is particularly vital when stakes are high, for example, when using AI to control expensive robots in changing environments or drive cars at high speeds. Testing such a system's decisions in real environments will allow measure the extent to which extent the machine's decisions are good. Unfortunately, many real-world applications, such as autonomous driving, have far too much variety that could be accounted ahead of time. Furthermore, the most critical situations may be relatively rare, further decreasing testing efficiency and leading to the deployment of autonomous systems that can make poor decisions in critical situations. To address this problem, this project develops new methods for active testing of autonomous decision-making. In contrast to passive testing by simply running the autonomous system, these methods actively seek out the most informative situations to test under. In doing so, they produce a higher confidence evaluation of a given autonomous system's decision-making. Consequently, the methods strengthen society’s confidence in autonomous systems since such systems can be tested and then only deployed if practitioners are confident they make good decisions. More specifically, this project focuses on evaluating autonomous systems that follow policies produced by reinforcement learning (RL) algorithms. The project develops fundamental theory and practical, domain-agnostic methods for active testing of RL-trained policies. These active-testing methods identify consequential data for policy evaluation and then focus test data collection on such data. Specifically, this project consists of three research thrusts. The first thrust derives and implements the optimal policy to follow when evaluating a given policy. The second thrust develops novel adaptive sampling algorithms that reduce inefficiency due to random sampling when collecting data for policy evaluation. Finally, the third thrust develops a methodology for adaptively setting a policy's initial state so as to obtain the most accurate evaluation of that policy in as few trials as possible. The novel methods produced bring new understanding to the study of policy evaluation in RL and the broader AI field. In particular, little research has gone into the fundamental question of how data collection affects the quality of policy evaluation. This project brings new understanding to this question and, in doing so, develops novel adaptive data collection methods that enable effective policy evaluation in realistic RL domains. This project advances foundational RL knowledge on tailoring data collection for accurate policy evaluation by introducing domain-agnostic and theoretically-based methods for adaptive data collection. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.