RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks

Princeton University, Princeton NJ

Investigators

Abstract

Reinforcement Learning (RL) is a generic and flexible framework for sequential decision-making problems. Modern RL commonly engages practical problems with an enormous number of states, where function approximation must be deployed to generalize knowledge from the visited states to the unvisited ones. Function approximation, particularly in the form of deep neural networks, lies at the heart of the recent practical successes of RL in domains such as robotics, autonomous vehicles, business management, and production systems. However, most existing theoretical understanding of RL has been restricted to the problems with a small number of states without using function approximation, and a significant gap remains between theory and practice of RL. This project seeks to bridge this gap by identifying and addressing the fundamental challenges that are persistent in RL with function approximation. To accomplish this goal, this project will develop a comprehensive set of fundamental theory and methodologies for RL with function approximation, with a special emphasis on its applicability to modern deep RL. Concretely, this project will proceed with two parallel thrusts. The first thrust investigates model-free RL with general function approximation. This thrust will identify the general structure of the function classes where RL problems are tractable, design new provably efficient algorithms for those general function classes, and address the challenging issues such as model misspecification. This thrust will further integrate these results with recent advances in representation, optimization and generalization of deep learning. The second thrust concerns model-based RL to incorporate domain knowledge. This thrust will first develop a general-purpose model-based RL method using the idea of value-targeted system identification. This thrust will also develop stochastic-approximation variants of the methods for tractable computation, and deep model reduction or feature learning methods for analyzing off-policy data prior to on-policy model-based RL. Important outcomes of this project will be new general and reliable RL algorithms that are guaranteed to perform well for a wide range of applications with both computational and statistical efficiency. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.