RI: Small: Towards Provably Efficient Representation Learning in Reinforcement Learning via Rich Function Approximation

Cornell University, Ithaca NY

Investigators

Abstract

Reinforcement Learning enables artificial intelligence systems to learn by themselves. While today’s reinforcement Learning systems can empirically outperform humans on some tasks (such as chess), these systems often rely on an extreme amount of data and computation resources. This makes them not suitable for real-world applications where data are expensive. Also, reinforcement learning algorithms used in these systems often do not have any performance guarantees, such as how many data points the algorithm needs in order to solve the task with high confidence, which also limits their usage in safety critical applications. The main novelty of this project will be the development of new reinforcement learning algorithms that can learn efficiently, using as few training data points as possible. The development of efficient reinforcement learning algorithms can expand the applications of these systems to real-world applications where data are expensive to collect. For example, in autonomous driving systems, the developed technologies would have the potential to enable self-driving cars to adapt to new road conditions faster by making fewer mistakes. In personalized navigation systems for visually impaired people, systems trained with efficient reinforcement learning algorithms can engage with users via high-quality interactions at an early stage of the learning process, thus positively influence the user experience. The project aims to bridge the gap between reinforcement learning theory and practice by developing computationally and statistically efficient algorithms for large-scale Markov Decision Processes where data are high-dimensional and complex. The key innovation proposed in this project is to open the black box by incorporating representation learning into the reinforcement learning framework. The representation learning approach allows algorithms to extract compact information from high dimensional and unstructured data, and perform reasoning and decision making only using the compact representation — thus vastly improving the sample and computation efficiency. Two main thrusts are: (1) how to learn representations for counterfactual reinforcement learning where the learner only has access to a static dataset and has no ability to further interact with the environment; (2) how to integrate representation learning, exploration, and exploitation in the online reinforcement learning setting where the agent needs to actively interact with the environment for data acquisition. In addition to the algorithms and reinforcement learning representation learning theory development, this project proposes to design personalized voice navigation systems that can adapt to end-users, where sample efficient offline and online reinforcement learning plays an important role in fast and safe adaptation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.