CAREER: Towards Real-world Reinforcement Learning

Cornell University, Ithaca NY

Investigators

Abstract

Reinforcement learning (RL) is one of the most important paradigms for modeling data-driven decision-making. Recent years have witnessed several empirical successes of RL, such as RL agents that outperform humans in video and board games. However, many empirical RL algorithms today often require many training examples to learn and can produce unreliable solutions (solutions that exhibit catastrophic failures, for example). While these issues are typically not problematic when training RL agents in simulators, they pose significant difficulties when deploying RL to real-world problems where data (including human feedback) is expensive, and reliability is essential. The main novelty of this project will be the development of new RL algorithms that can learn efficiently (from as few training data points as possible) and reliably (avoid catastrophic failures with high probability). The development of such RL algorithms can expand the applications of RL systems from simulation to real-world applications where data is expensive to collect and safety is critical. In autonomous driving, the developed technologies can make self-driving cars adapt to new road conditions safely by making fewer mistakes. In generative Artificial Intelligence (AI), efficient and reliable RL algorithms that can learn from rich human feedback will enable better human-AI alignment, making AI systems improve reliably and safely under human guidance. The main research goal of this project is to enable real-world RL by advancing RL techniques, theoretically and empirically. The critical innovation in the project is to develop safe and efficient RL algorithms by leveraging specific problem structures and rich human feedback. The project has three main thrusts. First, the project will establish risk-averse RL algorithms that are provably correct and scalable to high dimensional data. Second, the project will develop RL algorithms that can leverage common problem-specific structures for improved sample efficiency. Third, the project will create new algorithms for RL with rich feedback beyond scalar rewards (including preference-based feedback and positive demonstrations). In addition to the proposed work on algorithmic advancements, the project will focus on their deployment to real-world problems, including database query optimization and optimizing generative models such as Large Language Models and Diffusion Models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.