EAGER: Theoretical Foundations for Integrating Foundational Models into Reinforcement Learning

Purdue University, West Lafayette IN

Investigators

Abstract

Reinforcement learning (RL) is a promising approach for enabling machines, such as robots or cars, to make decisions in complex and unpredictable environments. Examples of these are robots that can run or autonomous cars that can navigate cluttered streets. To make these algorithms work, people use simulation. The problem is that in practice, these robots struggle to solve similar challenges in the real-world, due to the lack of controllability in these applications. The more realistic the environment, the more data that are required, and the more time that is needed for such robots to work effectively. These challenges are made even more difficult by what is known as the sim(ulatio)n-to-real gap, which refers to the problem of applying what has been learned in computer simulations to real-life situations where conditions are different and less predictable. These limitations restrict the use of the algorithms and robots in important areas where collecting data is expensive, risky, or impractical, such as healthcare robotics or emergency response systems. This project will address these key limitations by eliminating reliance on simulators altogether and developing reinforcement learning methods that can be trained directly in real-world settings. The project will create new theoretical frameworks and algorithms that integrate foundation models, which are large, pre-trained artificial intelligence models, into reinforcement learning. These models provide built-in knowledge about the world, enabling learning systems to acquire new skills more quickly, perform better in unfamiliar situations, and be deployed more rapidly in real-world environments. The research will focus on three core areas. First, it will establish formal methods for incorporating knowledge from foundation models into the state and action spaces of reinforcement learning agents, allowing them to leverage high-level abstractions and prior knowledge to inform decision-making. Second, it will derive theoretical analyses and provide performance guarantees, including bounds on sample complexity, generalization capability, and convergence rates, to demonstrate how these enhanced agents can learn more efficiently with fewer interactions in complex, real-world environments. Third, it will design new reinforcement learning algorithms that integrate foundation models to improve reward modeling, exploration strategies, and policy optimization, ensuring stability and robustness during both training and deployment. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.