CAREER: Soft-robust Methods for Offline Reinforcement Learning

University Of New Hampshire, Durham NH

Investigators

Abstract

Improvements in sensors, data collection, and computational power have driven the desire to harness data to improve decision-making in various domains, such as precision agriculture or medicine. Making effective data-driven decisions is a challenging problem studied by the reinforcement learning community. Despite their success in simulated domains, like board games, existing reinforcement learning algorithms are still too unreliable for widespread deployment. This project develops new reinforcement learning algorithms that achieve reliability by carefully balancing the expected quality of recommended decisions with their risk of failure. The risk of failure is measured using techniques from stochastic finance, which enable flexible and efficient algorithms. These new, reliable algorithms will help bring data-driven decision-making to new domains, including agriculture, ecological management, and medicine. The research is integrated with educational activities to provide graduate and undergraduate students with training opportunities and new study materials, including a textbook. This research develops and analyzes algorithms for reinforcement learning (RL) problems with 1) limited or expensive data and 2) a high cost of failure. Soft-robust objectives build on convex risk measures to balance robustness and average quality of data-driven decision-making problems. While soft-robust objectives are well understood in single-stage optimization problems, many fundamental and practical questions remain open in multi-stage problems, like RL. This project will answer these questions and develop reliable, tractable, and scalable algorithms by tackling three main objectives. First, the project will establish soft-robust RL formulations' statistical and computational properties. Second, the project will generate a new class of tabular soft-robust RL algorithms built on new insights into the relationship between soft-robustness and robust Markov decision processes. Third, the project will scale the tabular algorithms to value function approximation and gradient style methods. The project will address both batch RL, with known rewards and estimated transition probabilities, and inverse RL, with estimated rewards and known transition probabilities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.