Addressing Data Scarcity in Reinforcement Learning: Inference and Decision-Making under Complex Environments

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

This project addresses the increasing demand for reliable and interpretable reinforcement learning (RL) systems that can function effectively in complex, data-limited environments. RL has demonstrated potential in fields such as healthcare and public policy, where decisions often need to be made with limited and noisy data. However, ensuring that these systems are statistically robust, interpretable, and socially responsible remains a challenge. This research aims to develop tools that improve decision quality, support valid statistical inference, and enhance interpretability in RL algorithms, ultimately building trust and accountability for real-world applications. The broader impacts include advancing the science of machine learning, improving decision-making in resource-constrained settings, and training future data scientists through interdisciplinary mentorship and open educational resources. The research focuses on developing theoretical foundations and methods for reliable inference and decision-making in RL when data is scarce and models are misspecified. It consists of three main components: First, developing inference and decision tools for contextual bandits with misspecified reward models, where standard methods may fail. Second, constructing an inference framework for adaptive RL algorithms deployed across a population, utilizing a state-statistics decomposition to enhance interpretability and support principled personalization. Third, leveraging auxiliary offline data and structural assumptions to enable robust decision-making in nonstationary environments. This project will produce novel estimation methods and inference procedures with both finite-sample and asymptotic guarantees, along with publicly available software tools. Applications will include adaptive experimentation across various scientific domains, supported by interdisciplinary training in statistics, data science, and engineering. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.