RI: Small: Approximate Inference for Planning and Reinforcement Learning

Indiana University, Bloomington IN

Investigators

Abstract

Stochastic planning, the problem of controlling a system when the environment includes uncertainty, is ubiquitous across many industries and scientific areas. Some important examples include controlling wind farms for energy generation and manipulating a complex robotic system. In many cases, these problems have to be solved even though there is no precise model of how the world behaves. For example, when applying some physical force, we may not know whether a robot will overshoot a corner or slip during its movement. This requires the agent to combine learning about the world with planning. These learning and planning problems are computationally challenging and developing tools for solving them is still a major goal for AI research. Probabilistic inference provides a general mathematical framework that captures connections between potential observations and events of interest. An exciting line of work aims to develop planning algorithms through the lens of probabilistic inference, by associating hypothetical actions and hypothetical outcomes of such actions with potential future utility. The project will develop new machine learning and planning algorithms from this perspective. The focus is on the interaction between the need for approximations in the inference process and the implications of approximations for planning quality. Algorithmic solutions will be evaluated in a range of problems across several fields. The project explores problems across AI planning, optimal control in robotics, and reinforcement learning that share some technical core, but have often been studied separately. More specifically the project will develop algorithms for stochastic planning, for uncertainty quantification in machine learning and for model based reinforcement learning. Hybrid environments that include both discrete quantities and continuous quantities provide an additional computational challenge when solving these optimization problems. To devise effective planning algorithms, the project will develop and exploit ideas about differentiable symbolic probabilistic inference and its connection to message passing inference algorithms. In addition, long horizon search will be enabled by integrating transition models for multiple time scales into inference algorithms. To devise effective reinforcement learning algorithms, the project will first develop new methods for uncertainty quantification with deep neural networks that will be instrumental for model learning. In turn, combining model uncertainty with planning algorithms will yield novel model based reinforcement learning methods. These algorithms will be refined for real time control to enable applications in robotic systems. The algorithms will be evaluated on problems from all three fields to guarantee generality and cross fertilization. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.