Reinforcement Learning and Kullback-Leibler Stochastic Optimal Control for Complex Networks

University Of Florida, Gainesville FL

Investigators

Abstract

Natural and man-made networked systems are all around us. The power grid and the Internet are two examples of apparently complex interconnected systems, in which millions of "agents" are eager to extract value in the form of energy or bandwidth. While these systems are complex when measured in graph-theoretic terms, the behavior of communication and energy systems appears simple and highly predictable to the end users (in most of the world). This success is due in part to distributed control loops that manage system-wide supply-demand balance. An example of distributed control in the Internet is TCP/IP, and automatic generation control (AGC) in most electric power grids. While distributed control protocols are highly developed and widely accepted in communication applications, this is less true in other networked systems such as electric power and natural gas distribution. This project aims to advance control theory for complex interconnected systems. The application focus is on power systems, but the control techniques are general and are likely to have far broader impact. Recent control innovations are highlighted in the project as building blocks in the construction of algorithms for control, based on a combination of local decision making and global management of the ensemble: 1. Control techniques for local decision making will be a theme of the project using a new Kullback-Leibler-Quadratic optimal control approach introduced by the PI's group. 2. Reinforcement learning (RL) is the engine behind Google's recent computer game successes and is a natural framework for control synthesis in an uncertain complex environment. The Zap Q-learning algorithms introduced recently by the PI and his colleagues are a new class of RL algorithms that are virtually universally stable and have provably optimal convergence rate. 3. Mean field models have a long history in power systems (with roots in statistical physics), they will be used to approximate aggregate behavior, and as a foundation to construct algorithms to control the aggregate. Algorithm design will be complemented with simulation studies, focusing initially on applications to power systems. A course in smart grid technologies will be augmented and the project will include participation from undergraduate students. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.