CPS: Small: Data-Driven Reinforcement Learning Control of Large CPS Networks using Multi-Stage Hierarchical Decompositions

North Carolina State University, Raleigh NC

Investigators

Abstract

In the current state-of-the-art machine learning based real-time control of large complex networks such as electric power systems is largely bottlenecked by the curse of dimensionality. Even the simplest control designs demand numerical complexity to accomplish. The problem becomes even more challenging when the network model is unknown, due to which an additional learning time needs to be accommodated. This project will take a new stance for solving this problem, and develop a suite of hierarchical or nested machine learning-based schemes that take advantage of various forms of physical redundancies in the network dynamics to learn only the most important traits of its behavior instead of wasting time in learning minor traits that may improve the closed-loop performance only by a small amount. This selective learning approach will reduce learning time by several orders of magnitude, making real-time control more tractable and more implementable. Products will include numerical algorithms that are applicable across a wide range of machine learning based control. In terms of societal impact, the project is strongly envisioned to bring control theorists closer to data scientists so that these two research communities can work together, and answer important questions such as: why the value of big data has traditionally been under-utilized in controls, what new dimensions can control theory gain from machine learning and vice versa, and what primary analytical and experimental tools are needed to make this marriage more successful. The research will also support the cross-disciplinary development of a diverse cohort of PhD and undergraduate students, and the development of a graduate-level course on the applications of machine learning in control. The main technical philosophy behind this work will be to exploit the fact that most large-scale dynamic networks exhibit a lot of redundancy in their dynamics. These redundancies can arise from various factors such as time-scale separation, spatial-scale separation, low-rank controllability, spectral clustering, similarity in temporal snapshots, separation in the control objectives, etc. By deciphering these redundancies from online measurements of states and inputs using machine learning tools, one can devise appropriate decomposition rules to partition the network into non-overlapping groups. Multiple sets of composite controllers can then be learned independently for each group using model-free reinforcement learning. Accordingly, the control goals of the network will also be decomposed into local (microscopic) and global (macroscopic) reward functions. Local controllers will be designed via privacy preserving group learning, and the global controllers via model reduction and filtering. The study will be driven by examples from wide-area control of power systems using streaming Synchrophasor data from Phasor Measurements Units (PMUs). Validation experiments will be carried out in a cyber-physical systems testbed at North Carolina State University. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.