Enhanced Power System Resiliency through Adaptive Automatic Remedial Action Selection using Multi-Agent Reinforcement Learning

Rensselaer Polytechnic Institute, Troy NY

Investigators

Abstract

Electrical power grids have been facing increased operational challenges, including those due to natural disasters, extreme weather conditions, and increasing complexity with the ongoing upgrades to achieve zero-carbon footprint goals. These factors challenge electrical power system operators in making decisions on which action to take in order to keep most of the households powered by the grid or even to avoid massive blackouts. Even though the control centers’ technology and their practices undergo upgrades over time, grid failures and blackouts still happen. When a severe event occurs on the grid, the power system operator has to follow specific procedures to resolve the unfavorable condition in order to bring the system back to normal operation. With the aforementioned challenges in mind, it becomes necessary to automate the process of choosing the most suitable remedial actions. In this case, the speed of decision-making is crucial to avoid massive disconnections and minimize the impact of unfavorable events on grid equipment. A significant improvement in response time and even full automation may become possible in practice with the help of modern computer science methods such as machine learning algorithms and, more specifically, its subgroup of learning methods that is known as reinforcement learning. Reinforcement learning acquires knowledge by choosing actions that provide the largest benefit and has been shown to solve complex problems, such as training robots to solve complicated tasks or even playing complex games such as chess and the ancient game of Go. Leveraging reinforcement learning methods, the proposed solution in this research allows us to prioritize an action according to its influence on consumers by choosing the action that reduces potential negative impacts. This is done by assigning higher priority to a potential action if the risk of disconnection of critical facilities is lower. To fully exploit the advantages of reinforcement learning methods, multiple “agents” (decision makers) are used. These agents can be distributed all across different grid facilities and perform local control actions. However, to assure grid resiliency, there is a degree of central coordination between them. Each agent is responsible for its own control area where it can operate according to the prioritized instructions that are provided in the form of actions during the agent’s learning phase. In such a way, we aim to increase the trustworthiness of the agents by incorporating prior knowledge about the power system that is learned via ultra-high-fidelity nonlinear simulations. Once the agents have learned enough, simulations are no longer needed and they can make their decisions "on the fly". The advantage of the proposed solution with respect to previously developed optimization methods in use in some grid control centers is their ultra-fast online computational performance. Thus, the flexibility of machine learning methods to perform exhaustive and fast analysis of a power system's security while considering a broader impact on the connected households makes them attractive to complement or even replace existing solutions. The work in this project aims not only to advance the development of the above mentioned methods but also to build proof-of-concept tools able to derive actionable information for power system operators to improve power system resiliency considering practical real-world constraints. This will be achieved by working together with two US utilities and using their grid models and measurements. If successful, the results of the project may lay the foundation of an entirely new approach for power system operation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.