Reinforcement Learning-based Output Structured Feedback for Distributed Multi-Area Power System Frequency Control

TL;DR

This paper introduces a reinforcement learning approach with structured output feedback for distributed frequency control in multi-area power systems, enhancing stability and robustness amid uncertainties.

Contribution

It develops a data-driven constrained LQR framework with a novel minimax reformulation and applies zero-order policy gradient for optimal feedback gain learning in distributed power systems.

Findings

Successfully controls frequency in multi-area systems

Demonstrates robustness to load uncertainties

Adapts to model variations using an emulator grid

Abstract

Load frequency control (LFC) is a key factor to maintain the stable frequency in multi-area power systems. As the modern power systems evolve from centralized to distributed paradigm, LFC needs to consider the peer-to-peer (P2P) based scheme that considers limited information from the information-exchange graph for the generator control of each interconnected area. This paper aims to solve a data-driven constrained LQR problem with mean-variance risk constraints and output structured feedback, and applies this framework to solve the LFC problem in multi-area power systems. By reformulating the constrained optimization problem into a minimax problem, the stochastic gradient descent max-oracle (SGDmax) algorithm with zero-order policy gradient (ZOPG) is adopted to find the optimal feedback gain from the learning, while guaranteeing convergence. In addition, to improve the adaptation of the proposed learning method to new or varying models, we construct an emulator grid that approximates the dynamics of a physical grid and performs training based on this model. Once the feedback gain is obtained from the emulator grid, it is applied to the physical grid with a robustness test to check whether the controller from the approximated emulator applies to the actual system. Numerical tests show that the obtained feedback controller can successfully control the frequency of each area, while mitigating the uncertainty from the loads, with reliable robustness that ensures the adaptability of the obtained feedback gain to the actual physical grid.