Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning

TL;DR

This paper introduces a model-based deep reinforcement learning approach using a neural network surrogate model to efficiently learn voltage control strategies in power systems, significantly reducing training time and improving safety.

Contribution

It presents a novel framework combining model-based DRL with imitation learning and reward shaping, enhancing training efficiency and stability for large-scale power system control.

Findings

Achieved 97.5% sample efficiency in training.

Attained 87.7% training efficiency on IEEE 300-bus system.

Demonstrated effective voltage control strategy learning.

Abstract

This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time, both critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. And it is desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based-DRL framework where a deep neural network (DNN)-based dynamic surrogate model, instead of a real-world power-grid or physics-based simulation, is utilized with the policy learning framework, making the process faster and sample efficient. However, stabilizing model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We solved these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step surrogate loss. Finally, we achieved 97.5% sample efficiency and 87.7% training efficiency for an application to the IEEE 300-bus test system.