Decentralized Safe Reinforcement Learning for Voltage Control

TL;DR

This paper introduces a decentralized reinforcement learning framework for voltage control in power systems, ensuring safety and stability through Lipschitz constraints on neural network controllers.

Contribution

It develops a safe RL approach with Lipschitz constraints, explicitly designed neural networks, and a decentralized training framework for voltage regulation.

Findings

Guarantees exponential stability under Lipschitz constraints

Enlarges neural network search space by optimizing Lipschitz bounds

Constructs a decentralized RL framework for model-free voltage control

Abstract

Inverter-based distributed energy resources provide the possibility for fast time-scale voltage control by quickly adjusting their reactive power. The power-electronic interfaces allow these resources to realize almost arbitrary control law, but designing these decentralized controllers is nontrivial. Reinforcement learning (RL) approaches are becoming increasingly popular to search for policy parameterized by neural networks. It is difficult, however, to enforce that the learned controllers are safe, in the sense that they may introduce instabilities into the system. This paper proposes a safe learning approach for voltage control. We prove that the system is guaranteed to be exponentially stable if each controller satisfies certain Lipschitz constraints. The set of Lipschitz bound is optimized to enlarge the search space for neural network controllers. We explicitly engineer the structure of neural network controllers such that they satisfy the Lipschitz constraints by design. A decentralized RL framework is constructed to train local neural network controller at each bus in a model-free setting.