Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow

TL;DR

This paper introduces a primal-dual constrained deep reinforcement learning method for real-time dynamic optimal power flow, ensuring safety constraints are met while optimizing power system control.

Contribution

It proposes a novel primal-dual approach for constrained DRL in power systems and proves convergence of the networks, improving safety and optimality.

Findings

Outperforms existing methods in IEEE system case studies

Maintains safety constraints during dynamic control

Adapts effectively to changing system conditions

Abstract

Deep Reinforcement Learning (DRL) has become a popular method for solving control problems in power systems. Conventional DRL encourages the agent to explore various policies encoded in a neural network (NN) with the goal of maximizing the reward function. However, this approach can lead to infeasible solutions that violate physical constraints such as power flow equations, voltage limits, and dynamic constraints. Ensuring these constraints are met is crucial in power systems, as they are a safety critical infrastructure. To address this issue, existing DRL algorithms remedy the problem by projecting the actions onto the feasible set, which can result in sub-optimal solutions. This paper presents a novel primal-dual approach for learning optimal constrained DRL policies for dynamic optimal power flow problems, with the aim of controlling power generations and battery outputs. We also prove the convergence of the critic and actor networks. Our case studies on IEEE standard systems demonstrate the superiority of the proposed approach in dynamically adapting to the environment while maintaining safety constraints.