Optimal Energy System Scheduling Using A Constraint-Aware Reinforcement Learning Algorithm

TL;DR

This paper introduces a novel reinforcement learning algorithm, MIP-DQN, that strictly enforces operational constraints in energy system scheduling, improving feasibility and accuracy over existing methods in complex renewable-integrated systems.

Contribution

The paper presents a constraint-aware DRL algorithm that integrates MIP formulations into neural networks, ensuring feasible and optimal energy scheduling under operational constraints.

Findings

Outperforms existing DRL algorithms in accuracy and constraint enforcement

Achieves lower error compared to optimal solutions with perfect forecasts

Ensures feasibility of schedules even on unseen test days

Abstract

The massive integration of renewable-based distributed energy resources (DERs) inherently increases the energy system's complexity, especially when it comes to defining its operational schedule. Deep reinforcement learning (DRL) algorithms arise as a promising solution due to their data-driven and model-free features. However, current DRL algorithms fail to enforce rigorous operational constraints (e.g., power balance, ramping up or down constraints) limiting their implementation in real systems. To overcome this, in this paper, a DRL algorithm (namely MIP-DQN) is proposed, capable of \textit{strictly} enforcing all operational constraints in the action space, ensuring the feasibility of the defined schedule in real-time operation. This is done by leveraging recent optimization advances for deep neural networks (DNNs) that allow their representation as a MIP formulation, enabling further consideration of any action space constraints. Comprehensive numerical simulations show that the proposed algorithm outperforms existing state-of-the-art DRL algorithms, obtaining a lower error when compared with the optimal global solution (upper boundary) obtained after solving a mathematical programming formulation with perfect forecast information; while strictly enforcing all operational constraints (even in unseen test days).