A Constraint Enforcement Deep Reinforcement Learning Framework for Optimal Energy Storage Systems Dispatch

TL;DR

This paper introduces a deep reinforcement learning framework for energy storage dispatch that strictly enforces operational constraints, improving decision quality in uncertain, stochastic environments compared to existing methods.

Contribution

It develops a novel MIP-DRL framework that combines deep reinforcement learning with mixed-integer programming to ensure constraint satisfaction during energy storage dispatch.

Findings

Outperforms state-of-the-art DRL algorithms in constraint enforcement

Achieves dispatch decisions close to the optimal with perfect forecast

Effectively handles continuous action spaces with strict operational constraints

Abstract

The optimal dispatch of energy storage systems (ESSs) presents formidable challenges due to the uncertainty introduced by fluctuations in dynamic prices, demand consumption, and renewable-based energy generation. By exploiting the generalization capabilities of deep neural networks (DNNs), deep reinforcement learning (DRL) algorithms can learn good-quality control models that adaptively respond to distribution networks' stochastic nature. However, current DRL algorithms lack the capabilities to enforce operational constraints strictly, often even providing unfeasible control actions. To address this issue, we propose a DRL framework that effectively handles continuous action spaces while strictly enforcing the environments and action space operational constraints during online operation. Firstly, the proposed framework trains an action-value function modeled using DNNs. Subsequently, this action-value function is formulated as a mixed-integer programming (MIP) formulation enabling the consideration of the environment's operational constraints. Comprehensive numerical simulations show the superior performance of the proposed MIP-DRL framework, effectively enforcing all constraints while delivering high-quality dispatch decisions when compared with state-of-the-art DRL algorithms and the optimal solution obtained with a perfect forecast of the stochastic variables.