Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies

TL;DR

This paper introduces a safe reinforcement learning control framework for energy arbitrage in battery systems, combining revenue optimization with post-processing corrections to ensure safety, validated on real-world data and hardware.

Contribution

It presents a novel RL-based control framework with a post-processing correction step to guarantee safety in energy arbitrage applications, adaptable beyond this domain.

Findings

Effective in maximizing arbitrage revenue

Ensures safety through policy correction

Validated on real battery and Belgian market data

Abstract

A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.