Proximal Policy Optimization Based Reinforcement Learning for Joint Bidding in Energy and Frequency Regulation Markets

TL;DR

This paper introduces a reinforcement learning approach using Proximal Policy Optimization to optimize joint bidding strategies for battery energy storage systems in energy and frequency regulation markets, enhancing profitability amid market uncertainties.

Contribution

It formulates the BESS bidding problem as a Markov Decision Process and applies a model-free deep reinforcement learning algorithm for optimal strategy learning.

Findings

Joint bidding strategy outperforms individual market strategies.

Reinforcement learning effectively adapts to market dynamics.

Significant profit increase demonstrated with real-world data.

Abstract

Driven by the global decarbonization effort, the rapid integration of renewable energy into the conventional electricity grid presents new challenges and opportunities for the battery energy storage system (BESS) participating in the energy market. Energy arbitrage can be a significant source of revenue for the BESS due to the increasing price volatility in the spot market caused by the mismatch between renewable generation and electricity demand. In addition, the Frequency Control Ancillary Services (FCAS) markets established to stabilize the grid can offer higher returns for the BESS due to their capability to respond within milliseconds. Therefore, it is crucial for the BESS to carefully decide how much capacity to assign to each market to maximize the total profit under uncertain market conditions. This paper formulates the bidding problem of the BESS as a Markov Decision Process, which enables the BESS to participate in both the spot market and the FCAS market to maximize profit. Then, Proximal Policy Optimization, a model-free deep reinforcement learning algorithm, is employed to learn the optimal bidding strategy from the dynamic environment of the energy market under a continuous bidding scale. The proposed model is trained and validated using real-world historical data of the Australian National Electricity Market. The results demonstrate that our developed joint bidding strategy in both markets is significantly profitable compared to individual markets.