Reinforcement Learning Based Bidding Framework with High-dimensional Bids in Power Markets

TL;DR

This paper introduces a novel RL-based bidding framework that utilizes high-dimensional bids (N price-power pairs) through neural network supply functions, significantly enhancing flexibility and profits in power market bidding.

Contribution

It proposes integrating high-dimensional bids into RL methods using neural network supply functions within an MDP framework, addressing limitations of low-dimensional bid approaches.

Findings

HDBs improve bidding flexibility.

Enhanced profits in power market simulations.

Effective integration of NNSFs with RL methods.

Abstract

Over the past decade, bidding in power markets has attracted widespread attention. Reinforcement Learning (RL) has been widely used for power market bidding as a powerful AI tool to make decisions under real-world uncertainties. However, current RL methods mostly employ low dimensional bids, which significantly diverge from the N price-power pairs commonly used in the current power markets. The N-pair bidding format is denoted as High Dimensional Bids (HDBs), which has not been fully integrated into the existing RL-based bidding methods. The loss of flexibility in current RL bidding methods could greatly limit the bidding profits and make it difficult to tackle the rising uncertainties brought by renewable energy generations. In this paper, we intend to propose a framework to fully utilize HDBs for RL-based bidding methods. First, we employ a special type of neural network called Neural Network Supply Functions (NNSFs) to generate HDBs in the form of N price-power pairs. Second, we embed the NNSF into a Markov Decision Process (MDP) to make it compatible with most existing RL methods. Finally, experiments on Energy Storage Systems (ESSs) in the PJM Real-Time (RT) power market show that the proposed bidding method with HDBs can significantly improve bidding flexibility, thereby improving the profit of the state-of-the-art RL bidding methods.