Multi-market Energy Optimization with Renewables via Reinforcement Learning

TL;DR

This paper presents a deep reinforcement learning framework for optimizing power plant operations with renewables and storage, effectively handling market uncertainties, complex storage models, and system constraints to maximize revenue.

Contribution

It introduces a hierarchical MDP approach with a novel action projection method, enabling RL to incorporate complex, non-linear storage models in energy market optimization.

Findings

RL policies outperform baseline controls in experiments.

The framework adapts to various storage models and market conditions.

Effective policy constraints ensure system safety and feasibility.

Abstract

This paper introduces a deep reinforcement learning (RL) framework for optimizing the operations of power plants pairing renewable energy with storage. The objective is to maximize revenue from energy markets while minimizing storage degradation costs and renewable curtailment. The framework handles complexities such as time coupling by storage devices, uncertainty in renewable generation and energy prices, and non-linear storage models. The study treats the problem as a hierarchical Markov Decision Process (MDP) and uses component-level simulators for storage. It utilizes RL to incorporate complex storage models, overcoming restrictions of optimization-based methods that require convex and differentiable component models. A significant aspect of this approach is ensuring policy actions respect system constraints, achieved via a novel method of projecting potentially infeasible actions onto a safe state-action set. The paper demonstrates the efficacy of this approach through extensive experiments using data from US and Indian electricity markets, comparing the learned RL policies with a baseline control policy and a retrospective optimal control policy. It validates the adaptability of the learning framework with various storage models and shows the effectiveness of RL in a complex energy optimization setting, in the context of multi-market bidding, probabilistic forecasts, and accurate storage component models.