Reinforcement Learning Based Optimal Battery Control Under Cycle-based Degradation Cost

TL;DR

This paper develops a novel reinforcement learning approach for optimal battery control that explicitly models cycle-based degradation, leading to improved operational efficiency and reduced degradation in energy storage systems.

Contribution

It introduces a new MDP formulation with additional state variables to accurately represent cycle-based degradation, enabling effective use of deep Q-Network algorithms.

Findings

Enhanced battery operation performance with cycle-based degradation model

Reduced battery degradation compared to linear approximation methods

Effective application of deep RL algorithms to real market data

Abstract

Battery energy storage systems are providing increasing level of benefits to power grid operations by decreasing the resource uncertainty and supporting frequency regulation. Thus, it is crucial to obtain the optimal policy for battery to efficiently provide these grid-services while accounting for its degradation cost. To solve the optimal battery control (OBC) problem using the powerful reinforcement learning (RL) algorithms, this paper aims to develop a new representation of the cycle-based battery degradation model according to the rainflow algorithm. As the latter depends on the full trajectory, existing work has to rely on linearized approximation for converting it into instantaneous terms for the Markov Decision Process (MDP) based formulation. We propose a new MDP form by introducing additional state variables that can easily keep track of past switching points for determining the cycle depth. The proposed degradation model allows to adopt the powerful deep Q-Network (DQN) based RL algorithm to efficiently search for the OBC policy. Numerical tests using real market data have demonstrated the performance improvements of the proposed cycle-based degradation model in enhancing the battery operations while mitigating its degradation, as compared to earlier work using the linearized approximation.