Decentralized Multi-agent Reinforcement Learning based State-of-Charge Balancing Strategy for Distributed Energy Storage System

TL;DR

This paper introduces a decentralized multi-agent reinforcement learning approach for balancing the state-of-charge in distributed energy storage systems, enabling efficient, model-free coordination among storage units.

Contribution

It proposes a novel Dec-MARL method that leverages consensus algorithms and counterfactual demand balancing for decentralized control without expert knowledge.

Findings

Dec-MARL outperforms existing methods in simulation tests.

The approach achieves effective SoC balancing with decentralized information.

It demonstrates high flexibility and scalability in distributed energy systems.

Abstract

This paper develops a Decentralized Multi-Agent Reinforcement Learning (Dec-MARL) method to solve the SoC balancing problem in the distributed energy storage system (DESS). First, the SoC balancing problem is formulated into a finite Markov decision process with action constraints derived from demand balance, which can be solved by Dec-MARL. Specifically, the first-order average consensus algorithm is utilized to expand the observations of the DESS state in a fully-decentralized way, and the initial actions (i.e., output power) are decided by the agents (i.e., energy storage units) according to these observations. In order to get the final actions in the allowable range, a counterfactual demand balance algorithm is proposed to balance the total demand and the initial actions. Next, the agents execute the final actions and get local rewards from the environment, and the DESS steps into the next state. Finally, through the first-order average consensus algorithm, the agents get the average reward and the expended observation of the next state for later training. By the above procedure, Dec-MARL reveals outstanding performance in a fully-decentralized system without any expert experience or constructing any complicated model. Besides, it is flexible and can be extended to other decentralized multi-agent systems straightforwardly. Extensive simulations have validated the effectiveness and efficiency of Dec-MARL.