Decentralized Microgrid Energy Management: A Multi-agent Correlated Q-learning Approach

TL;DR

This paper introduces a multi-agent correlated Q-learning approach for decentralized microgrid energy management, improving coordination and revenue optimization among agents with guaranteed convergence.

Contribution

It proposes a novel correlated Q-learning algorithm for microgrid energy management, ensuring convergence and better revenue distribution among agents.

Findings

CEQ balances agent revenues without reducing total benefits.

CEQ reduces DSM agent costs by 19.3%.

CEQ increases ESS agent benefits by 44.2%.

Abstract

Microgrids (MG) are anticipated to be important players in the future smart grid. For proper operation of MGs an Energy Management System (EMS) is essential. The EMS of an MG could be rather complicated when renewable energy resources (RER), energy storage system (ESS) and demand side management (DSM) need to be orchestrated. Furthermore, these systems may belong to different entities and competition may exist between them. Nash equilibrium is most commonly used for coordination of such entities however the convergence and existence of Nash equilibrium can not always be guaranteed. To this end, we use the correlated equilibrium to coordinate agents, whose convergence can be guaranteed. In this paper, we build an energy trading model based on mid-market rate, and propose a correlated Q-learning (CEQ) algorithm to maximize the revenue of each agent. Our results show that CEQ is able to balance the revenue of agents without harming total benefit. In addition, compared with Q-learning without correlation, CEQ could save 19.3% cost for the DSM agent and 44.2% more benefits for the ESS agent.