Game Theory and Multi-Agent Reinforcement Learning for Zonal Ancillary Markets

TL;DR

This paper models zonal ancillary markets using game theory and compares multi-agent reinforcement learning with traditional optimization methods, revealing trade-offs in convergence, costs, and profit variability.

Contribution

It introduces a novel game-theoretic formulation of ancillary markets and evaluates multi-agent reinforcement learning against exact methods in real-world data.

Findings

Multi-agent deep reinforcement learning achieves the fastest convergence.

Reinforcement learning results in lower market costs but higher profit variability.

Stronger zone coupling reduces costs for larger zones.

Abstract

We characterize zonal ancillary market coupling relying on noncooperative game theory. To that purpose, we formulate the ancillary market as a multi-leader single follower bilevel problem, that we subsequently cast as a generalized Nash game with side constraints and nonconvex feasibility sets. We determine conditions for equilibrium existence and show that the game has a generalized potential game structure. To compute market equilibrium, we rely on two exact approaches: an integrated optimization approach and Gauss-Seidel best-response, that we compare against multi-agent deep reinforcement learning. On real data from Germany and Austria, simulations indicate that multi-agent deep reinforcement learning achieves the smallest convergence rate but requires pretraining, while best-response is the slowest. On the economics side, multi-agent deep reinforcement learning results in smaller market costs compared to the exact methods, but at the cost of higher variability in the profit allocation among stakeholders. Further, stronger coupling between zones tends to reduce costs for larger zones.