PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems

TL;DR

PowerGridworld is an open-source framework that enables rapid development and prototyping of multi-agent reinforcement learning environments tailored for power systems, integrating power flow solutions for realistic modeling.

Contribution

It introduces a modular, customizable software package for creating power system-focused multi-agent RL environments, filling a gap in existing frameworks.

Findings

Successfully demonstrated MARL policy learning with MADDPG and PPO algorithms.

Highlighted the integration of power flow solutions into agent reward structures.

Provided case studies showcasing the framework's capabilities.

Abstract

We present the PowerGridworld software package to provide users with a lightweight, modular, and customizable framework for creating power-systems-focused, multi-agent Gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). Although many frameworks exist for training multi-agent RL (MARL) policies, none can rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multi-device) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an open-source software package that helps to fill this gap. To highlight PowerGridworld's key features, we present two case studies and demonstrate learning MARL policies using both OpenAI's multi-agent deep deterministic policy gradient (MADDPG) and RLLib's proximal policy optimization (PPO) algorithms. In both cases, at least some subset of agents incorporates elements of the power flow solution at each time step as part of their reward (negative cost) structures.