Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environment

TL;DR

This paper benchmarks Multi-Agent Reinforcement Learning algorithms for urban energy management using the CityLearn environment, introducing multi-KPI evaluation to reveal strengths and weaknesses of different approaches.

Contribution

It establishes a comprehensive multi-KPI benchmarking framework for MARL in energy control, including novel KPIs for real-world challenges and comparative analysis of training schemes.

Findings

DTDE outperforms CTDE in average and worst-case scenarios

Temporal dependency learning enhances control of memory-dependent KPIs

Policies show robustness to agent or resource removal

Abstract

The optimization of urban energy systems is crucial for the advancement of sustainable and resilient smart cities, which are becoming increasingly complex with multiple decision-making units. To address scalability and coordination concerns, Multi-Agent Reinforcement Learning (MARL) is a promising solution. This paper addresses the imperative need for comprehensive and reliable benchmarking of MARL algorithms on energy management tasks. CityLearn is used as a case study environment because it realistically simulates urban energy systems, incorporates multiple storage systems, and utilizes renewable energy sources. By doing so, our work sets a new standard for evaluation, conducting a comparative study across multiple key performance indicators (KPIs). This approach illuminates the key strengths and weaknesses of various algorithms, moving beyond traditional KPI averaging which often masks critical insights. Our experiments utilize widely accepted baselines such as Proximal Policy Optimization (PPO) and Soft Actor Critic (SAC), and encompass diverse training schemes including Decentralized Training with Decentralized Execution (DTDE) and Centralized Training with Decentralized Execution (CTDE) approaches and different neural network architectures. Our work also proposes novel KPIs that tackle real world implementation challenges such as individual building contribution and battery storage lifetime. Our findings show that DTDE consistently outperforms CTDE in both average and worst-case performance. Additionally, temporal dependency learning improved control on memory dependent KPIs such as ramping and battery usage, contributing to more sustainable battery operation. Results also reveal robustness to agent or resource removal, highlighting both the resilience and decentralizability of the learned policies.