Adaptive Value Decomposition: Coordinating a Varying Number of Agents in Urban Systems

TL;DR

This paper introduces Adaptive Value Decomposition, a novel MARL framework designed to handle dynamic agent populations and asynchronous actions in urban systems, improving coordination and diversity.

Contribution

It proposes a new adaptive MARL method that manages varying agent numbers and mitigates policy-induced action homogenization, enhancing urban system coordination.

Findings

AVD outperforms existing methods in bike-sharing redistribution tasks.

The framework effectively adapts to changing agent populations.

It maintains behavioral diversity and coordination quality.

Abstract

Multi-agent reinforcement learning (MARL) provides a promising paradigm for coordinating multi-agent systems (MAS). However, most existing methods rely on restrictive assumptions, such as a fixed number of agents and fully synchronous action execution. These assumptions are often violated in urban systems, where the number of active agents varies over time, and actions may have heterogeneous durations, resulting in a semi-MARL setting. Moreover, while sharing policy parameters among agents is commonly adopted to improve learning efficiency, it can lead to highly homogeneous actions when a subset of agents make decisions concurrently under similar observations, potentially degrading coordination quality. To address these challenges, we propose Adaptive Value Decomposition (AVD), a cooperative MARL framework that adapts to a dynamically changing agent population. AVD further incorporates a lightweight mechanism to mitigate action homogenization induced by shared policies, thereby encouraging behavioral diversity and maintaining effective cooperation among agents. In addition, we design a training-execution strategy tailored to the semi-MARL setting that accommodates asynchronous decision-making when some agents act at different times. Experiments on real-world bike-sharing redistribution tasks in two major cities, London and Washington, D.C., demonstrate that AVD outperforms state-of-the-art baselines, confirming its effectiveness and generalizability.