Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information

TL;DR

This paper introduces SCAM-FQI, a scalable multi-agent offline RL method that balances dataset collection and policy learning through structured information sharing, ensuring convergence to near-optimal policies.

Contribution

It proposes a novel scalable routine for dataset collection and offline learning in multi-agent RL, with theoretical convergence guarantees and bounds based on shared information.

Findings

SCAM-FQI converges to near-optimal policies with high probability.

The approach balances scalability and performance in multi-agent offline RL.

Empirical results support theoretical convergence and effectiveness.

Abstract

Offline Reinforcement Learning (RL) focuses on learning policies solely from a batch of previously collected data. offering the potential to leverage such datasets effectively without the need for costly or risky active exploration. While recent advances in Offline Multi-Agent RL (MARL) have shown promise, most existing methods either rely on large datasets jointly collected by all agents or agent-specific datasets collected independently. The former approach ensures strong performance but raises scalability concerns, while the latter emphasizes scalability at the expense of performance guarantees. In this work, we propose a novel scalable routine for both dataset collection and offline learning. Agents first collect diverse datasets coherently with a pre-specified information-sharing network and subsequently learn coherent localized policies without requiring either full observability or falling back to complete decentralization. We theoretically demonstrate that this structured approach allows a multi-agent extension of the seminal Fitted Q-Iteration (FQI) algorithm to globally converge, in high probability, to near-optimal policies. The convergence is subject to error terms that depend on the informativeness of the shared information. Furthermore, we show how this approach allows to bound the inherent error of the supervised-learning phase of FQI with the mutual information between shared and unshared information. Our algorithm, SCAlable Multi-agent FQI (SCAM-FQI), is then evaluated on a distributed decision-making problem. The empirical results align with our theoretical findings, supporting the effectiveness of SCAM-FQI in achieving a balance between scalability and policy performance.