Investigating Relational State Abstraction in Collaborative MARL

TL;DR

This paper demonstrates that incorporating relational state abstraction based on spatial relationships into multi-agent reinforcement learning significantly improves sample efficiency and performance across various tasks.

Contribution

It introduces MARC, a critic architecture using spatial relational inductive biases via graph neural networks, showing benefits without complex engineering.

Findings

MARC outperforms state-of-the-art MARL baselines.

Relational abstraction improves sample efficiency and generalization.

Spatial relational biases are effective in multi-agent environments.

Abstract

This paper explores the impact of relational state abstraction on sample efficiency and performance in collaborative Multi-Agent Reinforcement Learning. The proposed abstraction is based on spatial relationships in environments where direct communication between agents is not allowed, leveraging the ubiquity of spatial reasoning in real-world multi-agent scenarios. We introduce MARC (Multi-Agent Relational Critic), a simple yet effective critic architecture incorporating spatial relational inductive biases by transforming the state into a spatial graph and processing it through a relational graph neural network. The performance of MARC is evaluated across six collaborative tasks, including a novel environment with heterogeneous agents. We conduct a comprehensive empirical analysis, comparing MARC against state-of-the-art MARL baselines, demonstrating improvements in both sample efficiency and asymptotic performance, as well as its potential for generalization. Our findings suggest that a minimal integration of spatial relational inductive biases as abstraction can yield substantial benefits without requiring complex designs or task-specific engineering. This work provides insights into the potential of relational state abstraction to address sample efficiency, a key challenge in MARL, offering a promising direction for developing more efficient algorithms in spatially complex environments.