From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos

TL;DR

This paper introduces SoCo, a framework that leverages solo demonstrations to improve the efficiency and effectiveness of multi-agent reinforcement learning by transferring solo knowledge into cooperative multi-agent policies.

Contribution

The paper proposes a novel Solo-to-Collaborative RL (SoCo) framework that pretrains solo policies and adapts them for multi-agent cooperation using a policy fusion mechanism.

Findings

SoCo significantly improves training efficiency.

SoCo enhances performance across various cooperative tasks.

Solo demonstrations effectively complement multi-agent data.

Abstract

Training a team of agents from scratch in multi-agent reinforcement learning (MARL) is highly inefficient, much like asking beginners to play a symphony together without first practicing solo. Existing methods, such as offline or transferable MARL, can ease this burden, but they still rely on costly multi-agent data, which often becomes the bottleneck. In contrast, solo experiences are far easier to obtain in many important scenarios, e.g., collaborative coding, household cooperation, and search-and-rescue. To unlock their potential, we propose Solo-to-Collaborative RL (SoCo), a framework that transfers solo knowledge into cooperative learning. SoCo first pretrains a shared solo policy from solo demonstrations, then adapts it for cooperation during multi-agent training through a policy fusion mechanism that combines an MoE-like gating selector and an action editor. Experiments across diverse cooperative tasks show that SoCo significantly boosts the training efficiency and performance of backbone algorithms. These results demonstrate that solo demonstrations provide a scalable and effective complement to multi-agent data, making cooperative learning more practical and broadly applicable.