When Do Multi-Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systems

TL;DR

This paper provides a theoretical analysis of when Multi-Agent Reinforcement Learning (MARL) outperforms Single-Agent RL (SARL) in training large language models, focusing on sample efficiency and task decomposition.

Contribution

It introduces a formal PAC framework for comparing MARL and SARL, deriving sample complexity bounds and analyzing the effects of task decomposition and alignment.

Findings

MARL outperforms SARL with independent subtasks

Dependent subtasks reduce MARL's advantage

Task alignment impacts learning efficiency

Abstract

Reinforcement Learning (RL) has emerged as a crucial method for training or fine-tuning large language models (LLMs), enabling adaptive, task-specific optimizations through interactive feedback. Multi-Agent Reinforcement Learning (MARL), in particular, offers a promising avenue by decomposing complex tasks into specialized subtasks learned by distinct interacting agents, potentially enhancing the ability and efficiency of LLM systems. However, theoretical insights regarding when and why MARL outperforms Single-Agent RL (SARL) remain limited, creating uncertainty in selecting the appropriate RL framework. In this paper, we address this critical gap by rigorously analyzing the comparative sample efficiency of MARL and SARL within the context of LLM. Leveraging the Probably Approximately Correct (PAC) framework, we formally define SARL and MARL setups for LLMs, derive explicit sample complexity bounds, and systematically characterize how task decomposition and alignment influence learning efficiency. Our results demonstrate that MARL improves sample complexity when tasks naturally decompose into independent subtasks, whereas dependent subtasks diminish MARL's comparative advantage. Additionally, we introduce and analyze the concept of task alignment, quantifying the trade-offs when enforcing independent task decomposition despite potential misalignments. These theoretical insights clarify empirical inconsistencies and provide practical criteria for deploying MARL strategies effectively in complex LLM scenarios.