Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity

TL;DR

This paper introduces an information-theoretic framework to understand the limits of scaling multi-agent systems with LLMs, highlighting the importance of diversity over sheer agent quantity for improved performance.

Contribution

It presents a novel theoretical framework and empirical evidence showing that heterogeneity among agents enhances performance more effectively than increasing homogeneous agents.

Findings

Heterogeneous agents outperform large homogeneous groups.

Performance is bounded by task uncertainty, not agent count.

Effective channel count correlates with system performance.

Abstract

LLM-based multi-agent systems (MAS) have emerged as a promising approach to tackle complex tasks that are difficult for individual LLMs. A natural strategy is to scale performance by increasing the number of agents; however, we find that such scaling exhibits strong diminishing returns in homogeneous settings, while introducing heterogeneity (e.g., different models, prompts, or tools) continues to yield substantial gains. This raises a fundamental question: what limits scaling, and why does diversity help? We present an information-theoretic framework showing that MAS performance is bounded by the intrinsic task uncertainty, not by agent count. We derive architecture-agnostic bounds demonstrating that improvements depend on how many effective channels the system accesses. Homogeneous agents saturate early because their outputs are strongly correlated, whereas heterogeneous agents contribute complementary evidence. We further introduce $K^*$, an effective channel count that quantifies the number of effective channels without ground-truth labels. Empirically, we show that heterogeneous configurations consistently outperform homogeneous scaling: 2 diverse agents can match or exceed the performance of 16 homogeneous agents. Our results provide principled guidelines for building efficient and robust MAS through diversity-aware design. Code and Dataset are available at the link: https://github.com/SafeRL-Lab/Agent-Scaling.