Towards a Science of Collective AI: LLM-based Multi-Agent Systems Need a Transition from Blind Trial-and-Error to Rigorous Science

TL;DR

This paper advocates for a scientific framework for Multi-Agent Systems with Large Language Models, emphasizing structured metrics and factor attribution to replace trial-and-error approaches and enable systematic progress.

Contribution

It introduces a unified collaboration gain metric and a factor attribution paradigm, along with a systematic MAS factor library, to establish a rigorous scientific methodology.

Findings

Proposes the collaboration gain metric ($\Gamma$) as a standard for intrinsic collaboration evaluation.

Develops a factor attribution paradigm to identify key factors driving collaboration.

Constructs a systematic MAS factor library to structure the design space.

Abstract

Recent advancements in Large Language Models (LLMs) have greatly extended the capabilities of Multi-Agent Systems (MAS), demonstrating significant effectiveness across a wide range of complex and open-ended domains. However, despite this rapid progress, the field still relies heavily on empirical trial-and-error. It lacks a unified and principled scientific framework necessary for systematic optimization and improvement. This bottleneck stems from the ambiguity of attribution: first, the absence of a structured taxonomy of factors leaves researchers restricted to unguided adjustments; second, the lack of a unified metric fails to distinguish genuine collaboration gain from mere resource accumulation. In this paper, we advocate for a transition to design science through an integrated framework. We advocate to establish the collaboration gain metric ($\Gamma$) as the scientific standard to isolate intrinsic gains from increased budgets. Leveraging $\Gamma$, we propose a factor attribution paradigm to systematically identify collaboration-driving factors. To support this, we construct a systematic MAS factor library, structuring the design space into control-level presets and information-level dynamics. Ultimately, this framework facilitates the transition from blind experimentation to rigorous science, paving the way towards a true science of Collective AI.