Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

TL;DR

SILO-BENCH is a comprehensive benchmark testing multi-agent LLM systems' ability to coordinate and synthesize distributed information, revealing a fundamental reasoning gap that worsens with scale.

Contribution

Introduces SILO-BENCH, a novel benchmark with 30 tasks across communication levels, exposing coordination and reasoning limitations in multi-agent LLM systems.

Findings

Agents form task-appropriate topologies and exchange info effectively.

Fail to synthesize distributed info into correct answers during reasoning.

Coordination overhead increases with scale, negating parallelization benefits.

Abstract

Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agents. Yet whether agents can reliably compute with distributed information, rather than merely exchange it, remains an open question. We introduce SILO-BENCH, a role-agnostic benchmark of 30 algorithmic tasks across three communication complexity levels, evaluating 54 configurations over 1,620 experiments. Our experiments expose a fundamental Communication-Reasoning Gap: agents spontaneously form task-appropriate coordination topologies and exchange information actively, yet systematically fail to synthesize distributed state into correct answers. The failure is localized to the reasoning-integration stage where agents often acquire sufficient information but cannot integrate it. This coordination overhead compounds with scale, eventually eliminating parallelization gains entirely. These findings demonstrate that naively scaling agent count cannot circumvent context limitations, and SILO-BENCH provides a foundation for tracking progress toward genuinely collaborative multi-agent systems. The code is available at https://github.com/jwyjohn/acl26-silo-bench .