LoopBench: Discovering Emergent Symmetry Breaking Strategies with LLM Swarms

TL;DR

LoopBench is a new benchmark that evaluates how well large language models can develop strategies to solve distributed symmetry breaking problems, revealing emergent reasoning capabilities in multi-agent settings.

Contribution

The paper introduces LoopBench, a benchmark for studying emergent distributed algorithms and reasoning in LLM swarms, focusing on symmetry breaking in graph coloring tasks.

Findings

Advanced reasoning models can escape deadlocks in symmetry breaking tasks.

Standard LLMs and classical heuristics often fail in distributed symmetry breaking.

LoopBench enables analysis of collective intelligence in language-based reasoning.

Abstract

Large Language Models (LLMs) are increasingly being utilized as autonomous agents, yet their ability to coordinate in distributed systems remains poorly understood. We introduce \textbf{LoopBench}, a benchmark to evaluate LLM reasoning in distributed symmetry breaking and meta-cognitive thinking. The benchmark focuses on coloring odd cycle graphs ($C_3, C_5, C_{11}$) with limited colors, where deterministic, non-communicating agents fail in infinite loops. A strategy passing mechanism is implemented as a form of consistent memory. We show that while standard LLMs and classical heuristics struggle, advanced reasoning models (e.g., O3) devise strategies to escape deadlocks. LoopBench allows the study of emergent distributed algorithms based on language-based reasoning, offering a testbed for collective intelligence.