Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning

TL;DR

This study evaluates the effectiveness of multi-agent debate among large language models in logical reasoning tasks, revealing key factors that influence success and providing insights for designing better reasoning systems.

Contribution

It offers a systematic controlled study analyzing structural and cognitive factors affecting LLM debate performance using a logic puzzle, highlighting the importance of reasoning strength and diversity.

Findings

Intrinsic reasoning strength and diversity drive success

Structural factors like order and confidence visibility have limited impact

Effective teams can overturn incorrect consensus

Abstract

Multi-agent debate (MAD) has recently emerged as a promising framework for improving the reasoning performance of large language models (LLMs). Yet, whether LLM agents can genuinely engage in deliberative reasoning, beyond simple ensembling or majority voting, remains unclear. We address this question through a controlled study using the Knight--Knave--Spy logic puzzle, which enables precise, step-wise evaluation of debate outcomes and processes under verifiable ground truth. We systematically set up six structural and cognitive factors, including agent team size, composition, confidence visibility, debate order, debate depth, and task difficulty, to disentangle their respective effects on collective reasoning. Our results show that intrinsic reasoning strength and group diversity are the dominant drivers of debate success, while structural parameters such as order or confidence visibility offer limited gains. Beyond outcomes, process-level analyses identify key behavioral patterns: majority pressure suppresses independent correction, effective teams overturn incorrect consensus, and rational, validity-aligned reasoning most strongly predicts improvement. These findings provide valuable insights into how and why LLM debates succeed or fail, offering guidance for designing interpretable and truth-seeking multi-agent reasoning systems.