Demystifying Multi-Agent Debate: The Role of Confidence and Diversity

TL;DR

This paper enhances multi-agent debate for large language models by introducing diversity and confidence mechanisms, leading to improved accuracy over traditional methods across multiple benchmarks.

Contribution

It identifies key missing elements in vanilla MAD—diversity and confidence—and proposes simple interventions that significantly improve debate outcomes.

Findings

Diversity-aware initialisation increases the chance of correct hypotheses.

Confidence-modulated updates guide debates towards correct answers.

Methods outperform vanilla MAD and majority vote on six benchmarks.

Abstract

Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simple majority vote despite higher computational cost. Studies show that, under homogeneous agents and uniform belief updates, debate preserves expected correctness and therefore cannot reliably improve outcomes. Drawing on findings from human deliberation and collective decision-making, we identify two key mechanisms missing from vanilla MAD: (i) diversity of initial viewpoints and (ii) explicit, calibrated confidence communication. We propose two lightweight interventions. First, a diversity-aware initialisation that selects a more diverse pool of candidate answers, increasing the likelihood that a correct hypothesis is present at the start of debate. Second, a confidence-modulated debate protocol in which agents express calibrated confidence and condition their updates on others' confidence. We show theoretically that diversity-aware initialisation improves the prior probability of MAD success without changing the underlying update dynamics, while confidence-modulated updates enable debate to systematically drift to the correct hypothesis. Empirically, across six reasoning-oriented QA benchmarks, our methods consistently outperform vanilla MAD and majority vote. Our results connect human deliberation with LLM-based debate and demonstrate that simple, principled modifications can substantially enhance debate effectiveness.