Key Decision-Makers in Multi-Agent Debates: Who Holds the Power?

TL;DR

This paper investigates role allocation strategies in Multi-Agent Debate with LLMs, introducing the 'Truth Last' approach and the MADC method, which significantly improve reasoning performance and address truth estimation challenges.

Contribution

It introduces a novel role allocation strategy 'Truth Last' and the MADC method for better role consistency and truth estimation in MAD, advancing LLM agent scaling.

Findings

'Truth Last' improves MAD reasoning by up to 22%.

MADC enhances performance across 9 LLM models.

MADC effectively addresses truth estimation in MAD.

Abstract

Recent studies on LLM agent scaling have highlighted the potential of Multi-Agent Debate (MAD) to enhance reasoning abilities. However, the critical aspect of role allocation strategies remains underexplored. In this study, we demonstrate that allocating roles with differing viewpoints to specific positions significantly impacts MAD's performance in reasoning tasks. Specifically, we find a novel role allocation strategy, "Truth Last", which can improve MAD performance by up to 22% in reasoning tasks. To address the issue of unknown truth in practical applications, we propose the Multi-Agent Debate Consistency (MADC) strategy, which systematically simulates and optimizes its core mechanisms. MADC incorporates path consistency to assess agreement among independent roles, simulating the role with the highest consistency score as the truth. We validated MADC across a range of LLMs (9 models), including the DeepSeek-R1 Distilled Models, on challenging reasoning tasks. MADC consistently demonstrated advanced performance, effectively overcoming MAD's performance bottlenecks and providing a crucial pathway for further improvements in LLM agent scaling.