Voting or Consensus? Decision-Making in Multi-Agent Debate

TL;DR

This paper systematically evaluates how different decision protocols affect multi-agent debate outcomes, revealing that voting improves reasoning performance and proposing new methods to enhance decision diversity.

Contribution

It provides a systematic comparison of seven decision protocols, analyzes their impact on various tasks, and introduces two novel methods to improve decision-making in multi-agent debates.

Findings

Voting protocols improve reasoning task performance by 13.2%.

Consensus protocols enhance knowledge task performance by 2.8%.

Proposed methods AAD and CI improve task performance by up to 3.3% and 7.4%.

Abstract

Much of the success of multi-agent debates depends on carefully choosing the right parameters. The decision-making protocol stands out as it can highly impact final model answers, depending on how decisions are reached. Systematic comparison of decision protocols is difficult because many studies alter multiple discussion parameters beyond the protocol. So far, it has been largely unknown how decision-making influences different tasks. This work systematically evaluates the impact of seven decision protocols (e.g., majority voting, unanimity consensus). We change only one variable at a time - the decision protocol - to analyze how different methods affect the collaboration between agents and measure differences in knowledge and reasoning tasks. Our results show that voting protocols improve performance by 13.2% in reasoning tasks and consensus protocols by 2.8% in knowledge tasks compared to other decision protocols. Increasing the number of agents improves performance, while more discussion rounds before voting reduce it. To improve decision-making by increasing answer diversity, we propose two new methods, All-Agents Drafting (AAD) and Collective Improvement (CI). Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI. This work demonstrates the importance of decision-making in multi-agent debates beyond scaling.