iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference

TL;DR

iMAD introduces an efficient framework that selectively employs multi-agent debate only when beneficial, significantly reducing computational costs and improving accuracy in large language model inference.

Contribution

The paper presents iMAD, a novel token-efficient approach that learns to decide when to trigger multi-agent debates, enhancing reasoning accuracy without extensive dataset tuning.

Findings

Reduces token usage by up to 92%.

Improves answer accuracy by up to 13.5%.

Effective across multiple visual question answering datasets.

Abstract

Large Language Model (LLM) agent systems have advanced rapidly, driven by their strong generalization in zero-shot settings. To further enhance reasoning and accuracy on complex tasks, Multi-Agent Debate (MAD) has emerged as a promising framework that engages multiple LLM agents in structured debates to encourage diverse reasoning. However, triggering MAD for every query is inefficient, as it incurs substantial computational (token) cost and may even degrade accuracy by overturning correct single-agent answers. To address these limitations, we propose intelligent Multi-Agent Debate (iMAD), a token-efficient framework that selectively triggers MAD only when it is likely to be beneficial (i.e., correcting an initially wrong answer). To achieve this goal, iMAD learns generalizable model behaviors to make accurate debate decisions. Specifically, iMAD first prompts a single agent to produce a structured self-critique response, from which we extract 41 interpretable linguistic and semantic features capturing hesitation cues. Then, iMAD uses a lightweight debate-decision classifier, trained using our proposed FocusCal loss, to determine whether to trigger MAD, enabling robust debate decisions without test dataset-specific tuning. Through extensive experiments using six (visual) question answering datasets against five competitive baselines, we have shown that iMAD significantly reduces token usage (by up to 92%) while also improving final answer accuracy (by up to 13.5%).