WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate

TL;DR

This paper introduces WISE, a modular multi-agent debate framework for multimodal reasoning that improves accuracy by leveraging heterogeneous experts and a novel aggregation method.

Contribution

The paper presents WISE, a generalized MAD framework for multimodal tasks, incorporating heterogeneous agents and a modified Dawid-Skene aggregation algorithm, advancing multi-agent debate capabilities.

Findings

WISE improves accuracy by 2-7% over state-of-the-art methods.

The framework effectively handles heterogeneous multimodal experts.

Results demonstrate robustness across diverse datasets and LLM configurations.

Abstract

Recent large language models (LLMs) are trained on diverse corpora and tasks, leading them to develop complementary strengths. Multi-agent debate (MAD) has emerged as a popular way to leverage these strengths for robust reasoning, though it has mostly been applied to language-only tasks, leaving its efficacy on multimodal problems underexplored. In this paper, we study MAD for solving vision-and-language reasoning problems. Our setup enables generalizing the debate protocol with heterogeneous experts that possess single- and multi-modal capabilities. To this end, we present Weighted Iterative Society-of-Experts (WISE), a generalized and modular MAD framework that partitions the agents into Solvers, that generate solutions, and Reflectors, that verify correctness, assign weights, and provide natural language feedback. To aggregate the agents' solutions across debate rounds, while accounting for variance in their responses and the feedback weights, we present a modified Dawid-Skene algorithm for post-processing that integrates our two-stage debate model. We evaluate WISE on SMART-840, VisualPuzzles, EvoChart-QA, and a new SMART-840++ dataset with programmatically generated problem instances of controlled difficulty. Our results show that WISE consistently improves accuracy by 2-7% over the state-of-the-art MAD setups and aggregation methods across diverse multimodal tasks and LLM configurations.