Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System

TL;DR

This paper introduces MADKE, a multi-agent debate framework enhanced with shared retrieval knowledge and adaptive knowledge selection, significantly improving the accuracy and consistency of multi-agent systems in truth-seeking tasks, surpassing GPT-4 in some benchmarks.

Contribution

The paper proposes MADKE, a novel multi-agent debate system with shared knowledge pools and adaptive knowledge selection, enabling agents to overcome cognitive limitations and achieve state-of-the-art results.

Findings

MADKE achieves state-of-the-art results on six datasets.

Retrieval knowledge helps break cognitive islands among agents.

MADKE surpasses GPT-4 performance on average in six datasets.

Abstract

Multi-agent debate system (MAD) imitating the process of human discussion in pursuit of truth, aims to align the correct cognition of different agents for the optimal solution. It is challenging to make various agents perform right and highly consistent cognition due to their limited and different knowledge backgrounds (i.e., cognitive islands), which hinders the search for the optimal solution. To address the challenge, we propose a novel \underline{M}ulti-\underline{A}gent \underline{D}ebate with \underline{K}nowledge-\underline{E}nhanced framework (\textbf{MADKE}) to promote the system to find the solution. First, we involve a shared retrieval knowledge pool in the debate process to solve the problem of limited and different knowledge backgrounds. Then, we propose an adaptive knowledge selection method to guarantee the accuracy and personalization of knowledge. This method allows agents to choose whether to use external knowledge in each conversation round according to their own needs. Our experimental results on six datasets show that our method achieves state-of-the-art results compared to existing single-agent and multi-agent methods. Further analysis reveals that the introduction of retrieval knowledge can help the agent to break cognitive islands in the debate process and effectively improve the consistency and correctness of the model. Moreover, MADKE using Qwen1.5-72B-Chat surpasses GPT-4 by +1.26\% on average in six datasets, which validates that our method can help open-source LLMs achieve or even surpass the performance of GPT-4. Our code is available at \url{https://github.com/FutureForMe/MADKE}.